2021-09-16:17:22:35,504 INFO     [iwn.py:43] Loading bengali language synsets...
2021-09-16:17:22:49,208 WARNING  [utils.py:429] using automatically assigned random_state=3179480306
2021-09-16:17:22:49,361 INFO     [splitting.py:60] done splitting triples to groups of sizes [267018, 38830, 38831]
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py:34: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from ._gradient_boosting import predict_stages
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py:34: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from ._gradient_boosting import predict_stages
[I 2021-09-16 17:22:53,483] A new study created in memory with name: no-name-8e955b25-a1f3-4a57-b42e-1f7594053d64
2021-09-16:17:22:53,565 INFO     [hpo.py:622] Using model: <class 'pykeen.models.unimodal.distmult.DistMult'>
2021-09-16:17:22:53,565 INFO     [hpo.py:626] Using loss: <class 'pykeen.losses.MarginRankingLoss'>
2021-09-16:17:22:53,565 INFO     [hpo.py:637] Using regularizer: <class 'pykeen.regularizers.LpRegularizer'>
2021-09-16:17:22:53,566 INFO     [hpo.py:641] Using optimizer: <class 'torch.optim.adam.Adam'>
2021-09-16:17:22:53,566 INFO     [hpo.py:645] Using training loop: <class 'pykeen.training.slcwa.SLCWATrainingLoop'>
2021-09-16:17:22:53,566 INFO     [hpo.py:651] Using negative sampler: <class 'pykeen.sampling.basic_negative_sampler.BasicNegativeSampler'>
2021-09-16:17:22:53,566 INFO     [hpo.py:662] Using evaluator: <class 'pykeen.evaluation.rank_based_evaluator.RankBasedEvaluator'>
2021-09-16:17:22:53,566 INFO     [hpo.py:666] Attempting to maximize adjusted_arithmetic_mean_rank_index
2021-09-16:17:22:53,566 INFO     [hpo.py:668] Filter validation triples when testing: True
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:22:53,584 WARNING  [api.py:823] No random seed is specified. Setting to 1437431581.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/119 [00:00<00:04, 28.35batch/s]

Training batches on cuda:   7%|▋         | 8/119 [00:00<00:03, 32.54batch/s]

Training batches on cuda:  12%|█▏        | 14/119 [00:00<00:02, 37.49batch/s]

Training batches on cuda:  18%|█▊        | 21/119 [00:00<00:02, 42.47batch/s]

Training batches on cuda:  24%|██▎       | 28/119 [00:00<00:01, 46.68batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:00<00:01, 50.55batch/s]

Training batches on cuda:  35%|███▌      | 42/119 [00:00<00:01, 53.41batch/s]

Training batches on cuda:  41%|████      | 49/119 [00:00<00:01, 55.67batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:00<00:01, 57.61batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:01<00:00, 60.02batch/s]

Training batches on cuda:  59%|█████▉    | 70/119 [00:01<00:00, 62.36batch/s]

Training batches on cuda:  66%|██████▌   | 78/119 [00:01<00:00, 65.01batch/s]

Training batches on cuda:  72%|███████▏  | 86/119 [00:01<00:00, 66.87batch/s]

Training batches on cuda:  79%|███████▉  | 94/119 [00:01<00:00, 67.87batch/s]

Training batches on cuda:  86%|████████▌ | 102/119 [00:01<00:00, 68.89batch/s]

Training batches on cuda:  92%|█████████▏| 110/119 [00:01<00:00, 70.22batch/s]

Training batches on cuda:  99%|█████████▉| 118/119 [00:01<00:00, 69.94batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000509, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:18,  2.06s/epoch, loss=0.000509, prev_loss=nan]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:03, 37.65batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:02, 43.41batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:02, 49.46batch/s]

Training batches on cuda:  22%|██▏       | 26/119 [00:00<00:01, 54.06batch/s]

Training batches on cuda:  28%|██▊       | 33/119 [00:00<00:01, 57.96batch/s]

Training batches on cuda:  34%|███▍      | 41/119 [00:00<00:01, 61.79batch/s]

Training batches on cuda:  41%|████      | 49/119 [00:00<00:01, 64.88batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:00<00:01, 50.70batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:01<00:01, 53.95batch/s]

Training batches on cuda:  59%|█████▉    | 70/119 [00:01<00:00, 56.35batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:01<00:00, 58.98batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:01<00:00, 62.00batch/s]

Training batches on cuda:  78%|███████▊  | 93/119 [00:01<00:00, 64.88batch/s]

Training batches on cuda:  85%|████████▍ | 101/119 [00:01<00:00, 66.42batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:01<00:00, 68.07batch/s]

Training batches on cuda:  97%|█████████▋| 116/119 [00:01<00:00, 68.02batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:18,  2.06s/epoch, loss=0.000503, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:16,  2.06s/epoch, loss=0.000503, prev_loss=nan]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▌         | 6/119 [00:00<00:02, 55.71batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:01, 58.87batch/s]

Training batches on cuda:  18%|█▊        | 21/119 [00:00<00:01, 62.25batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:00<00:01, 65.28batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:00<00:01, 67.44batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:00<00:01, 67.61batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:00<00:00, 69.78batch/s]

Training batches on cuda:  50%|█████     | 60/119 [00:00<00:00, 71.22batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:00<00:00, 69.17batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:01<00:00, 70.06batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:01<00:00, 71.18batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:01<00:00, 71.30batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:01<00:00, 71.23batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:01<00:00, 72.26batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:01<00:00, 71.82batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:16,  2.06s/epoch, loss=0.000503, prev_loss=0.000503]
Training epochs on cuda:  30%|███       | 3/10 [00:05<00:13,  1.99s/epoch, loss=0.000503, prev_loss=0.000503]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:02, 49.04batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:01, 53.37batch/s]

Training batches on cuda:  18%|█▊        | 21/119 [00:00<00:01, 58.15batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:00<00:01, 61.99batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:00<00:01, 64.57batch/s]

Training batches on cuda:  38%|███▊      | 45/119 [00:00<00:01, 65.35batch/s]

Training batches on cuda:  45%|████▍     | 53/119 [00:00<00:00, 67.46batch/s]

Training batches on cuda:  51%|█████▏    | 61/119 [00:00<00:00, 68.65batch/s]

Training batches on cuda:  58%|█████▊    | 69/119 [00:01<00:00, 69.58batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:01<00:00, 70.07batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:01<00:00, 70.21batch/s]

Training batches on cuda:  77%|███████▋  | 92/119 [00:01<00:00, 70.00batch/s]

Training batches on cuda:  84%|████████▍ | 100/119 [00:01<00:00, 70.69batch/s]

Training batches on cuda:  91%|█████████ | 108/119 [00:01<00:00, 71.14batch/s]

Training batches on cuda:  97%|█████████▋| 116/119 [00:01<00:00, 71.65batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:13,  1.99s/epoch, loss=0.000503, prev_loss=0.000503]
Training epochs on cuda:  40%|████      | 4/10 [00:07<00:11,  1.98s/epoch, loss=0.000503, prev_loss=0.000503]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:02, 45.53batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:02, 51.33batch/s]

Training batches on cuda:  18%|█▊        | 21/119 [00:00<00:01, 56.23batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:00<00:01, 60.45batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:00<00:01, 64.58batch/s]

Training batches on cuda:  38%|███▊      | 45/119 [00:00<00:01, 66.47batch/s]

Training batches on cuda:  45%|████▍     | 53/119 [00:00<00:00, 69.19batch/s]

Training batches on cuda:  51%|█████▏    | 61/119 [00:00<00:00, 70.15batch/s]

Training batches on cuda:  58%|█████▊    | 69/119 [00:00<00:00, 70.38batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:01<00:00, 70.70batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:01<00:00, 71.91batch/s]

Training batches on cuda:  78%|███████▊  | 93/119 [00:01<00:00, 72.01batch/s]

Training batches on cuda:  85%|████████▍ | 101/119 [00:01<00:00, 72.59batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:01<00:00, 72.39batch/s]

Training batches on cuda:  98%|█████████▊| 117/119 [00:01<00:00, 72.48batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:11,  1.98s/epoch, loss=0.000503, prev_loss=0.000503]
Training epochs on cuda:  50%|█████     | 5/10 [00:09<00:09,  1.95s/epoch, loss=0.000503, prev_loss=0.000503]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:02, 46.04batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:02, 51.76batch/s]

Training batches on cuda:  18%|█▊        | 21/119 [00:00<00:01, 56.91batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:00<00:01, 60.85batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:00<00:01, 63.92batch/s]

Training batches on cuda:  38%|███▊      | 45/119 [00:00<00:01, 66.26batch/s]

Training batches on cuda:  45%|████▍     | 53/119 [00:00<00:00, 68.54batch/s]

Training batches on cuda:  51%|█████▏    | 61/119 [00:00<00:00, 69.69batch/s]

Training batches on cuda:  58%|█████▊    | 69/119 [00:00<00:00, 70.61batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:01<00:00, 71.19batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:01<00:00, 71.92batch/s]

Training batches on cuda:  78%|███████▊  | 93/119 [00:01<00:00, 72.02batch/s]

Training batches on cuda:  85%|████████▍ | 101/119 [00:01<00:00, 72.72batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:01<00:00, 72.58batch/s]

Training batches on cuda:  98%|█████████▊| 117/119 [00:01<00:00, 71.81batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:09,  1.95s/epoch, loss=0.000503, prev_loss=0.000503]
Training epochs on cuda:  60%|██████    | 6/10 [00:11<00:07,  1.93s/epoch, loss=0.000503, prev_loss=0.000503]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:03, 37.95batch/s]

Training batches on cuda:  10%|█         | 12/119 [00:00<00:02, 44.16batch/s]

Training batches on cuda:  15%|█▌        | 18/119 [00:00<00:02, 47.45batch/s]

Training batches on cuda:  20%|██        | 24/119 [00:00<00:01, 50.50batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:00<00:01, 54.48batch/s]

Training batches on cuda:  32%|███▏      | 38/119 [00:00<00:01, 58.33batch/s]

Training batches on cuda:  38%|███▊      | 45/119 [00:00<00:01, 60.94batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:00<00:01, 58.83batch/s]

Training batches on cuda:  49%|████▊     | 58/119 [00:00<00:01, 57.22batch/s]

Training batches on cuda:  54%|█████▍    | 64/119 [00:01<00:00, 56.16batch/s]

Training batches on cuda:  59%|█████▉    | 70/119 [00:01<00:00, 55.27batch/s]

Training batches on cuda:  64%|██████▍   | 76/119 [00:01<00:00, 56.19batch/s]

Training batches on cuda:  69%|██████▉   | 82/119 [00:01<00:00, 51.59batch/s]

Training batches on cuda:  75%|███████▍  | 89/119 [00:01<00:00, 54.76batch/s]

Training batches on cuda:  81%|████████  | 96/119 [00:01<00:00, 56.39batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:01<00:00, 57.70batch/s]

Training batches on cuda:  92%|█████████▏| 110/119 [00:01<00:00, 58.65batch/s]

Training batches on cuda:  98%|█████████▊| 117/119 [00:01<00:00, 61.20batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:07,  1.93s/epoch, loss=0.000503, prev_loss=0.000503]
Training epochs on cuda:  70%|███████   | 7/10 [00:13<00:06,  2.01s/epoch, loss=0.000503, prev_loss=0.000503]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▌         | 6/119 [00:00<00:02, 52.33batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:01, 56.13batch/s]

Training batches on cuda:  17%|█▋        | 20/119 [00:00<00:01, 59.54batch/s]

Training batches on cuda:  24%|██▎       | 28/119 [00:00<00:01, 62.67batch/s]

Training batches on cuda:  30%|███       | 36/119 [00:00<00:01, 65.17batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:00<00:01, 66.75batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:00<00:00, 67.98batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:00<00:00, 67.63batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:00<00:00, 68.68batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:01<00:00, 69.27batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:01<00:00, 69.98batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:01<00:00, 70.59batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:01<00:00, 69.76batch/s]

Training batches on cuda:  89%|████████▉ | 106/119 [00:01<00:00, 69.66batch/s]

Training batches on cuda:  95%|█████████▍| 113/119 [00:01<00:00, 69.72batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:15<00:06,  2.01s/epoch, loss=0.000503, prev_loss=0.000503]
Training epochs on cuda:  80%|████████  | 8/10 [00:15<00:03,  1.97s/epoch, loss=0.000503, prev_loss=0.000503]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:02, 38.80batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:02, 43.87batch/s]

Training batches on cuda:  15%|█▌        | 18/119 [00:00<00:02, 48.01batch/s]

Training batches on cuda:  21%|██        | 25/119 [00:00<00:01, 52.23batch/s]

Training batches on cuda:  27%|██▋       | 32/119 [00:00<00:01, 55.23batch/s]

Training batches on cuda:  34%|███▎      | 40/119 [00:00<00:01, 60.14batch/s]

Training batches on cuda:  40%|████      | 48/119 [00:00<00:01, 63.86batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:00<00:00, 66.75batch/s]

Training batches on cuda:  54%|█████▍    | 64/119 [00:00<00:00, 68.57batch/s]

Training batches on cuda:  61%|██████    | 72/119 [00:01<00:00, 69.37batch/s]

Training batches on cuda:  67%|██████▋   | 80/119 [00:01<00:00, 71.09batch/s]

Training batches on cuda:  74%|███████▍  | 88/119 [00:01<00:00, 72.75batch/s]

Training batches on cuda:  81%|████████  | 96/119 [00:01<00:00, 74.13batch/s]

Training batches on cuda:  87%|████████▋ | 104/119 [00:01<00:00, 75.33batch/s]

Training batches on cuda:  94%|█████████▍| 112/119 [00:01<00:00, 75.69batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:17<00:03,  1.97s/epoch, loss=0.000503, prev_loss=0.000503]
Training epochs on cuda:  90%|█████████ | 9/10 [00:17<00:01,  1.95s/epoch, loss=0.000503, prev_loss=0.000503]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:02, 46.13batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:02, 52.58batch/s]

Training batches on cuda:  18%|█▊        | 21/119 [00:00<00:01, 58.11batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:00<00:01, 62.44batch/s]

Training batches on cuda:  30%|███       | 36/119 [00:00<00:01, 64.36batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:00<00:01, 66.03batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:00<00:00, 67.50batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:00<00:00, 67.56batch/s]

Training batches on cuda:  55%|█████▌    | 66/119 [00:00<00:00, 67.93batch/s]

Training batches on cuda:  62%|██████▏   | 74/119 [00:01<00:00, 68.73batch/s]

Training batches on cuda:  69%|██████▉   | 82/119 [00:01<00:00, 69.22batch/s]

Training batches on cuda:  75%|███████▍  | 89/119 [00:01<00:00, 69.38batch/s]

Training batches on cuda:  81%|████████  | 96/119 [00:01<00:00, 67.10batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:01<00:00, 62.64batch/s]

Training batches on cuda:  92%|█████████▏| 110/119 [00:01<00:00, 59.88batch/s]

Training batches on cuda:  98%|█████████▊| 117/119 [00:01<00:00, 58.22batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:19<00:01,  1.95s/epoch, loss=0.000503, prev_loss=0.000503]2021-09-16:17:23:27,627 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:23:28,864 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:23:43,639 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
2021-09-16:17:23:43,770 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:35<00:00,  6.81s/epoch, loss=0.000503, prev_loss=0.000503]
Training epochs on cuda: 100%|██████████| 10/10 [00:35<00:00,  3.58s/epoch, loss=0.000503, prev_loss=0.000503]
2021-09-16:17:23:43,771 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpicvbza94'
2021-09-16:17:23:43,840 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpicvbza94' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.51ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.62ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.63ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.66ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:12, 2.67ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.66ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.67ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.67ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.67ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.67ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:10, 2.67ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.67ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.67ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.67ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.67ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.67ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.67ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.67ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.67ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.67ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.67ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.67ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.67ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.67ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.67ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.67ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.67ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.67ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.67ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.67ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:06, 2.67ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.67ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:07<00:06, 2.67ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.67ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.67ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.67ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.67ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.67ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.67ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.67ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.67ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.67ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.67ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.67ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.67ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.67ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.67ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.67ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.67ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.67ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.67ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.67ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.67ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:11<00:02, 2.67ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.67ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.67ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.67ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.67ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:12<00:01, 2.67ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.67ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.67ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.67ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.67ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:13<00:00, 2.67ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.67ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.67ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.67ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
2021-09-16:17:23:58,500 INFO     [evaluator.py:657] Evaluation took 14.65s seconds
[I 2021-09-16 17:23:58,506] Trial 0 finished with value: 0.0034486420364256976 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.039000929258803704, 'optimizer.lr': 0.004040537337421067, 'negative_sampler.num_negs_per_pos': 3, 'training.batch_size': 2632}. Best is trial 0 with value: 0.0034486420364256976.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:23:58,508 WARNING  [api.py:823] No random seed is specified. Setting to 2520028856.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 9/584 [00:00<00:06, 83.07batch/s]

Training batches on cuda:   3%|▎         | 17/584 [00:00<00:06, 81.63batch/s]

Training batches on cuda:   5%|▌         | 30/584 [00:00<00:06, 91.11batch/s]

Training batches on cuda:   7%|▋         | 43/584 [00:00<00:05, 99.25batch/s]

Training batches on cuda:  10%|▉         | 56/584 [00:00<00:05, 104.89batch/s]

Training batches on cuda:  12%|█▏        | 69/584 [00:00<00:04, 110.02batch/s]

Training batches on cuda:  14%|█▍        | 82/584 [00:00<00:04, 114.29batch/s]

Training batches on cuda:  16%|█▋        | 96/584 [00:00<00:04, 120.88batch/s]

Training batches on cuda:  19%|█▉        | 111/584 [00:00<00:03, 126.70batch/s]

Training batches on cuda:  22%|██▏       | 126/584 [00:01<00:03, 131.49batch/s]

Training batches on cuda:  24%|██▍       | 141/584 [00:01<00:03, 134.02batch/s]

Training batches on cuda:  27%|██▋       | 156/584 [00:01<00:03, 136.16batch/s]

Training batches on cuda:  29%|██▉       | 170/584 [00:01<00:03, 136.15batch/s]

Training batches on cuda:  32%|███▏      | 185/584 [00:01<00:02, 138.09batch/s]

Training batches on cuda:  34%|███▍      | 200/584 [00:01<00:02, 138.93batch/s]

Training batches on cuda:  37%|███▋      | 215/584 [00:01<00:02, 139.92batch/s]

Training batches on cuda:  39%|███▉      | 230/584 [00:01<00:02, 140.60batch/s]

Training batches on cuda:  42%|████▏     | 245/584 [00:01<00:02, 141.13batch/s]

Training batches on cuda:  45%|████▍     | 260/584 [00:01<00:02, 141.52batch/s]

Training batches on cuda:  47%|████▋     | 275/584 [00:02<00:02, 141.89batch/s]

Training batches on cuda:  50%|████▉     | 290/584 [00:02<00:02, 142.69batch/s]

Training batches on cuda:  52%|█████▏    | 305/584 [00:02<00:01, 142.81batch/s]

Training batches on cuda:  55%|█████▍    | 320/584 [00:02<00:01, 143.41batch/s]

Training batches on cuda:  57%|█████▋    | 335/584 [00:02<00:01, 143.32batch/s]

Training batches on cuda:  60%|█████▉    | 350/584 [00:02<00:01, 143.60batch/s]

Training batches on cuda:  62%|██████▎   | 365/584 [00:02<00:01, 144.01batch/s]

Training batches on cuda:  65%|██████▌   | 380/584 [00:02<00:01, 144.24batch/s]

Training batches on cuda:  68%|██████▊   | 395/584 [00:02<00:01, 142.66batch/s]

Training batches on cuda:  71%|███████   | 413/584 [00:03<00:01, 150.35batch/s]

Training batches on cuda:  74%|███████▍  | 431/584 [00:03<00:00, 156.83batch/s]

Training batches on cuda:  77%|███████▋  | 449/584 [00:03<00:00, 161.54batch/s]

Training batches on cuda:  80%|███████▉  | 467/584 [00:03<00:00, 164.54batch/s]

Training batches on cuda:  83%|████████▎ | 484/584 [00:03<00:00, 165.25batch/s]

Training batches on cuda:  86%|████████▌ | 501/584 [00:03<00:00, 165.08batch/s]

Training batches on cuda:  89%|████████▊ | 518/584 [00:03<00:00, 165.49batch/s]

Training batches on cuda:  92%|█████████▏| 535/584 [00:03<00:00, 163.31batch/s]

Training batches on cuda:  95%|█████████▍| 552/584 [00:03<00:00, 163.28batch/s]

Training batches on cuda:  97%|█████████▋| 569/584 [00:03<00:00, 163.18batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.00247, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:38,  4.24s/epoch, loss=0.00247, prev_loss=nan]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 12/584 [00:00<00:04, 119.16batch/s]

Training batches on cuda:   5%|▍         | 29/584 [00:00<00:04, 130.30batch/s]

Training batches on cuda:   8%|▊         | 45/584 [00:00<00:03, 137.84batch/s]

Training batches on cuda:  11%|█         | 63/584 [00:00<00:03, 146.93batch/s]

Training batches on cuda:  14%|█▎        | 80/584 [00:00<00:03, 152.86batch/s]

Training batches on cuda:  17%|█▋        | 98/584 [00:00<00:03, 157.56batch/s]

Training batches on cuda:  20%|█▉        | 116/584 [00:00<00:02, 162.36batch/s]

Training batches on cuda:  23%|██▎       | 133/584 [00:00<00:02, 164.23batch/s]

Training batches on cuda:  26%|██▌       | 151/584 [00:00<00:02, 167.45batch/s]

Training batches on cuda:  29%|██▉       | 168/584 [00:01<00:02, 164.64batch/s]

Training batches on cuda:  32%|███▏      | 186/584 [00:01<00:02, 167.23batch/s]

Training batches on cuda:  35%|███▍      | 203/584 [00:01<00:02, 167.82batch/s]

Training batches on cuda:  38%|███▊      | 221/584 [00:01<00:02, 168.57batch/s]

Training batches on cuda:  41%|████      | 238/584 [00:01<00:02, 168.90batch/s]

Training batches on cuda:  44%|████▍     | 256/584 [00:01<00:01, 169.74batch/s]

Training batches on cuda:  47%|████▋     | 273/584 [00:01<00:01, 167.07batch/s]

Training batches on cuda:  50%|████▉     | 290/584 [00:01<00:01, 167.91batch/s]

Training batches on cuda:  53%|█████▎    | 307/584 [00:01<00:01, 164.59batch/s]

Training batches on cuda:  55%|█████▌    | 324/584 [00:01<00:01, 160.72batch/s]

Training batches on cuda:  58%|█████▊    | 341/584 [00:02<00:01, 158.95batch/s]

Training batches on cuda:  61%|██████    | 357/584 [00:02<00:01, 158.33batch/s]

Training batches on cuda:  64%|██████▍   | 373/584 [00:02<00:01, 156.98batch/s]

Training batches on cuda:  67%|██████▋   | 390/584 [00:02<00:01, 159.06batch/s]

Training batches on cuda:  70%|██████▉   | 406/584 [00:02<00:01, 158.70batch/s]

Training batches on cuda:  72%|███████▏  | 423/584 [00:02<00:01, 160.17batch/s]

Training batches on cuda:  75%|███████▌  | 440/584 [00:02<00:00, 158.67batch/s]

Training batches on cuda:  78%|███████▊  | 456/584 [00:02<00:00, 157.71batch/s]

Training batches on cuda:  81%|████████  | 473/584 [00:02<00:00, 158.96batch/s]

Training batches on cuda:  84%|████████▍ | 490/584 [00:03<00:00, 160.72batch/s]

Training batches on cuda:  87%|████████▋ | 509/584 [00:03<00:00, 166.29batch/s]

Training batches on cuda:  90%|█████████ | 527/584 [00:03<00:00, 168.79batch/s]

Training batches on cuda:  93%|█████████▎| 545/584 [00:03<00:00, 171.06batch/s]

Training batches on cuda:  96%|█████████▋| 563/584 [00:03<00:00, 169.97batch/s]

Training batches on cuda:  99%|█████████▉| 581/584 [00:03<00:00, 157.42batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:38,  4.24s/epoch, loss=0.00247, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:32,  4.11s/epoch, loss=0.00247, prev_loss=nan]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 9/584 [00:00<00:06, 89.51batch/s]

Training batches on cuda:   5%|▍         | 27/584 [00:00<00:05, 104.59batch/s]

Training batches on cuda:   8%|▊         | 44/584 [00:00<00:04, 117.97batch/s]

Training batches on cuda:  10%|█         | 61/584 [00:00<00:04, 129.72batch/s]

Training batches on cuda:  13%|█▎        | 78/584 [00:00<00:03, 139.46batch/s]

Training batches on cuda:  16%|█▋        | 95/584 [00:00<00:03, 145.85batch/s]

Training batches on cuda:  19%|█▉        | 112/584 [00:00<00:03, 151.99batch/s]

Training batches on cuda:  22%|██▏       | 130/584 [00:00<00:02, 157.12batch/s]

Training batches on cuda:  25%|██▌       | 147/584 [00:00<00:02, 160.47batch/s]

Training batches on cuda:  28%|██▊       | 164/584 [00:01<00:02, 160.35batch/s]

Training batches on cuda:  31%|███       | 181/584 [00:01<00:02, 161.99batch/s]

Training batches on cuda:  34%|███▍      | 198/584 [00:01<00:02, 164.19batch/s]

Training batches on cuda:  37%|███▋      | 215/584 [00:01<00:02, 165.53batch/s]

Training batches on cuda:  40%|███▉      | 232/584 [00:01<00:02, 166.28batch/s]

Training batches on cuda:  43%|████▎     | 250/584 [00:01<00:01, 168.20batch/s]

Training batches on cuda:  46%|████▌     | 267/584 [00:01<00:01, 166.72batch/s]

Training batches on cuda:  49%|████▉     | 285/584 [00:01<00:01, 167.83batch/s]

Training batches on cuda:  52%|█████▏    | 302/584 [00:01<00:01, 167.83batch/s]

Training batches on cuda:  55%|█████▍    | 320/584 [00:01<00:01, 169.29batch/s]

Training batches on cuda:  58%|█████▊    | 337/584 [00:02<00:01, 168.64batch/s]

Training batches on cuda:  61%|██████    | 355/584 [00:02<00:01, 169.84batch/s]

Training batches on cuda:  64%|██████▎   | 372/584 [00:02<00:01, 151.59batch/s]

Training batches on cuda:  67%|██████▋   | 390/584 [00:02<00:01, 157.50batch/s]

Training batches on cuda:  70%|██████▉   | 408/584 [00:02<00:01, 161.65batch/s]

Training batches on cuda:  73%|███████▎  | 426/584 [00:02<00:00, 164.06batch/s]

Training batches on cuda:  76%|███████▌  | 444/584 [00:02<00:00, 166.66batch/s]

Training batches on cuda:  79%|███████▉  | 461/584 [00:02<00:00, 167.27batch/s]

Training batches on cuda:  82%|████████▏ | 478/584 [00:02<00:00, 168.01batch/s]

Training batches on cuda:  85%|████████▍ | 495/584 [00:03<00:00, 167.37batch/s]

Training batches on cuda:  88%|████████▊ | 512/584 [00:03<00:00, 166.31batch/s]

Training batches on cuda:  91%|█████████ | 529/584 [00:03<00:00, 164.86batch/s]

Training batches on cuda:  93%|█████████▎| 546/584 [00:03<00:00, 164.76batch/s]

Training batches on cuda:  96%|█████████▋| 563/584 [00:03<00:00, 164.84batch/s]

Training batches on cuda:  99%|█████████▉| 580/584 [00:03<00:00, 165.71batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:32,  4.11s/epoch, loss=0.00247, prev_loss=0.00247]
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:27,  4.00s/epoch, loss=0.00247, prev_loss=0.00247]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 13/584 [00:00<00:04, 123.60batch/s]

Training batches on cuda:   5%|▌         | 30/584 [00:00<00:04, 134.22batch/s]

Training batches on cuda:   8%|▊         | 47/584 [00:00<00:03, 142.17batch/s]

Training batches on cuda:  11%|█         | 64/584 [00:00<00:03, 148.88batch/s]

Training batches on cuda:  14%|█▎        | 80/584 [00:00<00:03, 150.87batch/s]

Training batches on cuda:  17%|█▋        | 97/584 [00:00<00:03, 154.51batch/s]

Training batches on cuda:  20%|█▉        | 114/584 [00:00<00:03, 156.30batch/s]

Training batches on cuda:  22%|██▏       | 130/584 [00:00<00:02, 157.28batch/s]

Training batches on cuda:  25%|██▌       | 147/584 [00:00<00:02, 160.14batch/s]

Training batches on cuda:  28%|██▊       | 164/584 [00:01<00:02, 161.15batch/s]

Training batches on cuda:  31%|███       | 181/584 [00:01<00:02, 159.21batch/s]

Training batches on cuda:  34%|███▎      | 197/584 [00:01<00:02, 158.33batch/s]

Training batches on cuda:  37%|███▋      | 214/584 [00:01<00:02, 159.84batch/s]

Training batches on cuda:  40%|███▉      | 231/584 [00:01<00:02, 161.28batch/s]

Training batches on cuda:  42%|████▏     | 248/584 [00:01<00:02, 161.52batch/s]

Training batches on cuda:  45%|████▌     | 265/584 [00:01<00:01, 161.70batch/s]

Training batches on cuda:  48%|████▊     | 282/584 [00:01<00:01, 159.17batch/s]

Training batches on cuda:  51%|█████     | 298/584 [00:01<00:01, 151.21batch/s]

Training batches on cuda:  54%|█████▍    | 314/584 [00:01<00:01, 148.49batch/s]

Training batches on cuda:  56%|█████▋    | 329/584 [00:02<00:01, 146.99batch/s]

Training batches on cuda:  59%|█████▉    | 344/584 [00:02<00:01, 145.39batch/s]

Training batches on cuda:  62%|██████▏   | 361/584 [00:02<00:01, 149.94batch/s]

Training batches on cuda:  65%|██████▍   | 378/584 [00:02<00:01, 153.60batch/s]

Training batches on cuda:  68%|██████▊   | 395/584 [00:02<00:01, 156.72batch/s]

Training batches on cuda:  71%|███████   | 412/584 [00:02<00:01, 158.07batch/s]

Training batches on cuda:  73%|███████▎  | 428/584 [00:02<00:00, 156.46batch/s]

Training batches on cuda:  76%|███████▌  | 445/584 [00:02<00:00, 157.43batch/s]

Training batches on cuda:  79%|███████▉  | 462/584 [00:02<00:00, 158.38batch/s]

Training batches on cuda:  82%|████████▏ | 479/584 [00:03<00:00, 158.84batch/s]

Training batches on cuda:  85%|████████▍ | 495/584 [00:03<00:00, 156.90batch/s]

Training batches on cuda:  88%|████████▊ | 511/584 [00:03<00:00, 140.76batch/s]

Training batches on cuda:  90%|█████████ | 526/584 [00:03<00:00, 139.08batch/s]

Training batches on cuda:  93%|█████████▎| 541/584 [00:03<00:00, 127.16batch/s]

Training batches on cuda:  95%|█████████▌| 555/584 [00:03<00:00, 110.60batch/s]

Training batches on cuda:  97%|█████████▋| 568/584 [00:03<00:00, 114.29batch/s]

Training batches on cuda:  99%|█████████▉| 580/584 [00:03<00:00, 114.53batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:16<00:27,  4.00s/epoch, loss=0.00246, prev_loss=0.00247]
Training epochs on cuda:  40%|████      | 4/10 [00:16<00:24,  4.09s/epoch, loss=0.00246, prev_loss=0.00247]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 10/584 [00:00<00:05, 99.79batch/s]

Training batches on cuda:   4%|▍         | 24/584 [00:00<00:05, 108.73batch/s]

Training batches on cuda:   7%|▋         | 38/584 [00:00<00:04, 114.79batch/s]

Training batches on cuda:   9%|▉         | 52/584 [00:00<00:04, 119.39batch/s]

Training batches on cuda:  11%|█▏        | 67/584 [00:00<00:04, 125.63batch/s]

Training batches on cuda:  14%|█▍        | 83/584 [00:00<00:03, 133.07batch/s]

Training batches on cuda:  17%|█▋        | 99/584 [00:00<00:03, 139.18batch/s]

Training batches on cuda:  20%|█▉        | 115/584 [00:00<00:03, 144.66batch/s]

Training batches on cuda:  22%|██▏       | 130/584 [00:00<00:03, 144.08batch/s]

Training batches on cuda:  25%|██▍       | 145/584 [00:01<00:03, 143.78batch/s]

Training batches on cuda:  27%|██▋       | 160/584 [00:01<00:02, 143.36batch/s]

Training batches on cuda:  30%|███       | 177/584 [00:01<00:02, 148.56batch/s]

Training batches on cuda:  33%|███▎      | 192/584 [00:01<00:02, 147.96batch/s]

Training batches on cuda:  35%|███▌      | 207/584 [00:01<00:02, 148.54batch/s]

Training batches on cuda:  38%|███▊      | 223/584 [00:01<00:02, 151.54batch/s]

Training batches on cuda:  41%|████      | 239/584 [00:01<00:02, 153.73batch/s]

Training batches on cuda:  44%|████▍     | 256/584 [00:01<00:02, 157.71batch/s]

Training batches on cuda:  47%|████▋     | 272/584 [00:01<00:01, 158.27batch/s]

Training batches on cuda:  49%|████▉     | 289/584 [00:01<00:01, 159.11batch/s]

Training batches on cuda:  52%|█████▏    | 306/584 [00:02<00:01, 160.19batch/s]

Training batches on cuda:  55%|█████▌    | 323/584 [00:02<00:01, 157.56batch/s]

Training batches on cuda:  58%|█████▊    | 340/584 [00:02<00:01, 160.45batch/s]

Training batches on cuda:  61%|██████    | 357/584 [00:02<00:01, 157.69batch/s]

Training batches on cuda:  64%|██████▍   | 373/584 [00:02<00:01, 157.16batch/s]

Training batches on cuda:  67%|██████▋   | 390/584 [00:02<00:01, 159.22batch/s]

Training batches on cuda:  70%|██████▉   | 406/584 [00:02<00:01, 158.90batch/s]

Training batches on cuda:  72%|███████▏  | 422/584 [00:02<00:01, 157.36batch/s]

Training batches on cuda:  75%|███████▌  | 438/584 [00:02<00:00, 157.85batch/s]

Training batches on cuda:  78%|███████▊  | 454/584 [00:03<00:00, 157.63batch/s]

Training batches on cuda:  80%|████████  | 470/584 [00:03<00:00, 157.89batch/s]

Training batches on cuda:  83%|████████▎ | 486/584 [00:03<00:00, 157.92batch/s]

Training batches on cuda:  86%|████████▌ | 502/584 [00:03<00:00, 157.12batch/s]

Training batches on cuda:  89%|████████▊ | 518/584 [00:03<00:00, 154.79batch/s]

Training batches on cuda:  92%|█████████▏| 535/584 [00:03<00:00, 156.58batch/s]

Training batches on cuda:  95%|█████████▍| 552/584 [00:03<00:00, 158.04batch/s]

Training batches on cuda:  98%|█████████▊| 570/584 [00:03<00:00, 161.77batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:24,  4.09s/epoch, loss=0.00241, prev_loss=0.00246]
Training epochs on cuda:  50%|█████     | 5/10 [00:20<00:20,  4.06s/epoch, loss=0.00241, prev_loss=0.00246]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 9/584 [00:00<00:06, 86.06batch/s]

Training batches on cuda:   4%|▍         | 24/584 [00:00<00:05, 97.44batch/s]

Training batches on cuda:   6%|▌         | 36/584 [00:00<00:05, 100.51batch/s]

Training batches on cuda:   8%|▊         | 44/584 [00:00<00:06, 88.26batch/s] 

Training batches on cuda:  10%|▉         | 58/584 [00:00<00:05, 99.08batch/s]

Training batches on cuda:  12%|█▏        | 70/584 [00:00<00:04, 103.47batch/s]

Training batches on cuda:  14%|█▍        | 82/584 [00:00<00:04, 104.98batch/s]

Training batches on cuda:  16%|█▌        | 94/584 [00:00<00:04, 107.74batch/s]

Training batches on cuda:  18%|█▊        | 105/584 [00:00<00:04, 104.37batch/s]

Training batches on cuda:  21%|██        | 121/584 [00:01<00:04, 115.37batch/s]

Training batches on cuda:  23%|██▎       | 137/584 [00:01<00:03, 123.24batch/s]

Training batches on cuda:  26%|██▌       | 150/584 [00:01<00:03, 119.79batch/s]

Training batches on cuda:  28%|██▊       | 163/584 [00:01<00:03, 116.63batch/s]

Training batches on cuda:  30%|███       | 177/584 [00:01<00:03, 121.27batch/s]

Training batches on cuda:  33%|███▎      | 190/584 [00:01<00:03, 122.89batch/s]

Training batches on cuda:  35%|███▍      | 203/584 [00:01<00:03, 118.05batch/s]

Training batches on cuda:  37%|███▋      | 215/584 [00:01<00:03, 116.83batch/s]

Training batches on cuda:  40%|███▉      | 231/584 [00:01<00:02, 126.68batch/s]

Training batches on cuda:  42%|████▏     | 246/584 [00:02<00:02, 131.18batch/s]

Training batches on cuda:  45%|████▍     | 261/584 [00:02<00:02, 135.78batch/s]

Training batches on cuda:  47%|████▋     | 275/584 [00:02<00:02, 130.74batch/s]

Training batches on cuda:  50%|████▉     | 290/584 [00:02<00:02, 135.85batch/s]

Training batches on cuda:  53%|█████▎    | 307/584 [00:02<00:01, 142.45batch/s]

Training batches on cuda:  55%|█████▌    | 323/584 [00:02<00:01, 146.00batch/s]

Training batches on cuda:  58%|█████▊    | 338/584 [00:02<00:01, 133.17batch/s]

Training batches on cuda:  60%|██████    | 352/584 [00:02<00:01, 122.15batch/s]

Training batches on cuda:  62%|██████▎   | 365/584 [00:02<00:01, 120.96batch/s]

Training batches on cuda:  65%|██████▍   | 378/584 [00:03<00:01, 114.33batch/s]

Training batches on cuda:  67%|██████▋   | 391/584 [00:03<00:01, 118.12batch/s]

Training batches on cuda:  69%|██████▉   | 405/584 [00:03<00:01, 122.68batch/s]

Training batches on cuda:  72%|███████▏  | 418/584 [00:03<00:01, 119.87batch/s]

Training batches on cuda:  74%|███████▍  | 434/584 [00:03<00:01, 127.36batch/s]

Training batches on cuda:  77%|███████▋  | 449/584 [00:03<00:01, 133.19batch/s]

Training batches on cuda:  79%|███████▉  | 463/584 [00:03<00:00, 130.76batch/s]

Training batches on cuda:  82%|████████▏ | 477/584 [00:03<00:00, 128.56batch/s]

Training batches on cuda:  84%|████████▍ | 490/584 [00:03<00:00, 122.06batch/s]

Training batches on cuda:  86%|████████▌ | 503/584 [00:04<00:00, 123.08batch/s]

Training batches on cuda:  88%|████████▊ | 516/584 [00:04<00:00, 123.70batch/s]

Training batches on cuda:  91%|█████████ | 529/584 [00:04<00:00, 123.95batch/s]

Training batches on cuda:  93%|█████████▎| 543/584 [00:04<00:00, 128.16batch/s]

Training batches on cuda:  95%|█████████▌| 556/584 [00:04<00:00, 128.16batch/s]

Training batches on cuda:  97%|█████████▋| 569/584 [00:04<00:00, 118.55batch/s]

Training batches on cuda: 100%|█████████▉| 582/584 [00:04<00:00, 109.44batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:25<00:20,  4.06s/epoch, loss=0.00235, prev_loss=0.00241]
Training epochs on cuda:  60%|██████    | 6/10 [00:25<00:17,  4.37s/epoch, loss=0.00235, prev_loss=0.00241]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/584 [00:00<00:05, 104.89batch/s]

Training batches on cuda:   4%|▍         | 26/584 [00:00<00:04, 113.75batch/s]

Training batches on cuda:   7%|▋         | 38/584 [00:00<00:04, 114.83batch/s]

Training batches on cuda:   8%|▊         | 47/584 [00:00<00:05, 100.85batch/s]

Training batches on cuda:  10%|▉         | 58/584 [00:00<00:05, 101.32batch/s]

Training batches on cuda:  12%|█▏        | 68/584 [00:00<00:05, 97.46batch/s] 

Training batches on cuda:  14%|█▎        | 80/584 [00:00<00:04, 102.25batch/s]

Training batches on cuda:  16%|█▌        | 91/584 [00:00<00:04, 103.74batch/s]

Training batches on cuda:  18%|█▊        | 104/584 [00:00<00:04, 109.01batch/s]

Training batches on cuda:  20%|█▉        | 115/584 [00:01<00:04, 108.07batch/s]

Training batches on cuda:  22%|██▏       | 126/584 [00:01<00:04, 99.36batch/s] 

Training batches on cuda:  23%|██▎       | 136/584 [00:01<00:04, 96.10batch/s]

Training batches on cuda:  25%|██▌       | 146/584 [00:01<00:04, 96.44batch/s]

Training batches on cuda:  27%|██▋       | 156/584 [00:01<00:04, 95.31batch/s]

Training batches on cuda:  29%|██▊       | 167/584 [00:01<00:04, 98.07batch/s]

Training batches on cuda:  30%|███       | 177/584 [00:01<00:04, 97.56batch/s]

Training batches on cuda:  32%|███▏      | 188/584 [00:01<00:03, 100.61batch/s]

Training batches on cuda:  34%|███▍      | 199/584 [00:01<00:03, 98.73batch/s] 

Training batches on cuda:  36%|███▌      | 209/584 [00:02<00:03, 98.73batch/s]

Training batches on cuda:  38%|███▊      | 220/584 [00:02<00:03, 99.98batch/s]

Training batches on cuda:  40%|███▉      | 233/584 [00:02<00:03, 107.12batch/s]

Training batches on cuda:  42%|████▏     | 244/584 [00:02<00:03, 105.65batch/s]

Training batches on cuda:  44%|████▍     | 258/584 [00:02<00:02, 112.24batch/s]

Training batches on cuda:  47%|████▋     | 274/584 [00:02<00:02, 122.56batch/s]

Training batches on cuda:  49%|████▉     | 289/584 [00:02<00:02, 128.71batch/s]

Training batches on cuda:  52%|█████▏    | 303/584 [00:02<00:02, 125.24batch/s]

Training batches on cuda:  54%|█████▍    | 317/584 [00:02<00:02, 128.67batch/s]

Training batches on cuda:  57%|█████▋    | 331/584 [00:03<00:02, 118.80batch/s]

Training batches on cuda:  59%|█████▉    | 344/584 [00:03<00:02, 112.69batch/s]

Training batches on cuda:  61%|██████▏   | 358/584 [00:03<00:01, 118.80batch/s]

Training batches on cuda:  64%|██████▎   | 371/584 [00:03<00:01, 121.16batch/s]

Training batches on cuda:  66%|██████▌   | 384/584 [00:03<00:01, 123.47batch/s]

Training batches on cuda:  68%|██████▊   | 397/584 [00:03<00:01, 121.46batch/s]

Training batches on cuda:  71%|███████   | 412/584 [00:03<00:01, 128.14batch/s]

Training batches on cuda:  73%|███████▎  | 425/584 [00:03<00:01, 119.53batch/s]

Training batches on cuda:  75%|███████▌  | 438/584 [00:03<00:01, 117.68batch/s]

Training batches on cuda:  77%|███████▋  | 452/584 [00:04<00:01, 120.89batch/s]

Training batches on cuda:  80%|███████▉  | 465/584 [00:04<00:01, 118.76batch/s]

Training batches on cuda:  83%|████████▎ | 482/584 [00:04<00:00, 130.22batch/s]

Training batches on cuda:  85%|████████▍ | 496/584 [00:04<00:00, 126.05batch/s]

Training batches on cuda:  87%|████████▋ | 509/584 [00:04<00:00, 116.40batch/s]

Training batches on cuda:  89%|████████▉ | 522/584 [00:04<00:00, 110.89batch/s]

Training batches on cuda:  92%|█████████▏| 536/584 [00:04<00:00, 117.93batch/s]

Training batches on cuda:  95%|█████████▍| 553/584 [00:04<00:00, 128.21batch/s]

Training batches on cuda:  97%|█████████▋| 568/584 [00:04<00:00, 132.18batch/s]

Training batches on cuda: 100%|█████████▉| 583/584 [00:05<00:00, 135.91batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:17,  4.37s/epoch, loss=0.00238, prev_loss=0.00235]
Training epochs on cuda:  70%|███████   | 7/10 [00:30<00:13,  4.64s/epoch, loss=0.00238, prev_loss=0.00235]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 7/584 [00:00<00:08, 69.77batch/s]

Training batches on cuda:   3%|▎         | 17/584 [00:00<00:07, 76.05batch/s]

Training batches on cuda:   5%|▌         | 30/584 [00:00<00:06, 85.04batch/s]

Training batches on cuda:   7%|▋         | 41/584 [00:00<00:06, 89.78batch/s]

Training batches on cuda:   9%|▉         | 53/584 [00:00<00:05, 96.98batch/s]

Training batches on cuda:  11%|█         | 65/584 [00:00<00:05, 101.92batch/s]

Training batches on cuda:  13%|█▎        | 77/584 [00:00<00:04, 106.17batch/s]

Training batches on cuda:  15%|█▌        | 88/584 [00:00<00:04, 105.26batch/s]

Training batches on cuda:  18%|█▊        | 103/584 [00:00<00:04, 114.24batch/s]

Training batches on cuda:  20%|█▉        | 116/584 [00:01<00:04, 116.67batch/s]

Training batches on cuda:  22%|██▏       | 130/584 [00:01<00:03, 122.49batch/s]

Training batches on cuda:  25%|██▌       | 147/584 [00:01<00:03, 131.47batch/s]

Training batches on cuda:  28%|██▊       | 161/584 [00:01<00:03, 132.88batch/s]

Training batches on cuda:  30%|███       | 177/584 [00:01<00:02, 137.78batch/s]

Training batches on cuda:  33%|███▎      | 191/584 [00:01<00:02, 136.37batch/s]

Training batches on cuda:  35%|███▌      | 205/584 [00:01<00:02, 136.91batch/s]

Training batches on cuda:  38%|███▊      | 220/584 [00:01<00:02, 140.48batch/s]

Training batches on cuda:  40%|████      | 235/584 [00:01<00:02, 137.57batch/s]

Training batches on cuda:  43%|████▎     | 249/584 [00:01<00:02, 135.40batch/s]

Training batches on cuda:  45%|████▌     | 263/584 [00:02<00:02, 135.60batch/s]

Training batches on cuda:  48%|████▊     | 279/584 [00:02<00:02, 140.07batch/s]

Training batches on cuda:  51%|█████     | 295/584 [00:02<00:01, 144.92batch/s]

Training batches on cuda:  53%|█████▎    | 310/584 [00:02<00:01, 140.09batch/s]

Training batches on cuda:  56%|█████▌    | 325/584 [00:02<00:01, 133.67batch/s]

Training batches on cuda:  58%|█████▊    | 339/584 [00:02<00:01, 130.07batch/s]

Training batches on cuda:  60%|██████    | 353/584 [00:02<00:02, 112.38batch/s]

Training batches on cuda:  63%|██████▎   | 366/584 [00:02<00:01, 117.11batch/s]

Training batches on cuda:  65%|██████▍   | 379/584 [00:03<00:01, 117.55batch/s]

Training batches on cuda:  67%|██████▋   | 393/584 [00:03<00:01, 121.86batch/s]

Training batches on cuda:  70%|██████▉   | 406/584 [00:03<00:01, 116.82batch/s]

Training batches on cuda:  72%|███████▏  | 418/584 [00:03<00:01, 111.98batch/s]

Training batches on cuda:  74%|███████▍  | 431/584 [00:03<00:01, 114.72batch/s]

Training batches on cuda:  76%|███████▌  | 444/584 [00:03<00:01, 117.33batch/s]

Training batches on cuda:  78%|███████▊  | 456/584 [00:03<00:01, 114.47batch/s]

Training batches on cuda:  80%|████████  | 469/584 [00:03<00:00, 117.46batch/s]

Training batches on cuda:  82%|████████▏ | 481/584 [00:03<00:00, 104.58batch/s]

Training batches on cuda:  84%|████████▍ | 492/584 [00:04<00:00, 105.71batch/s]

Training batches on cuda:  86%|████████▋ | 505/584 [00:04<00:00, 110.15batch/s]

Training batches on cuda:  89%|████████▊ | 517/584 [00:04<00:00, 109.10batch/s]

Training batches on cuda:  91%|█████████ | 529/584 [00:04<00:00, 102.99batch/s]

Training batches on cuda:  93%|█████████▎| 541/584 [00:04<00:00, 106.37batch/s]

Training batches on cuda:  95%|█████████▍| 552/584 [00:04<00:00, 106.19batch/s]

Training batches on cuda:  97%|█████████▋| 564/584 [00:04<00:00, 109.00batch/s]

Training batches on cuda:  99%|█████████▉| 577/584 [00:04<00:00, 112.56batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:13,  4.64s/epoch, loss=0.0024, prev_loss=0.00238] 
Training epochs on cuda:  80%|████████  | 8/10 [00:35<00:09,  4.79s/epoch, loss=0.0024, prev_loss=0.00238]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 6/584 [00:00<00:09, 58.61batch/s]

Training batches on cuda:   3%|▎         | 19/584 [00:00<00:08, 69.68batch/s]

Training batches on cuda:   5%|▍         | 29/584 [00:00<00:07, 76.60batch/s]

Training batches on cuda:   8%|▊         | 44/584 [00:00<00:06, 88.75batch/s]

Training batches on cuda:  10%|▉         | 58/584 [00:00<00:05, 99.69batch/s]

Training batches on cuda:  12%|█▏        | 71/584 [00:00<00:04, 107.13batch/s]

Training batches on cuda:  15%|█▍        | 85/584 [00:00<00:04, 113.42batch/s]

Training batches on cuda:  17%|█▋        | 97/584 [00:00<00:04, 111.08batch/s]

Training batches on cuda:  19%|█▉        | 110/584 [00:00<00:04, 115.16batch/s]

Training batches on cuda:  21%|██        | 123/584 [00:01<00:03, 119.16batch/s]

Training batches on cuda:  23%|██▎       | 136/584 [00:01<00:03, 121.76batch/s]

Training batches on cuda:  26%|██▌       | 150/584 [00:01<00:03, 126.00batch/s]

Training batches on cuda:  28%|██▊       | 163/584 [00:01<00:03, 126.61batch/s]

Training batches on cuda:  30%|███       | 177/584 [00:01<00:03, 130.01batch/s]

Training batches on cuda:  33%|███▎      | 191/584 [00:01<00:02, 132.66batch/s]

Training batches on cuda:  35%|███▌      | 205/584 [00:01<00:02, 131.25batch/s]

Training batches on cuda:  38%|███▊      | 219/584 [00:01<00:03, 119.48batch/s]

Training batches on cuda:  40%|███▉      | 233/584 [00:01<00:02, 123.27batch/s]

Training batches on cuda:  42%|████▏     | 246/584 [00:02<00:02, 122.54batch/s]

Training batches on cuda:  45%|████▍     | 260/584 [00:02<00:02, 127.19batch/s]

Training batches on cuda:  47%|████▋     | 273/584 [00:02<00:02, 122.83batch/s]

Training batches on cuda:  49%|████▉     | 286/584 [00:02<00:02, 123.20batch/s]

Training batches on cuda:  51%|█████▏    | 300/584 [00:02<00:02, 125.49batch/s]

Training batches on cuda:  54%|█████▍    | 314/584 [00:02<00:02, 126.90batch/s]

Training batches on cuda:  56%|█████▋    | 329/584 [00:02<00:01, 130.76batch/s]

Training batches on cuda:  59%|█████▊    | 343/584 [00:02<00:01, 127.35batch/s]

Training batches on cuda:  61%|██████    | 356/584 [00:02<00:01, 127.93batch/s]

Training batches on cuda:  63%|██████▎   | 370/584 [00:02<00:01, 130.70batch/s]

Training batches on cuda:  66%|██████▌   | 384/584 [00:03<00:01, 130.78batch/s]

Training batches on cuda:  68%|██████▊   | 398/584 [00:03<00:01, 118.11batch/s]

Training batches on cuda:  70%|███████   | 411/584 [00:03<00:01, 115.50batch/s]

Training batches on cuda:  72%|███████▏  | 423/584 [00:03<00:01, 111.11batch/s]

Training batches on cuda:  75%|███████▌  | 438/584 [00:03<00:01, 118.41batch/s]

Training batches on cuda:  77%|███████▋  | 451/584 [00:03<00:01, 120.48batch/s]

Training batches on cuda:  79%|███████▉  | 464/584 [00:03<00:01, 113.56batch/s]

Training batches on cuda:  82%|████████▏ | 476/584 [00:03<00:00, 111.65batch/s]

Training batches on cuda:  84%|████████▎ | 489/584 [00:04<00:00, 113.97batch/s]

Training batches on cuda:  86%|████████▌ | 502/584 [00:04<00:00, 117.71batch/s]

Training batches on cuda:  88%|████████▊ | 514/584 [00:04<00:00, 116.80batch/s]

Training batches on cuda:  90%|█████████ | 526/584 [00:04<00:00, 109.96batch/s]

Training batches on cuda:  92%|█████████▏| 538/584 [00:04<00:00, 105.36batch/s]

Training batches on cuda:  95%|█████████▍| 552/584 [00:04<00:00, 113.26batch/s]

Training batches on cuda:  97%|█████████▋| 564/584 [00:04<00:00, 108.70batch/s]

Training batches on cuda:  99%|█████████▊| 576/584 [00:04<00:00, 104.70batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:40<00:09,  4.79s/epoch, loss=0.00238, prev_loss=0.0024]
Training epochs on cuda:  90%|█████████ | 9/10 [00:40<00:04,  4.92s/epoch, loss=0.00238, prev_loss=0.0024]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 9/584 [00:00<00:06, 86.98batch/s]

Training batches on cuda:   4%|▎         | 21/584 [00:00<00:05, 94.44batch/s]

Training batches on cuda:   5%|▌         | 32/584 [00:00<00:05, 96.30batch/s]

Training batches on cuda:   7%|▋         | 43/584 [00:00<00:05, 98.53batch/s]

Training batches on cuda:   9%|▉         | 55/584 [00:00<00:05, 102.91batch/s]

Training batches on cuda:  11%|█         | 64/584 [00:00<00:05, 95.81batch/s] 

Training batches on cuda:  12%|█▎        | 73/584 [00:00<00:05, 93.17batch/s]

Training batches on cuda:  15%|█▌        | 88/584 [00:00<00:04, 103.56batch/s]

Training batches on cuda:  17%|█▋        | 101/584 [00:00<00:04, 106.47batch/s]

Training batches on cuda:  19%|█▉        | 112/584 [00:01<00:04, 106.80batch/s]

Training batches on cuda:  22%|██▏       | 128/584 [00:01<00:03, 117.65batch/s]

Training batches on cuda:  25%|██▍       | 145/584 [00:01<00:03, 128.19batch/s]

Training batches on cuda:  28%|██▊       | 161/584 [00:01<00:03, 135.79batch/s]

Training batches on cuda:  30%|███       | 176/584 [00:01<00:02, 137.15batch/s]

Training batches on cuda:  33%|███▎      | 191/584 [00:01<00:02, 139.79batch/s]

Training batches on cuda:  35%|███▌      | 206/584 [00:01<00:02, 140.59batch/s]

Training batches on cuda:  38%|███▊      | 221/584 [00:01<00:02, 136.50batch/s]

Training batches on cuda:  40%|████      | 235/584 [00:01<00:02, 134.38batch/s]

Training batches on cuda:  43%|████▎     | 249/584 [00:02<00:02, 133.67batch/s]

Training batches on cuda:  45%|████▌     | 263/584 [00:02<00:02, 134.77batch/s]

Training batches on cuda:  47%|████▋     | 277/584 [00:02<00:02, 134.81batch/s]

Training batches on cuda:  50%|████▉     | 291/584 [00:02<00:02, 121.03batch/s]

Training batches on cuda:  52%|█████▏    | 304/584 [00:02<00:02, 119.08batch/s]

Training batches on cuda:  54%|█████▍    | 317/584 [00:02<00:02, 120.61batch/s]

Training batches on cuda:  57%|█████▋    | 330/584 [00:02<00:02, 109.26batch/s]

Training batches on cuda:  59%|█████▊    | 342/584 [00:02<00:02, 109.47batch/s]

Training batches on cuda:  61%|██████    | 354/584 [00:02<00:02, 106.91batch/s]

Training batches on cuda:  62%|██████▎   | 365/584 [00:03<00:02, 106.36batch/s]

Training batches on cuda:  65%|██████▌   | 381/584 [00:03<00:01, 117.02batch/s]

Training batches on cuda:  67%|██████▋   | 394/584 [00:03<00:01, 117.11batch/s]

Training batches on cuda:  70%|██████▉   | 406/584 [00:03<00:01, 114.63batch/s]

Training batches on cuda:  72%|███████▏  | 421/584 [00:03<00:01, 122.09batch/s]

Training batches on cuda:  74%|███████▍  | 434/584 [00:03<00:01, 122.75batch/s]

Training batches on cuda:  77%|███████▋  | 451/584 [00:03<00:01, 132.51batch/s]

Training batches on cuda:  80%|████████  | 468/584 [00:03<00:00, 140.94batch/s]

Training batches on cuda:  83%|████████▎ | 486/584 [00:03<00:00, 149.56batch/s]

Training batches on cuda:  86%|████████▋ | 504/584 [00:04<00:00, 155.73batch/s]

Training batches on cuda:  89%|████████▉ | 521/584 [00:04<00:00, 158.17batch/s]

Training batches on cuda:  92%|█████████▏| 538/584 [00:04<00:00, 159.89batch/s]

Training batches on cuda:  95%|█████████▌| 555/584 [00:04<00:00, 159.78batch/s]

Training batches on cuda:  98%|█████████▊| 572/584 [00:04<00:00, 161.99batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:45<00:04,  4.92s/epoch, loss=0.00237, prev_loss=0.00238]2021-09-16:17:24:44,446 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:24:45,555 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:25:00,413 INFO     [evaluator.py:657] Evaluation took 14.86s seconds
2021-09-16:17:25:00,556 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  9.68s/epoch, loss=0.00237, prev_loss=0.00238]
Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  6.16s/epoch, loss=0.00237, prev_loss=0.00238]
2021-09-16:17:25:00,556 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp5so3dixq'
2021-09-16:17:25:00,616 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5so3dixq' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.17ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.30ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.40ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.47ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.67ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:05<00:08, 2.67ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.67ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.67ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.67ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.67ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.67ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.67ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.67ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.67ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.67ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.66ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.66ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.66ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.66ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.66ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:10<00:03, 2.66ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.67ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.66ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:11<00:02, 2.66ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.67ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.67ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.67ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.67ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.67ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.66ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.66ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.67ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.66ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.67ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
2021-09-16:17:25:15,331 INFO     [evaluator.py:657] Evaluation took 14.70s seconds
[I 2021-09-16 17:25:15,336] Trial 1 finished with value: 0.29090778411927365 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.021564451036940394, 'optimizer.lr': 0.02415694040846957, 'negative_sampler.num_negs_per_pos': 7, 'training.batch_size': 532}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:25:15,338 WARNING  [api.py:823] No random seed is specified. Setting to 418971438.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 8/203 [00:00<00:02, 71.86batch/s]

Training batches on cuda:  10%|▉         | 20/203 [00:00<00:02, 80.63batch/s]

Training batches on cuda:  15%|█▌        | 31/203 [00:00<00:02, 85.08batch/s]

Training batches on cuda:  20%|██        | 41/203 [00:00<00:01, 87.81batch/s]

Training batches on cuda:  26%|██▌       | 52/203 [00:00<00:01, 92.06batch/s]

Training batches on cuda:  30%|███       | 61/203 [00:00<00:01, 90.95batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:00<00:01, 91.01batch/s]

Training batches on cuda:  40%|███▉      | 81/203 [00:00<00:01, 93.32batch/s]

Training batches on cuda:  45%|████▍     | 91/203 [00:00<00:01, 94.18batch/s]

Training batches on cuda:  50%|████▉     | 101/203 [00:01<00:01, 93.41batch/s]

Training batches on cuda:  55%|█████▍    | 111/203 [00:01<00:00, 94.33batch/s]

Training batches on cuda:  60%|█████▉    | 121/203 [00:01<00:00, 94.78batch/s]

Training batches on cuda:  65%|██████▍   | 131/203 [00:01<00:00, 94.76batch/s]

Training batches on cuda:  69%|██████▉   | 141/203 [00:01<00:00, 95.30batch/s]

Training batches on cuda:  74%|███████▍  | 151/203 [00:01<00:00, 95.57batch/s]

Training batches on cuda:  80%|███████▉  | 162/203 [00:01<00:00, 98.91batch/s]

Training batches on cuda:  85%|████████▌ | 173/203 [00:01<00:00, 101.12batch/s]

Training batches on cuda:  91%|█████████ | 184/203 [00:01<00:00, 100.01batch/s]

Training batches on cuda:  97%|█████████▋| 196/203 [00:02<00:00, 103.12batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.00143, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:20,  2.32s/epoch, loss=0.00143, prev_loss=nan]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 8/203 [00:00<00:02, 76.57batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:02, 83.66batch/s]

Training batches on cuda:  14%|█▍        | 29/203 [00:00<00:01, 87.26batch/s]

Training batches on cuda:  19%|█▉        | 39/203 [00:00<00:01, 89.87batch/s]

Training batches on cuda:  24%|██▍       | 49/203 [00:00<00:01, 92.29batch/s]

Training batches on cuda:  29%|██▉       | 59/203 [00:00<00:01, 94.01batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:00<00:01, 98.49batch/s]

Training batches on cuda:  40%|███▉      | 81/203 [00:00<00:01, 91.52batch/s]

Training batches on cuda:  45%|████▌     | 92/203 [00:00<00:01, 96.02batch/s]

Training batches on cuda:  51%|█████     | 104/203 [00:01<00:00, 100.78batch/s]

Training batches on cuda:  57%|█████▋    | 116/203 [00:01<00:00, 103.40batch/s]

Training batches on cuda:  63%|██████▎   | 127/203 [00:01<00:00, 105.25batch/s]

Training batches on cuda:  68%|██████▊   | 139/203 [00:01<00:00, 106.75batch/s]

Training batches on cuda:  74%|███████▍  | 151/203 [00:01<00:00, 109.18batch/s]

Training batches on cuda:  80%|████████  | 163/203 [00:01<00:00, 109.38batch/s]

Training batches on cuda:  86%|████████▌ | 175/203 [00:01<00:00, 109.57batch/s]

Training batches on cuda:  92%|█████████▏| 186/203 [00:01<00:00, 108.76batch/s]

Training batches on cuda:  97%|█████████▋| 197/203 [00:01<00:00, 108.30batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:20,  2.32s/epoch, loss=0.0014, prev_loss=nan] 
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:18,  2.27s/epoch, loss=0.0014, prev_loss=nan]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:03, 55.02batch/s]

Training batches on cuda:   7%|▋         | 15/203 [00:00<00:03, 61.30batch/s]

Training batches on cuda:  12%|█▏        | 24/203 [00:00<00:02, 66.66batch/s]

Training batches on cuda:  17%|█▋        | 34/203 [00:00<00:02, 72.47batch/s]

Training batches on cuda:  21%|██        | 42/203 [00:00<00:02, 74.06batch/s]

Training batches on cuda:  24%|██▍       | 49/203 [00:00<00:02, 69.90batch/s]

Training batches on cuda:  30%|██▉       | 60/203 [00:00<00:01, 78.13batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:00<00:01, 85.20batch/s]

Training batches on cuda:  40%|████      | 82/203 [00:00<00:01, 91.32batch/s]

Training batches on cuda:  46%|████▌     | 93/203 [00:01<00:01, 95.61batch/s]

Training batches on cuda:  51%|█████     | 104/203 [00:01<00:01, 98.05batch/s]

Training batches on cuda:  57%|█████▋    | 115/203 [00:01<00:00, 100.77batch/s]

Training batches on cuda:  62%|██████▏   | 126/203 [00:01<00:00, 102.40batch/s]

Training batches on cuda:  67%|██████▋   | 137/203 [00:01<00:00, 103.59batch/s]

Training batches on cuda:  73%|███████▎  | 148/203 [00:01<00:00, 105.14batch/s]

Training batches on cuda:  79%|███████▉  | 160/203 [00:01<00:00, 106.77batch/s]

Training batches on cuda:  85%|████████▍ | 172/203 [00:01<00:00, 108.98batch/s]

Training batches on cuda:  91%|█████████ | 184/203 [00:01<00:00, 110.76batch/s]

Training batches on cuda:  97%|█████████▋| 196/203 [00:01<00:00, 111.36batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:18,  2.27s/epoch, loss=0.0014, prev_loss=0.0014]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:15,  2.27s/epoch, loss=0.0014, prev_loss=0.0014]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 8/203 [00:00<00:02, 77.71batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:02, 85.06batch/s]

Training batches on cuda:  15%|█▌        | 31/203 [00:00<00:01, 91.48batch/s]

Training batches on cuda:  21%|██        | 42/203 [00:00<00:01, 96.18batch/s]

Training batches on cuda:  26%|██▌       | 52/203 [00:00<00:01, 97.02batch/s]

Training batches on cuda:  30%|███       | 61/203 [00:00<00:01, 91.33batch/s]

Training batches on cuda:  36%|███▌      | 73/203 [00:00<00:01, 96.31batch/s]

Training batches on cuda:  41%|████▏     | 84/203 [00:00<00:01, 98.64batch/s]

Training batches on cuda:  47%|████▋     | 95/203 [00:00<00:01, 101.05batch/s]

Training batches on cuda:  52%|█████▏    | 106/203 [00:01<00:00, 102.87batch/s]

Training batches on cuda:  58%|█████▊    | 117/203 [00:01<00:00, 104.37batch/s]

Training batches on cuda:  63%|██████▎   | 128/203 [00:01<00:00, 105.41batch/s]

Training batches on cuda:  68%|██████▊   | 139/203 [00:01<00:00, 105.95batch/s]

Training batches on cuda:  74%|███████▍  | 150/203 [00:01<00:00, 106.56batch/s]

Training batches on cuda:  79%|███████▉  | 161/203 [00:01<00:00, 107.40batch/s]

Training batches on cuda:  85%|████████▍ | 172/203 [00:01<00:00, 107.61batch/s]

Training batches on cuda:  90%|█████████ | 183/203 [00:01<00:00, 105.53batch/s]

Training batches on cuda:  96%|█████████▌| 194/203 [00:01<00:00, 104.76batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:15,  2.27s/epoch, loss=0.0014, prev_loss=0.0014]
Training epochs on cuda:  40%|████      | 4/10 [00:08<00:13,  2.23s/epoch, loss=0.0014, prev_loss=0.0014]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:03, 55.66batch/s]

Training batches on cuda:   8%|▊         | 17/203 [00:00<00:02, 65.13batch/s]

Training batches on cuda:  14%|█▍        | 28/203 [00:00<00:02, 73.68batch/s]

Training batches on cuda:  19%|█▉        | 39/203 [00:00<00:02, 81.04batch/s]

Training batches on cuda:  25%|██▍       | 50/203 [00:00<00:01, 87.67batch/s]

Training batches on cuda:  30%|███       | 61/203 [00:00<00:01, 92.95batch/s]

Training batches on cuda:  35%|███▌      | 72/203 [00:00<00:01, 96.85batch/s]

Training batches on cuda:  40%|████      | 82/203 [00:00<00:01, 97.28batch/s]

Training batches on cuda:  46%|████▌     | 93/203 [00:00<00:01, 100.74batch/s]

Training batches on cuda:  51%|█████     | 104/203 [00:01<00:00, 102.98batch/s]

Training batches on cuda:  57%|█████▋    | 116/203 [00:01<00:00, 105.59batch/s]

Training batches on cuda:  63%|██████▎   | 127/203 [00:01<00:00, 94.38batch/s] 

Training batches on cuda:  68%|██████▊   | 139/203 [00:01<00:00, 98.69batch/s]

Training batches on cuda:  74%|███████▍  | 150/203 [00:01<00:00, 101.49batch/s]

Training batches on cuda:  80%|███████▉  | 162/203 [00:01<00:00, 104.06batch/s]

Training batches on cuda:  85%|████████▌ | 173/203 [00:01<00:00, 105.05batch/s]

Training batches on cuda:  91%|█████████ | 184/203 [00:01<00:00, 105.94batch/s]

Training batches on cuda:  97%|█████████▋| 196/203 [00:01<00:00, 107.28batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:13,  2.23s/epoch, loss=0.0014, prev_loss=0.0014]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.21s/epoch, loss=0.0014, prev_loss=0.0014]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/203 [00:00<00:02, 65.81batch/s]

Training batches on cuda:   9%|▉         | 18/203 [00:00<00:02, 74.37batch/s]

Training batches on cuda:  14%|█▍        | 29/203 [00:00<00:02, 82.33batch/s]

Training batches on cuda:  20%|█▉        | 40/203 [00:00<00:01, 88.59batch/s]

Training batches on cuda:  25%|██▌       | 51/203 [00:00<00:01, 93.92batch/s]

Training batches on cuda:  31%|███       | 62/203 [00:00<00:01, 97.42batch/s]

Training batches on cuda:  36%|███▌      | 73/203 [00:00<00:01, 100.51batch/s]

Training batches on cuda:  41%|████▏     | 84/203 [00:00<00:01, 102.29batch/s]

Training batches on cuda:  47%|████▋     | 95/203 [00:00<00:01, 103.66batch/s]

Training batches on cuda:  52%|█████▏    | 106/203 [00:01<00:00, 105.01batch/s]

Training batches on cuda:  58%|█████▊    | 117/203 [00:01<00:00, 103.76batch/s]

Training batches on cuda:  63%|██████▎   | 128/203 [00:01<00:00, 101.14batch/s]

Training batches on cuda:  68%|██████▊   | 139/203 [00:01<00:00, 98.61batch/s] 

Training batches on cuda:  73%|███████▎  | 149/203 [00:01<00:00, 97.06batch/s]

Training batches on cuda:  78%|███████▊  | 159/203 [00:01<00:00, 95.93batch/s]

Training batches on cuda:  83%|████████▎ | 169/203 [00:01<00:00, 93.32batch/s]

Training batches on cuda:  88%|████████▊ | 179/203 [00:01<00:00, 92.73batch/s]

Training batches on cuda:  93%|█████████▎| 189/203 [00:01<00:00, 92.78batch/s]

Training batches on cuda:  98%|█████████▊| 199/203 [00:02<00:00, 92.98batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:11,  2.21s/epoch, loss=0.0014, prev_loss=0.0014]
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:08,  2.22s/epoch, loss=0.0014, prev_loss=0.0014]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/203 [00:00<00:05, 36.85batch/s]

Training batches on cuda:   7%|▋         | 14/203 [00:00<00:04, 44.98batch/s]

Training batches on cuda:  12%|█▏        | 24/203 [00:00<00:03, 53.42batch/s]

Training batches on cuda:  17%|█▋        | 34/203 [00:00<00:02, 61.34batch/s]

Training batches on cuda:  22%|██▏       | 44/203 [00:00<00:02, 68.58batch/s]

Training batches on cuda:  27%|██▋       | 54/203 [00:00<00:01, 74.88batch/s]

Training batches on cuda:  32%|███▏      | 64/203 [00:00<00:01, 79.35batch/s]

Training batches on cuda:  36%|███▋      | 74/203 [00:00<00:01, 83.08batch/s]

Training batches on cuda:  41%|████▏     | 84/203 [00:00<00:01, 86.36batch/s]

Training batches on cuda:  46%|████▋     | 94/203 [00:01<00:01, 87.72batch/s]

Training batches on cuda:  51%|█████     | 104/203 [00:01<00:01, 89.85batch/s]

Training batches on cuda:  56%|█████▌    | 114/203 [00:01<00:00, 91.71batch/s]

Training batches on cuda:  61%|██████    | 124/203 [00:01<00:00, 91.90batch/s]

Training batches on cuda:  66%|██████▌   | 134/203 [00:01<00:00, 92.65batch/s]

Training batches on cuda:  71%|███████   | 144/203 [00:01<00:00, 93.44batch/s]

Training batches on cuda:  76%|███████▋  | 155/203 [00:01<00:00, 97.29batch/s]

Training batches on cuda:  82%|████████▏ | 166/203 [00:01<00:00, 100.40batch/s]

Training batches on cuda:  87%|████████▋ | 177/203 [00:01<00:00, 90.06batch/s] 

Training batches on cuda:  93%|█████████▎| 188/203 [00:02<00:00, 95.14batch/s]

Training batches on cuda:  99%|█████████▊| 200/203 [00:02<00:00, 99.29batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:08,  2.22s/epoch, loss=0.0014, prev_loss=0.0014]
Training epochs on cuda:  70%|███████   | 7/10 [00:15<00:06,  2.26s/epoch, loss=0.0014, prev_loss=0.0014]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/203 [00:00<00:02, 66.24batch/s]

Training batches on cuda:   9%|▉         | 18/203 [00:00<00:02, 74.54batch/s]

Training batches on cuda:  14%|█▍        | 29/203 [00:00<00:02, 82.41batch/s]

Training batches on cuda:  20%|█▉        | 40/203 [00:00<00:01, 88.64batch/s]

Training batches on cuda:  26%|██▌       | 52/203 [00:00<00:01, 94.56batch/s]

Training batches on cuda:  31%|███       | 63/203 [00:00<00:01, 97.65batch/s]

Training batches on cuda:  36%|███▋      | 74/203 [00:00<00:01, 98.52batch/s]

Training batches on cuda:  42%|████▏     | 86/203 [00:00<00:01, 102.11batch/s]

Training batches on cuda:  48%|████▊     | 97/203 [00:00<00:01, 103.62batch/s]

Training batches on cuda:  53%|█████▎    | 108/203 [00:01<00:00, 104.42batch/s]

Training batches on cuda:  59%|█████▊    | 119/203 [00:01<00:00, 104.60batch/s]

Training batches on cuda:  64%|██████▍   | 130/203 [00:01<00:00, 104.45batch/s]

Training batches on cuda:  69%|██████▉   | 141/203 [00:01<00:00, 105.29batch/s]

Training batches on cuda:  75%|███████▍  | 152/203 [00:01<00:00, 105.42batch/s]

Training batches on cuda:  80%|████████  | 163/203 [00:01<00:00, 106.12batch/s]

Training batches on cuda:  86%|████████▌ | 174/203 [00:01<00:00, 104.60batch/s]

Training batches on cuda:  91%|█████████ | 185/203 [00:01<00:00, 105.38batch/s]

Training batches on cuda:  97%|█████████▋| 196/203 [00:01<00:00, 105.95batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:06,  2.26s/epoch, loss=0.0014, prev_loss=0.0014]
Training epochs on cuda:  80%|████████  | 8/10 [00:17<00:04,  2.22s/epoch, loss=0.0014, prev_loss=0.0014]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 8/203 [00:00<00:02, 75.33batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:02, 82.54batch/s]

Training batches on cuda:  15%|█▍        | 30/203 [00:00<00:01, 87.77batch/s]

Training batches on cuda:  19%|█▉        | 39/203 [00:00<00:01, 86.82batch/s]

Training batches on cuda:  24%|██▎       | 48/203 [00:00<00:01, 87.46batch/s]

Training batches on cuda:  29%|██▉       | 59/203 [00:00<00:01, 90.72batch/s]

Training batches on cuda:  34%|███▍      | 69/203 [00:00<00:01, 92.07batch/s]

Training batches on cuda:  39%|███▉      | 79/203 [00:00<00:01, 93.74batch/s]

Training batches on cuda:  44%|████▍     | 90/203 [00:00<00:01, 97.15batch/s]

Training batches on cuda:  50%|████▉     | 101/203 [00:01<00:01, 99.59batch/s]

Training batches on cuda:  55%|█████▌    | 112/203 [00:01<00:00, 102.28batch/s]

Training batches on cuda:  61%|██████    | 123/203 [00:01<00:00, 104.06batch/s]

Training batches on cuda:  66%|██████▌   | 134/203 [00:01<00:00, 104.92batch/s]

Training batches on cuda:  71%|███████▏  | 145/203 [00:01<00:00, 103.64batch/s]

Training batches on cuda:  77%|███████▋  | 156/203 [00:01<00:00, 100.01batch/s]

Training batches on cuda:  82%|████████▏ | 167/203 [00:01<00:00, 95.43batch/s] 

Training batches on cuda:  87%|████████▋ | 177/203 [00:01<00:00, 91.80batch/s]

Training batches on cuda:  92%|█████████▏| 187/203 [00:01<00:00, 93.03batch/s]

Training batches on cuda:  97%|█████████▋| 197/203 [00:02<00:00, 92.42batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:20<00:04,  2.22s/epoch, loss=0.0014, prev_loss=0.0014]
Training epochs on cuda:  90%|█████████ | 9/10 [00:20<00:02,  2.25s/epoch, loss=0.0014, prev_loss=0.0014]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/203 [00:00<00:04, 49.02batch/s]

Training batches on cuda:   7%|▋         | 14/203 [00:00<00:03, 56.74batch/s]

Training batches on cuda:  11%|█▏        | 23/203 [00:00<00:02, 62.14batch/s]

Training batches on cuda:  16%|█▌        | 32/203 [00:00<00:02, 68.36batch/s]

Training batches on cuda:  21%|██        | 42/203 [00:00<00:02, 74.52batch/s]

Training batches on cuda:  25%|██▌       | 51/203 [00:00<00:01, 77.85batch/s]

Training batches on cuda:  30%|██▉       | 60/203 [00:00<00:01, 80.28batch/s]

Training batches on cuda:  34%|███▍      | 69/203 [00:00<00:01, 82.65batch/s]

Training batches on cuda:  38%|███▊      | 78/203 [00:00<00:01, 84.72batch/s]

Training batches on cuda:  43%|████▎     | 88/203 [00:01<00:01, 88.54batch/s]

Training batches on cuda:  49%|████▉     | 99/203 [00:01<00:01, 92.79batch/s]

Training batches on cuda:  54%|█████▍    | 110/203 [00:01<00:00, 95.21batch/s]

Training batches on cuda:  59%|█████▉    | 120/203 [00:01<00:00, 95.65batch/s]

Training batches on cuda:  65%|██████▍   | 131/203 [00:01<00:00, 97.65batch/s]

Training batches on cuda:  69%|██████▉   | 141/203 [00:01<00:00, 95.04batch/s]

Training batches on cuda:  74%|███████▍  | 151/203 [00:01<00:00, 89.78batch/s]

Training batches on cuda:  79%|███████▉  | 161/203 [00:01<00:00, 86.61batch/s]

Training batches on cuda:  84%|████████▎ | 170/203 [00:01<00:00, 83.19batch/s]

Training batches on cuda:  88%|████████▊ | 179/203 [00:02<00:00, 82.42batch/s]

Training batches on cuda:  93%|█████████▎| 188/203 [00:02<00:00, 83.75batch/s]

Training batches on cuda:  98%|█████████▊| 198/203 [00:02<00:00, 87.16batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:22<00:02,  2.25s/epoch, loss=0.0014, prev_loss=0.0014]2021-09-16:17:25:38,484 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:25:39,496 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:25:54,371 INFO     [evaluator.py:657] Evaluation took 14.87s seconds
2021-09-16:17:25:54,491 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:38<00:00,  7.16s/epoch, loss=0.0014, prev_loss=0.0014]
Training epochs on cuda: 100%|██████████| 10/10 [00:38<00:00,  3.87s/epoch, loss=0.0014, prev_loss=0.0014]
2021-09-16:17:25:54,491 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp8tqfivgw'
2021-09-16:17:25:54,567 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp8tqfivgw' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.53ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.62ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.63ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:12, 2.65ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.53ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.67ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.67ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.67ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.67ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.67ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.67ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:09, 2.67ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.67ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.67ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.67ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.67ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:08, 2.67ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.67ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:05<00:08, 2.67ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.66ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.66ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.66ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.66ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:11<00:02, 2.66ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.66ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.66ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.66ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.66ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.66ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.66ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.66ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
2021-09-16:17:26:09,290 INFO     [evaluator.py:657] Evaluation took 14.71s seconds
[I 2021-09-16 17:26:09,294] Trial 2 finished with value: 0.01393291074065639 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.5617010747667871, 'optimizer.lr': 0.015166057850200584, 'negative_sampler.num_negs_per_pos': 3, 'training.batch_size': 1532}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:26:09,296 WARNING  [api.py:823] No random seed is specified. Setting to 4001998516.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:05, 17.54batch/s]

Training batches on cuda:   5%|▌         | 5/94 [00:00<00:04, 19.75batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:03, 21.52batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:03, 22.79batch/s]

Training batches on cuda:  15%|█▍        | 14/94 [00:00<00:03, 23.41batch/s]

Training batches on cuda:  18%|█▊        | 17/94 [00:00<00:03, 23.94batch/s]

Training batches on cuda:  21%|██▏       | 20/94 [00:00<00:03, 24.32batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:00<00:02, 23.80batch/s]

Training batches on cuda:  28%|██▊       | 26/94 [00:01<00:02, 23.59batch/s]

Training batches on cuda:  31%|███       | 29/94 [00:01<00:02, 23.48batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:01<00:02, 23.45batch/s]

Training batches on cuda:  37%|███▋      | 35/94 [00:01<00:02, 23.43batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:01<00:02, 23.12batch/s]

Training batches on cuda:  44%|████▎     | 41/94 [00:01<00:02, 22.32batch/s]

Training batches on cuda:  47%|████▋     | 44/94 [00:01<00:02, 22.64batch/s]

Training batches on cuda:  50%|█████     | 47/94 [00:01<00:02, 22.78batch/s]

Training batches on cuda:  53%|█████▎    | 50/94 [00:02<00:01, 22.83batch/s]

Training batches on cuda:  56%|█████▋    | 53/94 [00:02<00:01, 22.89batch/s]

Training batches on cuda:  60%|█████▉    | 56/94 [00:02<00:01, 22.91batch/s]

Training batches on cuda:  63%|██████▎   | 59/94 [00:02<00:01, 22.21batch/s]

Training batches on cuda:  66%|██████▌   | 62/94 [00:02<00:01, 21.93batch/s]

Training batches on cuda:  69%|██████▉   | 65/94 [00:02<00:01, 22.77batch/s]

Training batches on cuda:  72%|███████▏  | 68/94 [00:02<00:01, 23.08batch/s]

Training batches on cuda:  76%|███████▌  | 71/94 [00:03<00:00, 23.50batch/s]

Training batches on cuda:  79%|███████▊  | 74/94 [00:03<00:00, 24.02batch/s]

Training batches on cuda:  82%|████████▏ | 77/94 [00:03<00:00, 24.24batch/s]

Training batches on cuda:  85%|████████▌ | 80/94 [00:03<00:00, 24.38batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:03<00:00, 24.49batch/s]

Training batches on cuda:  91%|█████████▏| 86/94 [00:03<00:00, 24.55batch/s]

Training batches on cuda:  95%|█████████▍| 89/94 [00:03<00:00, 24.33batch/s]

Training batches on cuda:  98%|█████████▊| 92/94 [00:03<00:00, 24.53batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000451, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:37,  4.18s/epoch, loss=0.000451, prev_loss=nan]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:06, 14.57batch/s]

Training batches on cuda:   5%|▌         | 5/94 [00:00<00:05, 16.56batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:04, 18.37batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:04, 19.81batch/s]

Training batches on cuda:  15%|█▍        | 14/94 [00:00<00:03, 21.02batch/s]

Training batches on cuda:  17%|█▋        | 16/94 [00:00<00:04, 17.82batch/s]

Training batches on cuda:  20%|██        | 19/94 [00:00<00:03, 19.42batch/s]

Training batches on cuda:  23%|██▎       | 22/94 [00:01<00:03, 20.74batch/s]

Training batches on cuda:  27%|██▋       | 25/94 [00:01<00:03, 21.87batch/s]

Training batches on cuda:  30%|██▉       | 28/94 [00:01<00:02, 22.58batch/s]

Training batches on cuda:  33%|███▎      | 31/94 [00:01<00:02, 23.11batch/s]

Training batches on cuda:  36%|███▌      | 34/94 [00:01<00:02, 23.37batch/s]

Training batches on cuda:  39%|███▉      | 37/94 [00:01<00:02, 23.47batch/s]

Training batches on cuda:  43%|████▎     | 40/94 [00:01<00:02, 23.67batch/s]

Training batches on cuda:  46%|████▌     | 43/94 [00:01<00:02, 23.81batch/s]

Training batches on cuda:  49%|████▉     | 46/94 [00:02<00:01, 24.18batch/s]

Training batches on cuda:  52%|█████▏    | 49/94 [00:02<00:01, 24.31batch/s]

Training batches on cuda:  55%|█████▌    | 52/94 [00:02<00:01, 24.18batch/s]

Training batches on cuda:  59%|█████▊    | 55/94 [00:02<00:01, 24.30batch/s]

Training batches on cuda:  62%|██████▏   | 58/94 [00:02<00:01, 24.23batch/s]

Training batches on cuda:  65%|██████▍   | 61/94 [00:02<00:01, 24.38batch/s]

Training batches on cuda:  68%|██████▊   | 64/94 [00:02<00:01, 24.53batch/s]

Training batches on cuda:  71%|███████▏  | 67/94 [00:02<00:01, 24.63batch/s]

Training batches on cuda:  74%|███████▍  | 70/94 [00:02<00:00, 24.57batch/s]

Training batches on cuda:  78%|███████▊  | 73/94 [00:03<00:00, 24.60batch/s]

Training batches on cuda:  81%|████████  | 76/94 [00:03<00:00, 24.40batch/s]

Training batches on cuda:  84%|████████▍ | 79/94 [00:03<00:00, 23.87batch/s]

Training batches on cuda:  87%|████████▋ | 82/94 [00:03<00:00, 23.02batch/s]

Training batches on cuda:  90%|█████████ | 85/94 [00:03<00:00, 23.01batch/s]

Training batches on cuda:  94%|█████████▎| 88/94 [00:03<00:00, 22.70batch/s]

Training batches on cuda:  97%|█████████▋| 91/94 [00:03<00:00, 23.31batch/s]

Training batches on cuda: 100%|██████████| 94/94 [00:04<00:00, 24.28batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:37,  4.18s/epoch, loss=0.000414, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:33,  4.21s/epoch, loss=0.000414, prev_loss=nan]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:06, 15.20batch/s]

Training batches on cuda:   5%|▌         | 5/94 [00:00<00:05, 16.93batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:04, 18.73batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:04, 20.23batch/s]

Training batches on cuda:  15%|█▍        | 14/94 [00:00<00:03, 21.44batch/s]

Training batches on cuda:  18%|█▊        | 17/94 [00:00<00:03, 22.33batch/s]

Training batches on cuda:  21%|██▏       | 20/94 [00:00<00:03, 22.92batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:00<00:03, 23.42batch/s]

Training batches on cuda:  28%|██▊       | 26/94 [00:01<00:02, 23.77batch/s]

Training batches on cuda:  31%|███       | 29/94 [00:01<00:02, 24.13batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:01<00:02, 24.31batch/s]

Training batches on cuda:  37%|███▋      | 35/94 [00:01<00:02, 22.10batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:01<00:02, 22.59batch/s]

Training batches on cuda:  44%|████▎     | 41/94 [00:01<00:02, 23.20batch/s]

Training batches on cuda:  47%|████▋     | 44/94 [00:01<00:02, 23.44batch/s]

Training batches on cuda:  50%|█████     | 47/94 [00:02<00:01, 23.63batch/s]

Training batches on cuda:  53%|█████▎    | 50/94 [00:02<00:01, 23.92batch/s]

Training batches on cuda:  56%|█████▋    | 53/94 [00:02<00:01, 23.71batch/s]

Training batches on cuda:  60%|█████▉    | 56/94 [00:02<00:01, 23.83batch/s]

Training batches on cuda:  63%|██████▎   | 59/94 [00:02<00:01, 23.79batch/s]

Training batches on cuda:  66%|██████▌   | 62/94 [00:02<00:01, 23.70batch/s]

Training batches on cuda:  69%|██████▉   | 65/94 [00:02<00:01, 23.82batch/s]

Training batches on cuda:  72%|███████▏  | 68/94 [00:02<00:01, 23.93batch/s]

Training batches on cuda:  76%|███████▌  | 71/94 [00:03<00:00, 24.08batch/s]

Training batches on cuda:  79%|███████▊  | 74/94 [00:03<00:00, 24.26batch/s]

Training batches on cuda:  82%|████████▏ | 77/94 [00:03<00:00, 24.21batch/s]

Training batches on cuda:  85%|████████▌ | 80/94 [00:03<00:00, 24.06batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:03<00:00, 24.14batch/s]

Training batches on cuda:  91%|█████████▏| 86/94 [00:03<00:00, 24.06batch/s]

Training batches on cuda:  95%|█████████▍| 89/94 [00:03<00:00, 23.70batch/s]

Training batches on cuda:  98%|█████████▊| 92/94 [00:03<00:00, 23.37batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:33,  4.21s/epoch, loss=0.000415, prev_loss=0.000414]
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:29,  4.21s/epoch, loss=0.000415, prev_loss=0.000414]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/94 [00:00<00:10,  9.26batch/s]

Training batches on cuda:   4%|▍         | 4/94 [00:00<00:08, 11.22batch/s]

Training batches on cuda:   7%|▋         | 7/94 [00:00<00:06, 13.20batch/s]

Training batches on cuda:  11%|█         | 10/94 [00:00<00:05, 15.10batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:04, 16.83batch/s]

Training batches on cuda:  17%|█▋        | 16/94 [00:00<00:04, 18.24batch/s]

Training batches on cuda:  20%|██        | 19/94 [00:00<00:03, 19.40batch/s]

Training batches on cuda:  23%|██▎       | 22/94 [00:01<00:03, 20.22batch/s]

Training batches on cuda:  27%|██▋       | 25/94 [00:01<00:03, 20.90batch/s]

Training batches on cuda:  30%|██▉       | 28/94 [00:01<00:03, 21.43batch/s]

Training batches on cuda:  33%|███▎      | 31/94 [00:01<00:02, 21.44batch/s]

Training batches on cuda:  36%|███▌      | 34/94 [00:01<00:02, 21.74batch/s]

Training batches on cuda:  39%|███▉      | 37/94 [00:01<00:02, 22.26batch/s]

Training batches on cuda:  43%|████▎     | 40/94 [00:01<00:02, 22.83batch/s]

Training batches on cuda:  46%|████▌     | 43/94 [00:01<00:02, 23.08batch/s]

Training batches on cuda:  49%|████▉     | 46/94 [00:02<00:02, 23.39batch/s]

Training batches on cuda:  52%|█████▏    | 49/94 [00:02<00:01, 23.67batch/s]

Training batches on cuda:  55%|█████▌    | 52/94 [00:02<00:01, 21.15batch/s]

Training batches on cuda:  59%|█████▊    | 55/94 [00:02<00:01, 22.01batch/s]

Training batches on cuda:  62%|██████▏   | 58/94 [00:02<00:01, 22.60batch/s]

Training batches on cuda:  65%|██████▍   | 61/94 [00:02<00:01, 23.07batch/s]

Training batches on cuda:  68%|██████▊   | 64/94 [00:02<00:01, 23.58batch/s]

Training batches on cuda:  71%|███████▏  | 67/94 [00:02<00:01, 23.54batch/s]

Training batches on cuda:  74%|███████▍  | 70/94 [00:03<00:01, 23.27batch/s]

Training batches on cuda:  78%|███████▊  | 73/94 [00:03<00:00, 22.97batch/s]

Training batches on cuda:  81%|████████  | 76/94 [00:03<00:00, 22.51batch/s]

Training batches on cuda:  84%|████████▍ | 79/94 [00:03<00:00, 21.61batch/s]

Training batches on cuda:  87%|████████▋ | 82/94 [00:03<00:00, 22.04batch/s]

Training batches on cuda:  90%|█████████ | 85/94 [00:03<00:00, 22.91batch/s]

Training batches on cuda:  94%|█████████▎| 88/94 [00:03<00:00, 23.37batch/s]

Training batches on cuda:  97%|█████████▋| 91/94 [00:04<00:00, 23.64batch/s]

Training batches on cuda: 100%|██████████| 94/94 [00:04<00:00, 25.19batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:29,  4.21s/epoch, loss=0.000415, prev_loss=0.000415]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:25,  4.26s/epoch, loss=0.000415, prev_loss=0.000415]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/94 [00:00<00:09,  9.59batch/s]

Training batches on cuda:   4%|▍         | 4/94 [00:00<00:07, 11.75batch/s]

Training batches on cuda:   7%|▋         | 7/94 [00:00<00:06, 14.09batch/s]

Training batches on cuda:  11%|█         | 10/94 [00:00<00:05, 16.05batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:04, 18.12batch/s]

Training batches on cuda:  17%|█▋        | 16/94 [00:00<00:03, 20.00batch/s]

Training batches on cuda:  20%|██        | 19/94 [00:00<00:03, 21.42batch/s]

Training batches on cuda:  23%|██▎       | 22/94 [00:00<00:03, 22.28batch/s]

Training batches on cuda:  27%|██▋       | 25/94 [00:01<00:02, 23.15batch/s]

Training batches on cuda:  30%|██▉       | 28/94 [00:01<00:02, 23.68batch/s]

Training batches on cuda:  33%|███▎      | 31/94 [00:01<00:02, 23.62batch/s]

Training batches on cuda:  36%|███▌      | 34/94 [00:01<00:02, 23.42batch/s]

Training batches on cuda:  39%|███▉      | 37/94 [00:01<00:02, 23.50batch/s]

Training batches on cuda:  43%|████▎     | 40/94 [00:01<00:02, 23.69batch/s]

Training batches on cuda:  46%|████▌     | 43/94 [00:01<00:02, 23.59batch/s]

Training batches on cuda:  49%|████▉     | 46/94 [00:01<00:02, 23.42batch/s]

Training batches on cuda:  52%|█████▏    | 49/94 [00:02<00:01, 23.78batch/s]

Training batches on cuda:  55%|█████▌    | 52/94 [00:02<00:01, 24.08batch/s]

Training batches on cuda:  59%|█████▊    | 55/94 [00:02<00:01, 24.20batch/s]

Training batches on cuda:  62%|██████▏   | 58/94 [00:02<00:01, 24.21batch/s]

Training batches on cuda:  65%|██████▍   | 61/94 [00:02<00:01, 24.04batch/s]

Training batches on cuda:  68%|██████▊   | 64/94 [00:02<00:01, 24.10batch/s]

Training batches on cuda:  71%|███████▏  | 67/94 [00:02<00:01, 24.18batch/s]

Training batches on cuda:  74%|███████▍  | 70/94 [00:02<00:00, 24.39batch/s]

Training batches on cuda:  78%|███████▊  | 73/94 [00:03<00:00, 23.35batch/s]

Training batches on cuda:  81%|████████  | 76/94 [00:03<00:00, 23.69batch/s]

Training batches on cuda:  84%|████████▍ | 79/94 [00:03<00:00, 23.94batch/s]

Training batches on cuda:  87%|████████▋ | 82/94 [00:03<00:00, 24.06batch/s]

Training batches on cuda:  90%|█████████ | 85/94 [00:03<00:00, 24.14batch/s]

Training batches on cuda:  94%|█████████▎| 88/94 [00:03<00:00, 24.32batch/s]

Training batches on cuda:  97%|█████████▋| 91/94 [00:03<00:00, 24.35batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:21<00:25,  4.26s/epoch, loss=0.000415, prev_loss=0.000415]
Training epochs on cuda:  50%|█████     | 5/10 [00:21<00:21,  4.25s/epoch, loss=0.000415, prev_loss=0.000415]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:05, 17.90batch/s]

Training batches on cuda:   5%|▌         | 5/94 [00:00<00:04, 19.43batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:04, 20.79batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:03, 21.48batch/s]

Training batches on cuda:  15%|█▍        | 14/94 [00:00<00:03, 21.95batch/s]

Training batches on cuda:  18%|█▊        | 17/94 [00:00<00:03, 22.80batch/s]

Training batches on cuda:  21%|██▏       | 20/94 [00:00<00:03, 23.39batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:00<00:02, 23.79batch/s]

Training batches on cuda:  28%|██▊       | 26/94 [00:01<00:02, 24.25batch/s]

Training batches on cuda:  31%|███       | 29/94 [00:01<00:02, 24.50batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:01<00:02, 24.38batch/s]

Training batches on cuda:  37%|███▋      | 35/94 [00:01<00:02, 24.34batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:01<00:02, 24.36batch/s]

Training batches on cuda:  44%|████▎     | 41/94 [00:01<00:02, 24.49batch/s]

Training batches on cuda:  47%|████▋     | 44/94 [00:01<00:02, 24.60batch/s]

Training batches on cuda:  50%|█████     | 47/94 [00:01<00:01, 24.69batch/s]

Training batches on cuda:  53%|█████▎    | 50/94 [00:02<00:01, 24.72batch/s]

Training batches on cuda:  56%|█████▋    | 53/94 [00:02<00:01, 24.73batch/s]

Training batches on cuda:  60%|█████▉    | 56/94 [00:02<00:01, 24.81batch/s]

Training batches on cuda:  63%|██████▎   | 59/94 [00:02<00:01, 24.88batch/s]

Training batches on cuda:  66%|██████▌   | 62/94 [00:02<00:01, 24.96batch/s]

Training batches on cuda:  69%|██████▉   | 65/94 [00:02<00:01, 25.02batch/s]

Training batches on cuda:  72%|███████▏  | 68/94 [00:02<00:01, 24.94batch/s]

Training batches on cuda:  76%|███████▌  | 71/94 [00:02<00:00, 25.01batch/s]

Training batches on cuda:  79%|███████▊  | 74/94 [00:03<00:00, 24.99batch/s]

Training batches on cuda:  82%|████████▏ | 77/94 [00:03<00:00, 25.06batch/s]

Training batches on cuda:  85%|████████▌ | 80/94 [00:03<00:00, 25.03batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:03<00:00, 24.97batch/s]

Training batches on cuda:  91%|█████████▏| 86/94 [00:03<00:00, 24.99batch/s]

Training batches on cuda:  95%|█████████▍| 89/94 [00:03<00:00, 25.00batch/s]

Training batches on cuda:  98%|█████████▊| 92/94 [00:03<00:00, 24.97batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:25<00:21,  4.25s/epoch, loss=0.000416, prev_loss=0.000415]
Training epochs on cuda:  60%|██████    | 6/10 [00:25<00:16,  4.18s/epoch, loss=0.000416, prev_loss=0.000415]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:05, 15.60batch/s]

Training batches on cuda:   5%|▌         | 5/94 [00:00<00:05, 17.45batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:04, 19.18batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:04, 20.27batch/s]

Training batches on cuda:  15%|█▍        | 14/94 [00:00<00:03, 21.00batch/s]

Training batches on cuda:  18%|█▊        | 17/94 [00:00<00:03, 21.84batch/s]

Training batches on cuda:  21%|██▏       | 20/94 [00:00<00:03, 22.46batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:01<00:03, 22.96batch/s]

Training batches on cuda:  28%|██▊       | 26/94 [00:01<00:02, 23.58batch/s]

Training batches on cuda:  31%|███       | 29/94 [00:01<00:02, 23.90batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:01<00:02, 24.06batch/s]

Training batches on cuda:  37%|███▋      | 35/94 [00:01<00:02, 24.23batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:01<00:02, 24.38batch/s]

Training batches on cuda:  44%|████▎     | 41/94 [00:01<00:02, 24.46batch/s]

Training batches on cuda:  47%|████▋     | 44/94 [00:01<00:02, 24.67batch/s]

Training batches on cuda:  50%|█████     | 47/94 [00:01<00:01, 24.76batch/s]

Training batches on cuda:  53%|█████▎    | 50/94 [00:02<00:01, 24.79batch/s]

Training batches on cuda:  56%|█████▋    | 53/94 [00:02<00:01, 24.79batch/s]

Training batches on cuda:  60%|█████▉    | 56/94 [00:02<00:01, 24.69batch/s]

Training batches on cuda:  63%|██████▎   | 59/94 [00:02<00:01, 24.80batch/s]

Training batches on cuda:  66%|██████▌   | 62/94 [00:02<00:01, 24.90batch/s]

Training batches on cuda:  69%|██████▉   | 65/94 [00:02<00:01, 24.86batch/s]

Training batches on cuda:  72%|███████▏  | 68/94 [00:02<00:01, 24.85batch/s]

Training batches on cuda:  76%|███████▌  | 71/94 [00:02<00:00, 24.63batch/s]

Training batches on cuda:  79%|███████▊  | 74/94 [00:03<00:00, 25.09batch/s]

Training batches on cuda:  82%|████████▏ | 77/94 [00:03<00:00, 25.38batch/s]

Training batches on cuda:  85%|████████▌ | 80/94 [00:03<00:00, 25.33batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:03<00:00, 25.63batch/s]

Training batches on cuda:  91%|█████████▏| 86/94 [00:03<00:00, 25.72batch/s]

Training batches on cuda:  95%|█████████▍| 89/94 [00:03<00:00, 25.41batch/s]

Training batches on cuda:  98%|█████████▊| 92/94 [00:03<00:00, 25.61batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [00:29<00:16,  4.18s/epoch, loss=0.000416, prev_loss=0.000416]
Training epochs on cuda:  70%|███████   | 7/10 [00:29<00:12,  4.13s/epoch, loss=0.000416, prev_loss=0.000416]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:04, 19.34batch/s]

Training batches on cuda:   5%|▌         | 5/94 [00:00<00:04, 21.01batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:03, 22.69batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:03, 23.87batch/s]

Training batches on cuda:  15%|█▍        | 14/94 [00:00<00:03, 24.46batch/s]

Training batches on cuda:  18%|█▊        | 17/94 [00:00<00:03, 25.27batch/s]

Training batches on cuda:  21%|██▏       | 20/94 [00:00<00:02, 25.96batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:00<00:02, 26.45batch/s]

Training batches on cuda:  28%|██▊       | 26/94 [00:00<00:02, 26.85batch/s]

Training batches on cuda:  31%|███       | 29/94 [00:01<00:02, 27.13batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:01<00:02, 27.33batch/s]

Training batches on cuda:  37%|███▋      | 35/94 [00:01<00:02, 27.26batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:01<00:02, 27.14batch/s]

Training batches on cuda:  44%|████▎     | 41/94 [00:01<00:01, 27.10batch/s]

Training batches on cuda:  47%|████▋     | 44/94 [00:01<00:01, 27.05batch/s]

Training batches on cuda:  50%|█████     | 47/94 [00:01<00:01, 27.04batch/s]

Training batches on cuda:  53%|█████▎    | 50/94 [00:01<00:01, 26.94batch/s]

Training batches on cuda:  56%|█████▋    | 53/94 [00:01<00:01, 26.96batch/s]

Training batches on cuda:  60%|█████▉    | 56/94 [00:02<00:01, 26.88batch/s]

Training batches on cuda:  63%|██████▎   | 59/94 [00:02<00:01, 26.89batch/s]

Training batches on cuda:  66%|██████▌   | 62/94 [00:02<00:01, 26.84batch/s]

Training batches on cuda:  69%|██████▉   | 65/94 [00:02<00:01, 26.81batch/s]

Training batches on cuda:  72%|███████▏  | 68/94 [00:02<00:00, 26.25batch/s]

Training batches on cuda:  76%|███████▌  | 71/94 [00:02<00:00, 25.98batch/s]

Training batches on cuda:  79%|███████▊  | 74/94 [00:02<00:00, 25.69batch/s]

Training batches on cuda:  82%|████████▏ | 77/94 [00:02<00:00, 25.44batch/s]

Training batches on cuda:  85%|████████▌ | 80/94 [00:03<00:00, 25.38batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:03<00:00, 23.57batch/s]

Training batches on cuda:  91%|█████████▏| 86/94 [00:03<00:00, 23.98batch/s]

Training batches on cuda:  95%|█████████▍| 89/94 [00:03<00:00, 24.25batch/s]

Training batches on cuda:  98%|█████████▊| 92/94 [00:03<00:00, 24.37batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [00:33<00:12,  4.13s/epoch, loss=0.000416, prev_loss=0.000416]
Training epochs on cuda:  80%|████████  | 8/10 [00:33<00:08,  4.06s/epoch, loss=0.000416, prev_loss=0.000416]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:06, 13.90batch/s]

Training batches on cuda:   5%|▌         | 5/94 [00:00<00:05, 15.88batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:04, 17.48batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:04, 19.08batch/s]

Training batches on cuda:  15%|█▍        | 14/94 [00:00<00:03, 20.33batch/s]

Training batches on cuda:  18%|█▊        | 17/94 [00:00<00:03, 20.99batch/s]

Training batches on cuda:  21%|██▏       | 20/94 [00:00<00:03, 21.79batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:01<00:03, 22.33batch/s]

Training batches on cuda:  28%|██▊       | 26/94 [00:01<00:02, 22.74batch/s]

Training batches on cuda:  31%|███       | 29/94 [00:01<00:02, 22.91batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:01<00:02, 23.52batch/s]

Training batches on cuda:  37%|███▋      | 35/94 [00:01<00:02, 24.39batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:01<00:02, 24.86batch/s]

Training batches on cuda:  44%|████▎     | 41/94 [00:01<00:02, 25.23batch/s]

Training batches on cuda:  47%|████▋     | 44/94 [00:01<00:01, 25.48batch/s]

Training batches on cuda:  50%|█████     | 47/94 [00:01<00:01, 25.45batch/s]

Training batches on cuda:  53%|█████▎    | 50/94 [00:02<00:01, 25.31batch/s]

Training batches on cuda:  56%|█████▋    | 53/94 [00:02<00:01, 25.20batch/s]

Training batches on cuda:  60%|█████▉    | 56/94 [00:02<00:01, 25.09batch/s]

Training batches on cuda:  63%|██████▎   | 59/94 [00:02<00:01, 24.96batch/s]

Training batches on cuda:  66%|██████▌   | 62/94 [00:02<00:01, 24.85batch/s]

Training batches on cuda:  69%|██████▉   | 65/94 [00:02<00:01, 24.71batch/s]

Training batches on cuda:  72%|███████▏  | 68/94 [00:02<00:01, 24.42batch/s]

Training batches on cuda:  76%|███████▌  | 71/94 [00:02<00:00, 24.60batch/s]

Training batches on cuda:  79%|███████▊  | 74/94 [00:03<00:00, 24.77batch/s]

Training batches on cuda:  82%|████████▏ | 77/94 [00:03<00:00, 25.09batch/s]

Training batches on cuda:  85%|████████▌ | 80/94 [00:03<00:00, 25.44batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:03<00:00, 25.55batch/s]

Training batches on cuda:  91%|█████████▏| 86/94 [00:03<00:00, 24.72batch/s]

Training batches on cuda:  95%|█████████▍| 89/94 [00:03<00:00, 24.70batch/s]

Training batches on cuda:  98%|█████████▊| 92/94 [00:03<00:00, 24.70batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [00:37<00:08,  4.06s/epoch, loss=0.000416, prev_loss=0.000416]
Training epochs on cuda:  90%|█████████ | 9/10 [00:37<00:04,  4.06s/epoch, loss=0.000416, prev_loss=0.000416]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:05, 15.46batch/s]

Training batches on cuda:   4%|▍         | 4/94 [00:00<00:05, 15.98batch/s]

Training batches on cuda:   7%|▋         | 7/94 [00:00<00:04, 17.55batch/s]

Training batches on cuda:  11%|█         | 10/94 [00:00<00:04, 19.17batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:03, 20.53batch/s]

Training batches on cuda:  17%|█▋        | 16/94 [00:00<00:03, 21.60batch/s]

Training batches on cuda:  20%|██        | 19/94 [00:00<00:03, 22.31batch/s]

Training batches on cuda:  23%|██▎       | 22/94 [00:00<00:03, 22.72batch/s]

Training batches on cuda:  27%|██▋       | 25/94 [00:01<00:02, 23.27batch/s]

Training batches on cuda:  30%|██▉       | 28/94 [00:01<00:02, 23.46batch/s]

Training batches on cuda:  33%|███▎      | 31/94 [00:01<00:02, 23.49batch/s]

Training batches on cuda:  36%|███▌      | 34/94 [00:01<00:02, 23.51batch/s]

Training batches on cuda:  39%|███▉      | 37/94 [00:01<00:02, 23.83batch/s]

Training batches on cuda:  43%|████▎     | 40/94 [00:01<00:02, 23.68batch/s]

Training batches on cuda:  46%|████▌     | 43/94 [00:01<00:02, 23.62batch/s]

Training batches on cuda:  49%|████▉     | 46/94 [00:01<00:02, 23.77batch/s]

Training batches on cuda:  52%|█████▏    | 49/94 [00:02<00:01, 23.63batch/s]

Training batches on cuda:  55%|█████▌    | 52/94 [00:02<00:01, 24.50batch/s]

Training batches on cuda:  59%|█████▊    | 55/94 [00:02<00:01, 24.86batch/s]

Training batches on cuda:  62%|██████▏   | 58/94 [00:02<00:01, 25.18batch/s]

Training batches on cuda:  65%|██████▍   | 61/94 [00:02<00:01, 25.33batch/s]

Training batches on cuda:  68%|██████▊   | 64/94 [00:02<00:01, 24.57batch/s]

Training batches on cuda:  71%|███████▏  | 67/94 [00:02<00:01, 24.00batch/s]

Training batches on cuda:  74%|███████▍  | 70/94 [00:02<00:00, 24.16batch/s]

Training batches on cuda:  78%|███████▊  | 73/94 [00:03<00:00, 24.36batch/s]

Training batches on cuda:  81%|████████  | 76/94 [00:03<00:00, 24.51batch/s]

Training batches on cuda:  84%|████████▍ | 79/94 [00:03<00:00, 24.60batch/s]

Training batches on cuda:  87%|████████▋ | 82/94 [00:03<00:00, 24.40batch/s]

Training batches on cuda:  90%|█████████ | 85/94 [00:03<00:00, 24.29batch/s]

Training batches on cuda:  94%|█████████▎| 88/94 [00:03<00:00, 24.47batch/s]

Training batches on cuda:  97%|█████████▋| 91/94 [00:03<00:00, 24.59batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [00:41<00:04,  4.06s/epoch, loss=0.000416, prev_loss=0.000416]2021-09-16:17:26:51,229 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:26:52,242 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:27:07,245 INFO     [evaluator.py:657] Evaluation took 15.00s seconds
2021-09-16:17:27:07,371 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00,  8.94s/epoch, loss=0.000416, prev_loss=0.000416]
Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00,  5.76s/epoch, loss=0.000416, prev_loss=0.000416]
2021-09-16:17:27:07,371 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpm7mr68zi'
2021-09-16:17:27:07,445 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpm7mr68zi' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:18, 2.07ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:17, 2.22ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:16, 2.33ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.41ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.48ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.52ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.56ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.54ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:17:27:22,301 INFO     [evaluator.py:657] Evaluation took 14.83s seconds
[I 2021-09-16 17:27:22,306] Trial 3 finished with value: -0.004568206450257906 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.9643026546511089, 'optimizer.lr': 0.0652287550332071, 'negative_sampler.num_negs_per_pos': 16, 'training.batch_size': 3332}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:27:22,308 WARNING  [api.py:823] No random seed is specified. Setting to 787638659.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 26.62batch/s]

Training batches on cuda:   7%|▋         | 8/110 [00:00<00:03, 29.66batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:03, 32.33batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:00<00:02, 33.73batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:00<00:02, 34.81batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:00<00:02, 33.57batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:00<00:02, 34.78batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:00<00:02, 35.88batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:01<00:01, 36.86batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:01<00:01, 37.29batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:01<00:01, 36.15batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:01<00:01, 36.96batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:01<00:01, 37.82batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:01<00:01, 37.15batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:01<00:01, 36.96batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:01<00:01, 37.74batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:01, 38.05batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:01<00:00, 37.88batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:02<00:00, 37.96batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:02<00:00, 37.80batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:02<00:00, 38.00batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:02<00:00, 37.73batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:02<00:00, 37.00batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:02<00:00, 36.78batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:02<00:00, 36.59batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:02<00:00, 33.75batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:02<00:00, 35.01batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=5.32e-5, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:28,  3.18s/epoch, loss=5.32e-5, prev_loss=nan]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 24.81batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 27.91batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 30.47batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:02, 32.70batch/s]

Training batches on cuda:  18%|█▊        | 20/110 [00:00<00:02, 34.78batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:00<00:02, 36.32batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:00<00:02, 37.19batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:00<00:02, 37.66batch/s]

Training batches on cuda:  35%|███▍      | 38/110 [00:01<00:01, 38.29batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:01<00:01, 39.11batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:01<00:01, 39.25batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:01<00:01, 39.33batch/s]

Training batches on cuda:  51%|█████     | 56/110 [00:01<00:01, 39.58batch/s]

Training batches on cuda:  55%|█████▍    | 60/110 [00:01<00:01, 39.51batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:01<00:01, 39.66batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:01, 38.83batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:01<00:00, 38.62batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:01<00:00, 38.04batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:02<00:00, 38.31batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:02<00:00, 38.29batch/s]

Training batches on cuda:  82%|████████▏ | 90/110 [00:02<00:00, 38.80batch/s]

Training batches on cuda:  85%|████████▌ | 94/110 [00:02<00:00, 37.95batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:02<00:00, 39.11batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:02<00:00, 38.32batch/s]

Training batches on cuda:  98%|█████████▊| 108/110 [00:02<00:00, 39.24batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:28,  3.18s/epoch, loss=1.94e-5, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:25,  3.13s/epoch, loss=1.94e-5, prev_loss=nan]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/110 [00:00<00:05, 19.10batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:04, 22.74batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 26.10batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:03, 28.12batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:03, 29.26batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:00<00:02, 30.80batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:00<00:02, 31.54batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:00<00:02, 32.17batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:01<00:02, 32.36batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:01<00:02, 34.18batch/s]

Training batches on cuda:  40%|████      | 44/110 [00:01<00:01, 35.95batch/s]

Training batches on cuda:  44%|████▎     | 48/110 [00:01<00:01, 36.62batch/s]

Training batches on cuda:  47%|████▋     | 52/110 [00:01<00:01, 37.48batch/s]

Training batches on cuda:  51%|█████     | 56/110 [00:01<00:01, 37.74batch/s]

Training batches on cuda:  55%|█████▍    | 60/110 [00:01<00:01, 34.59batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:01<00:01, 36.06batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:01, 37.04batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:02<00:00, 37.60batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:02<00:00, 38.09batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:02<00:00, 38.32batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:02<00:00, 38.65batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:02<00:00, 38.75batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:02<00:00, 38.87batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:02<00:00, 39.12batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:02<00:00, 39.30batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:02<00:00, 39.24batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:02<00:00, 39.24batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:25,  3.13s/epoch, loss=2.03e-5, prev_loss=1.94e-5]
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:22,  3.17s/epoch, loss=2.03e-5, prev_loss=1.94e-5]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 23.56batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 26.76batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 29.39batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:03, 31.37batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:02, 33.25batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:00<00:02, 34.42batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:00<00:02, 35.50batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:00<00:02, 36.50batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:00<00:02, 36.79batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:01<00:01, 36.98batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:01<00:01, 36.74batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:01<00:01, 36.84batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:01<00:01, 36.73batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:01<00:01, 37.05batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:01<00:01, 36.78batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:01<00:01, 36.74batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:01<00:01, 36.90batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:01<00:01, 37.54batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:02<00:00, 38.19batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:02<00:00, 38.38batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:02<00:00, 36.91batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:02<00:00, 35.23batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:02<00:00, 34.85batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:02<00:00, 34.80batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:02<00:00, 35.50batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:02<00:00, 35.66batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:02<00:00, 34.91batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:22,  3.17s/epoch, loss=2.02e-5, prev_loss=2.03e-5]
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:19,  3.23s/epoch, loss=2.02e-5, prev_loss=2.03e-5]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/110 [00:00<00:06, 17.64batch/s]

Training batches on cuda:   5%|▌         | 6/110 [00:00<00:04, 20.83batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:04, 24.32batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:03, 27.48batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:03, 29.72batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:00<00:02, 31.19batch/s]

Training batches on cuda:  25%|██▌       | 28/110 [00:00<00:02, 33.48batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:00<00:02, 35.22batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:01<00:02, 35.71batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:01<00:01, 36.05batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:01<00:01, 36.41batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:01<00:01, 36.26batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:01<00:01, 36.77batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:01<00:01, 37.42batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:01<00:01, 36.68batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:01<00:01, 36.84batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:01, 35.56batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:02<00:01, 36.43batch/s]

Training batches on cuda:  71%|███████   | 78/110 [00:02<00:00, 37.36batch/s]

Training batches on cuda:  75%|███████▍  | 82/110 [00:02<00:00, 37.55batch/s]

Training batches on cuda:  78%|███████▊  | 86/110 [00:02<00:00, 38.18batch/s]

Training batches on cuda:  82%|████████▏ | 90/110 [00:02<00:00, 38.51batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:02<00:00, 39.53batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:02<00:00, 38.54batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:02<00:00, 38.94batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:02<00:00, 38.30batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:16<00:19,  3.23s/epoch, loss=2.04e-5, prev_loss=2.02e-5]
Training epochs on cuda:  50%|█████     | 5/10 [00:16<00:16,  3.25s/epoch, loss=2.04e-5, prev_loss=2.02e-5]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 24.06batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 27.23batch/s]

Training batches on cuda:  11%|█         | 12/110 [00:00<00:03, 30.36batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:00<00:02, 32.46batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:00<00:02, 34.16batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:00<00:02, 33.74batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:00<00:02, 34.94batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:00<00:02, 35.62batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:01<00:02, 36.14batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:01<00:01, 36.25batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:01<00:01, 36.78batch/s]

Training batches on cuda:  45%|████▌     | 50/110 [00:01<00:01, 37.66batch/s]

Training batches on cuda:  49%|████▉     | 54/110 [00:01<00:01, 37.60batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:01<00:01, 38.64batch/s]

Training batches on cuda:  58%|█████▊    | 64/110 [00:01<00:01, 39.21batch/s]

Training batches on cuda:  62%|██████▏   | 68/110 [00:01<00:01, 36.72batch/s]

Training batches on cuda:  65%|██████▌   | 72/110 [00:01<00:01, 37.42batch/s]

Training batches on cuda:  69%|██████▉   | 76/110 [00:02<00:00, 37.95batch/s]

Training batches on cuda:  73%|███████▎  | 80/110 [00:02<00:00, 38.31batch/s]

Training batches on cuda:  76%|███████▋  | 84/110 [00:02<00:00, 38.10batch/s]

Training batches on cuda:  80%|████████  | 88/110 [00:02<00:00, 38.25batch/s]

Training batches on cuda:  84%|████████▎ | 92/110 [00:02<00:00, 37.08batch/s]

Training batches on cuda:  87%|████████▋ | 96/110 [00:02<00:00, 37.32batch/s]

Training batches on cuda:  91%|█████████ | 100/110 [00:02<00:00, 36.47batch/s]

Training batches on cuda:  95%|█████████▍| 104/110 [00:02<00:00, 36.62batch/s]

Training batches on cuda:  98%|█████████▊| 108/110 [00:02<00:00, 36.86batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:19<00:16,  3.25s/epoch, loss=2.16e-5, prev_loss=2.04e-5]
Training epochs on cuda:  60%|██████    | 6/10 [00:19<00:12,  3.24s/epoch, loss=2.16e-5, prev_loss=2.04e-5]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 23.92batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 27.20batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 30.05batch/s]

Training batches on cuda:  15%|█▍        | 16/110 [00:00<00:02, 32.90batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:00<00:02, 35.34batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:00<00:02, 36.08batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:00<00:02, 36.07batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:00<00:02, 34.85batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:01<00:02, 33.69batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:01<00:02, 33.53batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:01<00:01, 34.47batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:01<00:01, 34.94batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:01<00:01, 35.11batch/s]

Training batches on cuda:  53%|█████▎    | 58/110 [00:01<00:01, 35.75batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:01<00:01, 36.90batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:01<00:01, 37.52batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:01<00:01, 37.92batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:02<00:00, 38.29batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:02<00:00, 38.50batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:02<00:00, 37.52batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:02<00:00, 36.76batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:02<00:00, 36.61batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:02<00:00, 34.30batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:02<00:00, 34.89batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:02<00:00, 35.12batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:02<00:00, 34.39batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:22<00:12,  3.24s/epoch, loss=2.07e-5, prev_loss=2.16e-5]
Training epochs on cuda:  70%|███████   | 7/10 [00:22<00:09,  3.27s/epoch, loss=2.07e-5, prev_loss=2.16e-5]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/110 [00:00<00:06, 17.14batch/s]

Training batches on cuda:   5%|▌         | 6/110 [00:00<00:05, 19.97batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:04, 22.13batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:03, 25.55batch/s]

Training batches on cuda:  16%|█▋        | 18/110 [00:00<00:03, 28.60batch/s]

Training batches on cuda:  20%|██        | 22/110 [00:00<00:02, 30.74batch/s]

Training batches on cuda:  24%|██▎       | 26/110 [00:00<00:02, 29.95batch/s]

Training batches on cuda:  27%|██▋       | 30/110 [00:00<00:02, 31.19batch/s]

Training batches on cuda:  31%|███       | 34/110 [00:01<00:02, 32.19batch/s]

Training batches on cuda:  35%|███▍      | 38/110 [00:01<00:02, 32.83batch/s]

Training batches on cuda:  38%|███▊      | 42/110 [00:01<00:02, 33.00batch/s]

Training batches on cuda:  42%|████▏     | 46/110 [00:01<00:01, 34.13batch/s]

Training batches on cuda:  45%|████▌     | 50/110 [00:01<00:01, 34.72batch/s]

Training batches on cuda:  49%|████▉     | 54/110 [00:01<00:01, 35.48batch/s]

Training batches on cuda:  53%|█████▎    | 58/110 [00:01<00:01, 36.33batch/s]

Training batches on cuda:  56%|█████▋    | 62/110 [00:01<00:01, 37.19batch/s]

Training batches on cuda:  60%|██████    | 66/110 [00:01<00:01, 37.08batch/s]

Training batches on cuda:  64%|██████▎   | 70/110 [00:02<00:01, 37.42batch/s]

Training batches on cuda:  67%|██████▋   | 74/110 [00:02<00:00, 37.76batch/s]

Training batches on cuda:  71%|███████   | 78/110 [00:02<00:00, 37.86batch/s]

Training batches on cuda:  75%|███████▍  | 82/110 [00:02<00:00, 38.04batch/s]

Training batches on cuda:  78%|███████▊  | 86/110 [00:02<00:00, 37.67batch/s]

Training batches on cuda:  82%|████████▏ | 90/110 [00:02<00:00, 36.82batch/s]

Training batches on cuda:  85%|████████▌ | 94/110 [00:02<00:00, 36.24batch/s]

Training batches on cuda:  89%|████████▉ | 98/110 [00:02<00:00, 36.24batch/s]

Training batches on cuda:  93%|█████████▎| 102/110 [00:02<00:00, 37.08batch/s]

Training batches on cuda:  96%|█████████▋| 106/110 [00:03<00:00, 36.19batch/s]

Training batches on cuda: 100%|██████████| 110/110 [00:03<00:00, 36.51batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:25<00:09,  3.27s/epoch, loss=2.03e-5, prev_loss=2.07e-5]
Training epochs on cuda:  80%|████████  | 8/10 [00:25<00:06,  3.28s/epoch, loss=2.03e-5, prev_loss=2.07e-5]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 26.25batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 29.13batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 31.35batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:02, 32.83batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:02, 33.57batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:00<00:02, 34.26batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:00<00:02, 34.49batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:00<00:02, 35.24batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:00<00:02, 35.74batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:01<00:01, 36.35batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:01<00:01, 36.43batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:01<00:01, 36.46batch/s]

Training batches on cuda:  47%|████▋     | 52/110 [00:01<00:01, 37.55batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:01<00:01, 38.04batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:01<00:01, 37.61batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:01<00:01, 38.10batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:01, 35.95batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:02<00:01, 36.18batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:02<00:00, 33.25batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:02<00:00, 33.11batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:02<00:00, 33.72batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:02<00:00, 34.97batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:02<00:00, 36.33batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:02<00:00, 36.52batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:02<00:00, 37.37batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:02<00:00, 37.95batch/s]

Training batches on cuda: 100%|██████████| 110/110 [00:03<00:00, 37.04batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:29<00:06,  3.28s/epoch, loss=2.04e-5, prev_loss=2.03e-5]
Training epochs on cuda:  90%|█████████ | 9/10 [00:29<00:03,  3.27s/epoch, loss=2.04e-5, prev_loss=2.03e-5]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 23.73batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 26.50batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 29.00batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:03, 31.09batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:02, 32.58batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:00<00:02, 33.76batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:00<00:02, 34.37batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:00<00:02, 35.01batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:00<00:02, 35.40batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:01<00:01, 35.79batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:01<00:01, 36.53batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:01<00:01, 37.38batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:01<00:01, 37.89batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:01<00:01, 38.12batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:01<00:01, 38.09batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:01<00:01, 35.00batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:01<00:01, 34.61batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:01<00:01, 35.50batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:02<00:00, 36.46batch/s]

Training batches on cuda:  73%|███████▎  | 80/110 [00:02<00:00, 37.47batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:02<00:00, 38.68batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:02<00:00, 38.29batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:02<00:00, 37.48batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:02<00:00, 36.48batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:02<00:00, 36.50batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:02<00:00, 36.67batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:02<00:00, 36.76batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:32<00:03,  3.27s/epoch, loss=2.1e-5, prev_loss=2.04e-5] 2021-09-16:17:27:55,317 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:27:56,311 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:28:11,277 INFO     [evaluator.py:657] Evaluation took 14.97s seconds
2021-09-16:17:28:11,407 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00,  8.11s/epoch, loss=2.1e-5, prev_loss=2.04e-5]
Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00,  4.86s/epoch, loss=2.1e-5, prev_loss=2.04e-5]
2021-09-16:17:28:11,407 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpi0em8w3k'
2021-09-16:17:28:11,461 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi0em8w3k' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.39ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.53ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.56ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.58ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:17:28:26,305 INFO     [evaluator.py:657] Evaluation took 14.83s seconds
[I 2021-09-16 17:28:26,310] Trial 4 finished with value: -0.0005182428139558493 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.9061600386951018, 'optimizer.lr': 0.06677449304843394, 'negative_sampler.num_negs_per_pos': 10, 'training.batch_size': 2832}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:28:26,312 WARNING  [api.py:823] No random seed is specified. Setting to 1363339379.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/153 [00:00<00:08, 18.58batch/s]

Training batches on cuda:   4%|▍         | 6/153 [00:00<00:07, 20.96batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:06, 23.80batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:05, 26.11batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:04, 27.68batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:04, 29.72batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:04, 31.30batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:03, 32.14batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:01<00:03, 33.25batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:01<00:03, 33.94batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:01<00:03, 33.82batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:01<00:03, 34.02batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:01<00:03, 34.32batch/s]

Training batches on cuda:  35%|███▌      | 54/153 [00:01<00:02, 35.07batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:02, 34.98batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:02, 34.62batch/s]

Training batches on cuda:  43%|████▎     | 66/153 [00:01<00:02, 34.95batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:02<00:02, 34.88batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:02<00:02, 34.95batch/s]

Training batches on cuda:  51%|█████     | 78/153 [00:02<00:02, 35.29batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:02<00:02, 35.13batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:02<00:01, 35.38batch/s]

Training batches on cuda:  59%|█████▉    | 90/153 [00:02<00:01, 35.90batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:02<00:01, 36.01batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:02<00:01, 35.95batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:02<00:01, 35.83batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:03<00:01, 35.84batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:03<00:01, 36.06batch/s]

Training batches on cuda:  75%|███████▍  | 114/153 [00:03<00:01, 36.01batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:03<00:00, 36.28batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:03<00:00, 36.49batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:03<00:00, 36.34batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:03<00:00, 36.74batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:03<00:00, 37.01batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:03<00:00, 36.71batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:04<00:00, 37.11batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:04<00:00, 36.98batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:04<00:00, 36.41batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=6.68e-5, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:42,  4.68s/epoch, loss=6.68e-5, prev_loss=nan]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/153 [00:00<00:08, 18.27batch/s]

Training batches on cuda:   4%|▍         | 6/153 [00:00<00:06, 21.33batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:05, 24.63batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:05, 27.48batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:04, 29.81batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:04, 31.82batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:03, 33.45batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:03, 34.43batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:00<00:03, 35.27batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:01<00:03, 35.93batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:01<00:03, 36.12batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:01<00:02, 36.53batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:01<00:02, 36.78batch/s]

Training batches on cuda:  35%|███▌      | 54/153 [00:01<00:02, 36.81batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:02, 36.88batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:02, 37.03batch/s]

Training batches on cuda:  43%|████▎     | 66/153 [00:01<00:02, 37.12batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:02, 37.12batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:02<00:02, 37.05batch/s]

Training batches on cuda:  51%|█████     | 78/153 [00:02<00:02, 36.67batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:02<00:01, 36.85batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:02<00:01, 36.96batch/s]

Training batches on cuda:  59%|█████▉    | 90/153 [00:02<00:01, 37.10batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:02<00:01, 37.31batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:02<00:01, 37.16batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:02<00:01, 37.26batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:02<00:01, 36.52batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:03<00:01, 34.87batch/s]

Training batches on cuda:  75%|███████▍  | 114/153 [00:03<00:01, 35.50batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:03<00:00, 35.77batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:03<00:00, 36.28batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:03<00:00, 36.43batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:03<00:00, 36.47batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:03<00:00, 36.61batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:03<00:00, 36.62batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:03<00:00, 35.82batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:04<00:00, 36.29batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:04<00:00, 36.01batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:09<00:42,  4.68s/epoch, loss=3.48e-5, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:36,  4.62s/epoch, loss=3.48e-5, prev_loss=nan]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 27.04batch/s]

Training batches on cuda:   5%|▍         | 7/153 [00:00<00:05, 29.18batch/s]

Training batches on cuda:   7%|▋         | 11/153 [00:00<00:04, 30.93batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:04, 32.53batch/s]

Training batches on cuda:  12%|█▏        | 19/153 [00:00<00:03, 33.74batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:04, 30.74batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:03, 31.96batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:03, 32.81batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:01<00:03, 34.22batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:01<00:03, 35.05batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:01<00:03, 35.87batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:01<00:02, 36.36batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:01<00:02, 36.80batch/s]

Training batches on cuda:  35%|███▌      | 54/153 [00:01<00:02, 37.00batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:02, 37.06batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:02, 35.84batch/s]

Training batches on cuda:  43%|████▎     | 66/153 [00:01<00:02, 35.99batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:02, 36.15batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:02<00:02, 36.43batch/s]

Training batches on cuda:  51%|█████     | 78/153 [00:02<00:02, 35.92batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:02<00:01, 36.46batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:02<00:01, 35.53batch/s]

Training batches on cuda:  59%|█████▉    | 90/153 [00:02<00:01, 36.05batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:02<00:01, 35.50batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:02<00:01, 36.68batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:02<00:01, 36.87batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:02<00:01, 36.33batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:03<00:01, 36.22batch/s]

Training batches on cuda:  75%|███████▍  | 114/153 [00:03<00:01, 35.88batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:03<00:00, 35.78batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:03<00:00, 35.85batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:03<00:00, 35.79batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:03<00:00, 35.52batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:03<00:00, 35.41batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:03<00:00, 35.01batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:04<00:00, 34.51batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:04<00:00, 34.78batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:04<00:00, 34.61batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:36,  4.62s/epoch, loss=3.62e-5, prev_loss=3.48e-5]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:32,  4.60s/epoch, loss=3.62e-5, prev_loss=3.48e-5]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:06, 24.63batch/s]

Training batches on cuda:   5%|▍         | 7/153 [00:00<00:05, 27.14batch/s]

Training batches on cuda:   7%|▋         | 11/153 [00:00<00:04, 29.11batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:04, 31.49batch/s]

Training batches on cuda:  12%|█▏        | 19/153 [00:00<00:04, 33.24batch/s]

Training batches on cuda:  15%|█▌        | 23/153 [00:00<00:03, 34.23batch/s]

Training batches on cuda:  18%|█▊        | 27/153 [00:00<00:03, 34.73batch/s]

Training batches on cuda:  20%|██        | 31/153 [00:00<00:03, 34.60batch/s]

Training batches on cuda:  23%|██▎       | 35/153 [00:01<00:03, 34.99batch/s]

Training batches on cuda:  25%|██▌       | 39/153 [00:01<00:03, 33.83batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:01<00:03, 32.35batch/s]

Training batches on cuda:  31%|███       | 47/153 [00:01<00:03, 32.20batch/s]

Training batches on cuda:  33%|███▎      | 51/153 [00:01<00:03, 32.58batch/s]

Training batches on cuda:  36%|███▌      | 55/153 [00:01<00:03, 32.41batch/s]

Training batches on cuda:  39%|███▊      | 59/153 [00:01<00:02, 33.37batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:02, 34.24batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:02, 34.93batch/s]

Training batches on cuda:  46%|████▋     | 71/153 [00:02<00:02, 34.92batch/s]

Training batches on cuda:  49%|████▉     | 75/153 [00:02<00:02, 34.01batch/s]

Training batches on cuda:  52%|█████▏    | 79/153 [00:02<00:02, 35.05batch/s]

Training batches on cuda:  54%|█████▍    | 83/153 [00:02<00:01, 36.14batch/s]

Training batches on cuda:  57%|█████▋    | 87/153 [00:02<00:01, 36.17batch/s]

Training batches on cuda:  59%|█████▉    | 91/153 [00:02<00:01, 36.22batch/s]

Training batches on cuda:  62%|██████▏   | 95/153 [00:02<00:01, 35.64batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:02<00:01, 36.16batch/s]

Training batches on cuda:  67%|██████▋   | 103/153 [00:02<00:01, 36.29batch/s]

Training batches on cuda:  70%|██████▉   | 107/153 [00:03<00:01, 36.56batch/s]

Training batches on cuda:  73%|███████▎  | 111/153 [00:03<00:01, 36.39batch/s]

Training batches on cuda:  75%|███████▌  | 115/153 [00:03<00:01, 36.00batch/s]

Training batches on cuda:  78%|███████▊  | 119/153 [00:03<00:00, 35.91batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:03<00:00, 35.57batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:03<00:00, 36.27batch/s]

Training batches on cuda:  86%|████████▌ | 131/153 [00:03<00:00, 36.63batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:03<00:00, 36.19batch/s]

Training batches on cuda:  91%|█████████ | 139/153 [00:03<00:00, 36.69batch/s]

Training batches on cuda:  93%|█████████▎| 143/153 [00:04<00:00, 36.67batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:04<00:00, 36.07batch/s]

Training batches on cuda:  99%|█████████▊| 151/153 [00:04<00:00, 36.09batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:32,  4.60s/epoch, loss=3.91e-5, prev_loss=3.62e-5]
Training epochs on cuda:  40%|████      | 4/10 [00:18<00:27,  4.61s/epoch, loss=3.91e-5, prev_loss=3.62e-5]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:06, 24.02batch/s]

Training batches on cuda:   5%|▍         | 7/153 [00:00<00:05, 26.65batch/s]

Training batches on cuda:   7%|▋         | 11/153 [00:00<00:04, 28.92batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:04, 29.17batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:04, 30.46batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:04, 31.97batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:03, 33.16batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:03, 33.79batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:01<00:03, 34.17batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:01<00:03, 34.27batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:01<00:03, 34.98batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:01<00:03, 34.46batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:01<00:02, 34.35batch/s]

Training batches on cuda:  35%|███▌      | 54/153 [00:01<00:02, 34.97batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:02, 35.85batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:02, 36.17batch/s]

Training batches on cuda:  43%|████▎     | 66/153 [00:01<00:02, 36.64batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:02<00:02, 36.79batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:02<00:02, 36.87batch/s]

Training batches on cuda:  51%|█████     | 78/153 [00:02<00:02, 37.30batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:02<00:01, 36.98batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:02<00:01, 37.20batch/s]

Training batches on cuda:  59%|█████▉    | 90/153 [00:02<00:01, 36.33batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:02<00:01, 36.50batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:02<00:01, 36.86batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:02<00:01, 37.04batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:02<00:01, 37.32batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:03<00:01, 37.38batch/s]

Training batches on cuda:  75%|███████▍  | 114/153 [00:03<00:01, 38.02batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:03<00:00, 37.54batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:03<00:00, 37.58batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:03<00:00, 37.82batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:03<00:00, 37.64batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:03<00:00, 37.85batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:03<00:00, 37.66batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:03<00:00, 37.34batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:04<00:00, 37.18batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:04<00:00, 37.35batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:27,  4.61s/epoch, loss=3.98e-5, prev_loss=3.91e-5]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:22,  4.58s/epoch, loss=3.98e-5, prev_loss=3.91e-5]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/153 [00:00<00:15,  9.51batch/s]

Training batches on cuda:   3%|▎         | 5/153 [00:00<00:12, 12.14batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:09, 15.15batch/s]

Training batches on cuda:   8%|▊         | 13/153 [00:00<00:07, 18.40batch/s]

Training batches on cuda:  11%|█         | 17/153 [00:00<00:06, 21.69batch/s]

Training batches on cuda:  14%|█▎        | 21/153 [00:00<00:05, 24.77batch/s]

Training batches on cuda:  16%|█▋        | 25/153 [00:00<00:04, 27.42batch/s]

Training batches on cuda:  19%|█▉        | 29/153 [00:00<00:04, 29.39batch/s]

Training batches on cuda:  22%|██▏       | 33/153 [00:00<00:03, 31.31batch/s]

Training batches on cuda:  24%|██▍       | 37/153 [00:01<00:03, 32.88batch/s]

Training batches on cuda:  27%|██▋       | 41/153 [00:01<00:03, 33.78batch/s]

Training batches on cuda:  29%|██▉       | 45/153 [00:01<00:03, 34.49batch/s]

Training batches on cuda:  32%|███▏      | 49/153 [00:01<00:02, 35.16batch/s]

Training batches on cuda:  35%|███▍      | 53/153 [00:01<00:02, 35.65batch/s]

Training batches on cuda:  37%|███▋      | 57/153 [00:01<00:02, 34.99batch/s]

Training batches on cuda:  40%|███▉      | 61/153 [00:01<00:02, 35.51batch/s]

Training batches on cuda:  42%|████▏     | 65/153 [00:01<00:02, 35.67batch/s]

Training batches on cuda:  45%|████▌     | 69/153 [00:01<00:02, 34.78batch/s]

Training batches on cuda:  48%|████▊     | 73/153 [00:02<00:02, 34.84batch/s]

Training batches on cuda:  50%|█████     | 77/153 [00:02<00:02, 31.33batch/s]

Training batches on cuda:  53%|█████▎    | 81/153 [00:02<00:02, 31.25batch/s]

Training batches on cuda:  56%|█████▌    | 85/153 [00:02<00:02, 32.21batch/s]

Training batches on cuda:  58%|█████▊    | 89/153 [00:02<00:01, 32.97batch/s]

Training batches on cuda:  61%|██████    | 93/153 [00:02<00:01, 32.66batch/s]

Training batches on cuda:  63%|██████▎   | 97/153 [00:02<00:01, 33.07batch/s]

Training batches on cuda:  66%|██████▌   | 101/153 [00:02<00:01, 33.43batch/s]

Training batches on cuda:  69%|██████▊   | 105/153 [00:03<00:01, 33.35batch/s]

Training batches on cuda:  71%|███████   | 109/153 [00:03<00:01, 33.50batch/s]

Training batches on cuda:  74%|███████▍  | 113/153 [00:03<00:01, 33.89batch/s]

Training batches on cuda:  76%|███████▋  | 117/153 [00:03<00:01, 33.47batch/s]

Training batches on cuda:  79%|███████▉  | 121/153 [00:03<00:00, 33.76batch/s]

Training batches on cuda:  82%|████████▏ | 125/153 [00:03<00:00, 33.56batch/s]

Training batches on cuda:  84%|████████▍ | 129/153 [00:03<00:00, 33.67batch/s]

Training batches on cuda:  87%|████████▋ | 133/153 [00:03<00:00, 33.57batch/s]

Training batches on cuda:  90%|████████▉ | 137/153 [00:04<00:00, 33.74batch/s]

Training batches on cuda:  92%|█████████▏| 141/153 [00:04<00:00, 33.68batch/s]

Training batches on cuda:  95%|█████████▍| 145/153 [00:04<00:00, 33.81batch/s]

Training batches on cuda:  97%|█████████▋| 149/153 [00:04<00:00, 33.93batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:04<00:00, 33.96batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:27<00:22,  4.58s/epoch, loss=3.9e-5, prev_loss=3.98e-5] 
Training epochs on cuda:  60%|██████    | 6/10 [00:27<00:18,  4.65s/epoch, loss=3.9e-5, prev_loss=3.98e-5]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:06, 24.68batch/s]

Training batches on cuda:   5%|▍         | 7/153 [00:00<00:05, 27.43batch/s]

Training batches on cuda:   7%|▋         | 11/153 [00:00<00:04, 30.08batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:04, 32.10batch/s]

Training batches on cuda:  12%|█▏        | 19/153 [00:00<00:03, 33.86batch/s]

Training batches on cuda:  15%|█▌        | 23/153 [00:00<00:03, 35.19batch/s]

Training batches on cuda:  18%|█▊        | 27/153 [00:00<00:03, 36.12batch/s]

Training batches on cuda:  20%|██        | 31/153 [00:00<00:03, 36.56batch/s]

Training batches on cuda:  23%|██▎       | 35/153 [00:00<00:03, 37.34batch/s]

Training batches on cuda:  25%|██▌       | 39/153 [00:01<00:03, 37.09batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:01<00:02, 37.46batch/s]

Training batches on cuda:  31%|███       | 47/153 [00:01<00:02, 37.41batch/s]

Training batches on cuda:  33%|███▎      | 51/153 [00:01<00:02, 36.49batch/s]

Training batches on cuda:  36%|███▌      | 55/153 [00:01<00:02, 36.33batch/s]

Training batches on cuda:  39%|███▊      | 59/153 [00:01<00:02, 36.35batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:02, 36.49batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:02, 36.26batch/s]

Training batches on cuda:  46%|████▋     | 71/153 [00:01<00:02, 36.30batch/s]

Training batches on cuda:  49%|████▉     | 75/153 [00:02<00:02, 35.79batch/s]

Training batches on cuda:  52%|█████▏    | 79/153 [00:02<00:02, 36.02batch/s]

Training batches on cuda:  54%|█████▍    | 83/153 [00:02<00:01, 36.22batch/s]

Training batches on cuda:  57%|█████▋    | 87/153 [00:02<00:01, 36.15batch/s]

Training batches on cuda:  59%|█████▉    | 91/153 [00:02<00:01, 33.84batch/s]

Training batches on cuda:  62%|██████▏   | 95/153 [00:02<00:01, 34.96batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:02<00:01, 35.01batch/s]

Training batches on cuda:  67%|██████▋   | 103/153 [00:02<00:01, 34.98batch/s]

Training batches on cuda:  70%|██████▉   | 107/153 [00:02<00:01, 35.09batch/s]

Training batches on cuda:  73%|███████▎  | 111/153 [00:03<00:01, 34.65batch/s]

Training batches on cuda:  75%|███████▌  | 115/153 [00:03<00:01, 34.09batch/s]

Training batches on cuda:  78%|███████▊  | 119/153 [00:03<00:00, 34.54batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:03<00:00, 35.07batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:03<00:00, 35.46batch/s]

Training batches on cuda:  86%|████████▌ | 131/153 [00:03<00:00, 35.65batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:03<00:00, 35.57batch/s]

Training batches on cuda:  91%|█████████ | 139/153 [00:03<00:00, 34.41batch/s]

Training batches on cuda:  93%|█████████▎| 143/153 [00:04<00:00, 34.84batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:04<00:00, 33.93batch/s]

Training batches on cuda:  99%|█████████▊| 151/153 [00:04<00:00, 32.24batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:32<00:18,  4.65s/epoch, loss=3.79e-5, prev_loss=3.9e-5]
Training epochs on cuda:  70%|███████   | 7/10 [00:32<00:13,  4.64s/epoch, loss=3.79e-5, prev_loss=3.9e-5]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:06, 24.56batch/s]

Training batches on cuda:   5%|▍         | 7/153 [00:00<00:05, 27.29batch/s]

Training batches on cuda:   7%|▋         | 11/153 [00:00<00:04, 29.56batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:04, 31.65batch/s]

Training batches on cuda:  12%|█▏        | 19/153 [00:00<00:04, 32.74batch/s]

Training batches on cuda:  15%|█▌        | 23/153 [00:00<00:03, 33.73batch/s]

Training batches on cuda:  18%|█▊        | 27/153 [00:00<00:03, 34.18batch/s]

Training batches on cuda:  20%|██        | 31/153 [00:00<00:03, 34.47batch/s]

Training batches on cuda:  23%|██▎       | 35/153 [00:01<00:03, 34.44batch/s]

Training batches on cuda:  25%|██▌       | 39/153 [00:01<00:03, 34.00batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:01<00:03, 33.73batch/s]

Training batches on cuda:  31%|███       | 47/153 [00:01<00:03, 33.18batch/s]

Training batches on cuda:  33%|███▎      | 51/153 [00:01<00:03, 33.63batch/s]

Training batches on cuda:  36%|███▌      | 55/153 [00:01<00:02, 33.78batch/s]

Training batches on cuda:  39%|███▊      | 59/153 [00:01<00:02, 33.94batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:02, 32.88batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:02, 33.10batch/s]

Training batches on cuda:  46%|████▋     | 71/153 [00:02<00:02, 31.75batch/s]

Training batches on cuda:  49%|████▉     | 75/153 [00:02<00:02, 31.73batch/s]

Training batches on cuda:  52%|█████▏    | 79/153 [00:02<00:02, 32.22batch/s]

Training batches on cuda:  54%|█████▍    | 83/153 [00:02<00:02, 31.25batch/s]

Training batches on cuda:  57%|█████▋    | 87/153 [00:02<00:02, 31.07batch/s]

Training batches on cuda:  59%|█████▉    | 91/153 [00:02<00:02, 30.68batch/s]

Training batches on cuda:  62%|██████▏   | 95/153 [00:02<00:02, 28.84batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:03<00:01, 29.94batch/s]

Training batches on cuda:  67%|██████▋   | 103/153 [00:03<00:01, 31.28batch/s]

Training batches on cuda:  70%|██████▉   | 107/153 [00:03<00:01, 32.06batch/s]

Training batches on cuda:  73%|███████▎  | 111/153 [00:03<00:01, 33.18batch/s]

Training batches on cuda:  75%|███████▌  | 115/153 [00:03<00:01, 33.82batch/s]

Training batches on cuda:  78%|███████▊  | 119/153 [00:03<00:01, 33.81batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:03<00:00, 34.30batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:03<00:00, 34.58batch/s]

Training batches on cuda:  86%|████████▌ | 131/153 [00:03<00:00, 34.70batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:04<00:00, 35.19batch/s]

Training batches on cuda:  91%|█████████ | 139/153 [00:04<00:00, 35.42batch/s]

Training batches on cuda:  93%|█████████▎| 143/153 [00:04<00:00, 34.33batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:04<00:00, 34.27batch/s]

Training batches on cuda:  99%|█████████▊| 151/153 [00:04<00:00, 34.73batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:37<00:13,  4.64s/epoch, loss=3.93e-5, prev_loss=3.79e-5]
Training epochs on cuda:  80%|████████  | 8/10 [00:37<00:09,  4.69s/epoch, loss=3.93e-5, prev_loss=3.79e-5]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/153 [00:00<00:08, 17.95batch/s]

Training batches on cuda:   4%|▍         | 6/153 [00:00<00:06, 21.00batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:05, 24.15batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:05, 26.81batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:04, 29.71batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:04, 31.84batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:03, 33.42batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:03, 34.17batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:00<00:03, 35.23batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:01<00:03, 35.74batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:01<00:03, 36.04batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:01<00:02, 36.16batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:01<00:02, 34.53batch/s]

Training batches on cuda:  35%|███▌      | 54/153 [00:01<00:02, 34.01batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:02, 35.19batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:02, 36.15batch/s]

Training batches on cuda:  43%|████▎     | 66/153 [00:01<00:02, 36.84batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:02, 37.07batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:02<00:02, 36.85batch/s]

Training batches on cuda:  51%|█████     | 78/153 [00:02<00:02, 36.39batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:02<00:01, 36.48batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:02<00:01, 35.83batch/s]

Training batches on cuda:  59%|█████▉    | 90/153 [00:02<00:01, 36.04batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:02<00:01, 35.46batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:02<00:01, 34.24batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:02<00:01, 34.94batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:02<00:01, 35.40batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:03<00:01, 32.14batch/s]

Training batches on cuda:  75%|███████▍  | 114/153 [00:03<00:01, 32.80batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:03<00:01, 33.80batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:03<00:00, 34.65batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:03<00:00, 35.16batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:03<00:00, 35.56batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:03<00:00, 35.80batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:03<00:00, 35.99batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:04<00:00, 35.99batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:04<00:00, 36.04batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:04<00:00, 36.26batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:41<00:09,  4.69s/epoch, loss=4.12e-5, prev_loss=3.93e-5]
Training epochs on cuda:  90%|█████████ | 9/10 [00:41<00:04,  4.67s/epoch, loss=4.12e-5, prev_loss=3.93e-5]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/153 [00:00<00:07, 19.19batch/s]

Training batches on cuda:   4%|▍         | 6/153 [00:00<00:06, 22.17batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:05, 25.34batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:04, 28.28batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:04, 30.49batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:04, 32.27batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:03, 32.45batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:03, 33.31batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:00<00:03, 34.35batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:01<00:03, 33.83batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:01<00:03, 34.49batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:01<00:03, 34.99batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:01<00:03, 33.73batch/s]

Training batches on cuda:  35%|███▌      | 54/153 [00:01<00:02, 34.48batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:02, 34.31batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:02, 35.07batch/s]

Training batches on cuda:  43%|████▎     | 66/153 [00:01<00:02, 35.30batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:02<00:02, 35.56batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:02<00:02, 35.04batch/s]

Training batches on cuda:  51%|█████     | 78/153 [00:02<00:02, 35.65batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:02<00:01, 35.87batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:02<00:01, 36.07batch/s]

Training batches on cuda:  59%|█████▉    | 90/153 [00:02<00:01, 35.92batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:02<00:01, 36.26batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:02<00:01, 36.28batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:02<00:01, 36.37batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:03<00:01, 36.59batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:03<00:01, 36.85batch/s]

Training batches on cuda:  75%|███████▍  | 114/153 [00:03<00:01, 36.91batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:03<00:00, 35.22batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:03<00:00, 35.64batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:03<00:00, 36.15batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:03<00:00, 36.39batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:03<00:00, 36.34batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:03<00:00, 36.44batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:03<00:00, 36.62batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:04<00:00, 36.51batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:04<00:00, 35.84batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:46<00:04,  4.67s/epoch, loss=4.02e-5, prev_loss=4.12e-5]2021-09-16:17:29:13,86 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:29:14,124 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:29:29,108 INFO     [evaluator.py:657] Evaluation took 14.98s seconds
2021-09-16:17:29:29,254 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00,  9.49s/epoch, loss=4.02e-5, prev_loss=4.12e-5]
Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00,  6.25s/epoch, loss=4.02e-5, prev_loss=4.12e-5]
2021-09-16:17:29:29,254 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpd843idb0'
2021-09-16:17:29:29,306 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpd843idb0' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:19, 2.03ktriple/s]
Evaluating on cuda:   1%|▏         | 512/38.8k [00:00<00:17, 2.13ktriple/s]
Evaluating on cuda:   3%|▎         | 1.02k/38.8k [00:00<00:16, 2.26ktriple/s]
Evaluating on cuda:   4%|▍         | 1.54k/38.8k [00:00<00:15, 2.35ktriple/s]
Evaluating on cuda:   5%|▌         | 2.05k/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   7%|▋         | 2.56k/38.8k [00:01<00:14, 2.49ktriple/s]
Evaluating on cuda:   8%|▊         | 3.07k/38.8k [00:01<00:14, 2.53ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:14, 2.53ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.56ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:17:29:44,193 INFO     [evaluator.py:657] Evaluation took 14.88s seconds
[I 2021-09-16 17:29:44,198] Trial 5 finished with value: -0.00044433768578566735 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.8134851776643713, 'optimizer.lr': 0.0943822633291318, 'negative_sampler.num_negs_per_pos': 26, 'training.batch_size': 2032}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:29:44,201 WARNING  [api.py:823] No random seed is specified. Setting to 2430329720.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 14/492 [00:00<00:03, 136.00batch/s]

Training batches on cuda:   7%|▋         | 32/492 [00:00<00:03, 146.48batch/s]

Training batches on cuda:  10%|█         | 51/492 [00:00<00:02, 155.31batch/s]

Training batches on cuda:  14%|█▍        | 69/492 [00:00<00:02, 161.85batch/s]

Training batches on cuda:  18%|█▊        | 87/492 [00:00<00:02, 166.62batch/s]

Training batches on cuda:  21%|██▏       | 105/492 [00:00<00:02, 168.48batch/s]

Training batches on cuda:  25%|██▌       | 123/492 [00:00<00:02, 170.76batch/s]

Training batches on cuda:  28%|██▊       | 140/492 [00:00<00:02, 169.90batch/s]

Training batches on cuda:  32%|███▏      | 157/492 [00:00<00:01, 168.19batch/s]

Training batches on cuda:  35%|███▌      | 174/492 [00:01<00:01, 168.09batch/s]

Training batches on cuda:  39%|███▉      | 191/492 [00:01<00:01, 167.08batch/s]

Training batches on cuda:  42%|████▏     | 208/492 [00:01<00:01, 164.72batch/s]

Training batches on cuda:  46%|████▌     | 225/492 [00:01<00:01, 163.13batch/s]

Training batches on cuda:  49%|████▉     | 242/492 [00:01<00:01, 163.20batch/s]

Training batches on cuda:  53%|█████▎    | 259/492 [00:01<00:01, 161.80batch/s]

Training batches on cuda:  56%|█████▌    | 276/492 [00:01<00:01, 162.15batch/s]

Training batches on cuda:  60%|█████▉    | 295/492 [00:01<00:01, 167.74batch/s]

Training batches on cuda:  64%|██████▍   | 314/492 [00:01<00:01, 171.10batch/s]

Training batches on cuda:  67%|██████▋   | 332/492 [00:01<00:00, 167.07batch/s]

Training batches on cuda:  71%|███████   | 349/492 [00:02<00:00, 156.79batch/s]

Training batches on cuda:  74%|███████▍  | 365/492 [00:02<00:00, 146.45batch/s]

Training batches on cuda:  77%|███████▋  | 380/492 [00:02<00:00, 137.12batch/s]

Training batches on cuda:  80%|████████  | 395/492 [00:02<00:00, 132.54batch/s]

Training batches on cuda:  83%|████████▎ | 409/492 [00:02<00:00, 129.92batch/s]

Training batches on cuda:  86%|████████▋ | 425/492 [00:02<00:00, 136.82batch/s]

Training batches on cuda:  90%|████████▉ | 441/492 [00:02<00:00, 142.92batch/s]

Training batches on cuda:  93%|█████████▎| 456/492 [00:02<00:00, 131.11batch/s]

Training batches on cuda:  96%|█████████▋| 474/492 [00:03<00:00, 141.16batch/s]

Training batches on cuda: 100%|█████████▉| 490/492 [00:03<00:00, 145.99batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.0016, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:30,  3.42s/epoch, loss=0.0016, prev_loss=nan]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 13/492 [00:00<00:03, 122.56batch/s]

Training batches on cuda:   6%|▌         | 29/492 [00:00<00:03, 128.95batch/s]

Training batches on cuda:   9%|▉         | 44/492 [00:00<00:03, 132.14batch/s]

Training batches on cuda:  12%|█▏        | 59/492 [00:00<00:03, 135.52batch/s]

Training batches on cuda:  15%|█▌        | 74/492 [00:00<00:03, 138.00batch/s]

Training batches on cuda:  18%|█▊        | 88/492 [00:00<00:02, 137.04batch/s]

Training batches on cuda:  21%|██        | 102/492 [00:00<00:02, 136.23batch/s]

Training batches on cuda:  24%|██▎       | 116/492 [00:00<00:02, 137.11batch/s]

Training batches on cuda:  27%|██▋       | 132/492 [00:00<00:02, 142.65batch/s]

Training batches on cuda:  30%|███       | 149/492 [00:01<00:02, 149.58batch/s]

Training batches on cuda:  34%|███▎      | 166/492 [00:01<00:02, 154.42batch/s]

Training batches on cuda:  37%|███▋      | 182/492 [00:01<00:01, 155.44batch/s]

Training batches on cuda:  41%|████      | 201/492 [00:01<00:01, 162.45batch/s]

Training batches on cuda:  45%|████▍     | 219/492 [00:01<00:01, 165.48batch/s]

Training batches on cuda:  48%|████▊     | 236/492 [00:01<00:01, 162.35batch/s]

Training batches on cuda:  51%|█████▏    | 253/492 [00:01<00:01, 152.79batch/s]

Training batches on cuda:  55%|█████▍    | 269/492 [00:01<00:01, 152.80batch/s]

Training batches on cuda:  58%|█████▊    | 286/492 [00:01<00:01, 156.59batch/s]

Training batches on cuda:  61%|██████▏   | 302/492 [00:02<00:01, 154.33batch/s]

Training batches on cuda:  65%|██████▌   | 320/492 [00:02<00:01, 159.16batch/s]

Training batches on cuda:  68%|██████▊   | 337/492 [00:02<00:00, 161.22batch/s]

Training batches on cuda:  72%|███████▏  | 354/492 [00:02<00:00, 163.59batch/s]

Training batches on cuda:  75%|███████▌  | 371/492 [00:02<00:00, 161.89batch/s]

Training batches on cuda:  79%|███████▉  | 388/492 [00:02<00:00, 163.82batch/s]

Training batches on cuda:  82%|████████▏ | 405/492 [00:02<00:00, 162.44batch/s]

Training batches on cuda:  86%|████████▌ | 422/492 [00:02<00:00, 159.59batch/s]

Training batches on cuda:  89%|████████▉ | 438/492 [00:02<00:00, 154.75batch/s]

Training batches on cuda:  92%|█████████▏| 454/492 [00:02<00:00, 153.02batch/s]

Training batches on cuda:  96%|█████████▌| 470/492 [00:03<00:00, 149.38batch/s]

Training batches on cuda:  99%|█████████▉| 488/492 [00:03<00:00, 155.94batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:30,  3.42s/epoch, loss=0.0016, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:27,  3.46s/epoch, loss=0.0016, prev_loss=nan]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/492 [00:00<00:04, 108.94batch/s]

Training batches on cuda:   5%|▌         | 27/492 [00:00<00:03, 119.20batch/s]

Training batches on cuda:   9%|▊         | 43/492 [00:00<00:03, 127.10batch/s]

Training batches on cuda:  12%|█▏        | 59/492 [00:00<00:03, 133.51batch/s]

Training batches on cuda:  15%|█▌        | 74/492 [00:00<00:03, 137.89batch/s]

Training batches on cuda:  18%|█▊        | 90/492 [00:00<00:02, 141.47batch/s]

Training batches on cuda:  21%|██        | 104/492 [00:00<00:02, 140.62batch/s]

Training batches on cuda:  24%|██▍       | 119/492 [00:00<00:02, 141.02batch/s]

Training batches on cuda:  27%|██▋       | 133/492 [00:00<00:02, 126.74batch/s]

Training batches on cuda:  30%|███       | 148/492 [00:01<00:02, 130.68batch/s]

Training batches on cuda:  33%|███▎      | 163/492 [00:01<00:02, 133.76batch/s]

Training batches on cuda:  36%|███▌      | 178/492 [00:01<00:02, 137.74batch/s]

Training batches on cuda:  39%|███▉      | 192/492 [00:01<00:02, 137.09batch/s]

Training batches on cuda:  42%|████▏     | 206/492 [00:01<00:02, 137.84batch/s]

Training batches on cuda:  45%|████▍     | 220/492 [00:01<00:02, 126.89batch/s]

Training batches on cuda:  48%|████▊     | 236/492 [00:01<00:01, 133.34batch/s]

Training batches on cuda:  51%|█████     | 252/492 [00:01<00:01, 138.56batch/s]

Training batches on cuda:  54%|█████▍    | 268/492 [00:01<00:01, 143.20batch/s]

Training batches on cuda:  58%|█████▊    | 285/492 [00:02<00:01, 149.60batch/s]

Training batches on cuda:  61%|██████▏   | 302/492 [00:02<00:01, 155.16batch/s]

Training batches on cuda:  65%|██████▍   | 319/492 [00:02<00:01, 158.51batch/s]

Training batches on cuda:  68%|██████▊   | 337/492 [00:02<00:00, 161.98batch/s]

Training batches on cuda:  72%|███████▏  | 354/492 [00:02<00:00, 162.55batch/s]

Training batches on cuda:  76%|███████▌  | 372/492 [00:02<00:00, 165.21batch/s]

Training batches on cuda:  79%|███████▉  | 390/492 [00:02<00:00, 167.05batch/s]

Training batches on cuda:  83%|████████▎ | 407/492 [00:02<00:00, 166.01batch/s]

Training batches on cuda:  86%|████████▋ | 425/492 [00:02<00:00, 167.92batch/s]

Training batches on cuda:  90%|█████████ | 443/492 [00:02<00:00, 169.38batch/s]

Training batches on cuda:  93%|█████████▎| 460/492 [00:03<00:00, 167.01batch/s]

Training batches on cuda:  97%|█████████▋| 477/492 [00:03<00:00, 167.46batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:10<00:27,  3.46s/epoch, loss=0.0016, prev_loss=0.0016]
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:24,  3.48s/epoch, loss=0.0016, prev_loss=0.0016]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 13/492 [00:00<00:03, 125.94batch/s]

Training batches on cuda:   7%|▋         | 32/492 [00:00<00:03, 139.10batch/s]

Training batches on cuda:  10%|█         | 50/492 [00:00<00:02, 148.86batch/s]

Training batches on cuda:  13%|█▎        | 64/492 [00:00<00:02, 144.33batch/s]

Training batches on cuda:  16%|█▌        | 79/492 [00:00<00:02, 145.25batch/s]

Training batches on cuda:  19%|█▉        | 95/492 [00:00<00:02, 146.72batch/s]

Training batches on cuda:  23%|██▎       | 111/492 [00:00<00:02, 149.36batch/s]

Training batches on cuda:  26%|██▌       | 126/492 [00:00<00:02, 149.49batch/s]

Training batches on cuda:  29%|██▉       | 142/492 [00:00<00:02, 151.10batch/s]

Training batches on cuda:  32%|███▏      | 159/492 [00:01<00:02, 156.25batch/s]

Training batches on cuda:  36%|███▌      | 175/492 [00:01<00:02, 156.16batch/s]

Training batches on cuda:  39%|███▉      | 191/492 [00:01<00:01, 154.60batch/s]

Training batches on cuda:  42%|████▏     | 207/492 [00:01<00:01, 154.22batch/s]

Training batches on cuda:  45%|████▌     | 223/492 [00:01<00:01, 150.58batch/s]

Training batches on cuda:  49%|████▊     | 239/492 [00:01<00:01, 150.28batch/s]

Training batches on cuda:  52%|█████▏    | 256/492 [00:01<00:01, 155.26batch/s]

Training batches on cuda:  55%|█████▌    | 273/492 [00:01<00:01, 158.49batch/s]

Training batches on cuda:  59%|█████▉    | 291/492 [00:01<00:01, 163.16batch/s]

Training batches on cuda:  63%|██████▎   | 308/492 [00:01<00:01, 163.52batch/s]

Training batches on cuda:  66%|██████▋   | 326/492 [00:02<00:00, 166.34batch/s]

Training batches on cuda:  70%|██████▉   | 344/492 [00:02<00:00, 169.37batch/s]

Training batches on cuda:  73%|███████▎  | 361/492 [00:02<00:00, 165.46batch/s]

Training batches on cuda:  77%|███████▋  | 378/492 [00:02<00:00, 158.61batch/s]

Training batches on cuda:  80%|████████  | 394/492 [00:02<00:00, 130.27batch/s]

Training batches on cuda:  83%|████████▎ | 408/492 [00:02<00:00, 127.48batch/s]

Training batches on cuda:  86%|████████▌ | 422/492 [00:02<00:00, 126.13batch/s]

Training batches on cuda:  89%|████████▊ | 436/492 [00:02<00:00, 126.11batch/s]

Training batches on cuda:  91%|█████████▏| 450/492 [00:03<00:00, 128.15batch/s]

Training batches on cuda:  95%|█████████▍| 465/492 [00:03<00:00, 132.79batch/s]

Training batches on cuda:  97%|█████████▋| 479/492 [00:03<00:00, 134.86batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:14<00:24,  3.48s/epoch, loss=0.0016, prev_loss=0.0016]
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:21,  3.53s/epoch, loss=0.0016, prev_loss=0.0016]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 14/492 [00:00<00:03, 133.47batch/s]

Training batches on cuda:   7%|▋         | 34/492 [00:00<00:03, 148.16batch/s]

Training batches on cuda:  11%|█         | 55/492 [00:00<00:02, 160.71batch/s]

Training batches on cuda:  15%|█▌        | 76/492 [00:00<00:02, 171.05batch/s]

Training batches on cuda:  20%|█▉        | 97/492 [00:00<00:02, 179.02batch/s]

Training batches on cuda:  24%|██▍       | 117/492 [00:00<00:02, 184.82batch/s]

Training batches on cuda:  28%|██▊       | 137/492 [00:00<00:01, 188.59batch/s]

Training batches on cuda:  32%|███▏      | 157/492 [00:00<00:01, 191.69batch/s]

Training batches on cuda:  36%|███▌      | 177/492 [00:00<00:01, 192.77batch/s]

Training batches on cuda:  40%|███▉      | 196/492 [00:01<00:01, 191.64batch/s]

Training batches on cuda:  44%|████▎     | 215/492 [00:01<00:01, 180.94batch/s]

Training batches on cuda:  48%|████▊     | 234/492 [00:01<00:01, 173.85batch/s]

Training batches on cuda:  51%|█████     | 252/492 [00:01<00:01, 170.81batch/s]

Training batches on cuda:  55%|█████▍    | 270/492 [00:01<00:01, 168.44batch/s]

Training batches on cuda:  58%|█████▊    | 287/492 [00:01<00:01, 166.28batch/s]

Training batches on cuda:  62%|██████▏   | 304/492 [00:01<00:01, 153.52batch/s]

Training batches on cuda:  65%|██████▌   | 320/492 [00:01<00:01, 153.79batch/s]

Training batches on cuda:  68%|██████▊   | 336/492 [00:01<00:01, 153.77batch/s]

Training batches on cuda:  72%|███████▏  | 352/492 [00:02<00:00, 152.19batch/s]

Training batches on cuda:  75%|███████▍  | 368/492 [00:02<00:00, 146.96batch/s]

Training batches on cuda:  78%|███████▊  | 383/492 [00:02<00:00, 146.24batch/s]

Training batches on cuda:  81%|████████  | 398/492 [00:02<00:00, 138.68batch/s]

Training batches on cuda:  84%|████████▎ | 412/492 [00:02<00:00, 127.35batch/s]

Training batches on cuda:  87%|████████▋ | 426/492 [00:02<00:00, 124.27batch/s]

Training batches on cuda:  89%|████████▉ | 440/492 [00:02<00:00, 127.88batch/s]

Training batches on cuda:  92%|█████████▏| 455/492 [00:02<00:00, 132.26batch/s]

Training batches on cuda:  95%|█████████▌| 469/492 [00:02<00:00, 129.56batch/s]

Training batches on cuda:  98%|█████████▊| 483/492 [00:03<00:00, 131.04batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:21,  3.53s/epoch, loss=0.0016, prev_loss=0.0016]
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:17,  3.47s/epoch, loss=0.0016, prev_loss=0.0016]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 10/492 [00:00<00:04, 99.31batch/s]

Training batches on cuda:   5%|▌         | 25/492 [00:00<00:04, 109.07batch/s]

Training batches on cuda:   8%|▊         | 41/492 [00:00<00:03, 119.36batch/s]

Training batches on cuda:  12%|█▏        | 57/492 [00:00<00:03, 128.52batch/s]

Training batches on cuda:  15%|█▍        | 72/492 [00:00<00:03, 133.37batch/s]

Training batches on cuda:  17%|█▋        | 85/492 [00:00<00:03, 122.77batch/s]

Training batches on cuda:  20%|██        | 100/492 [00:00<00:03, 128.48batch/s]

Training batches on cuda:  23%|██▎       | 115/492 [00:00<00:02, 133.73batch/s]

Training batches on cuda:  26%|██▋       | 130/492 [00:00<00:02, 136.15batch/s]

Training batches on cuda:  29%|██▉       | 145/492 [00:01<00:02, 139.78batch/s]

Training batches on cuda:  33%|███▎      | 161/492 [00:01<00:02, 143.94batch/s]

Training batches on cuda:  36%|███▌      | 177/492 [00:01<00:02, 147.61batch/s]

Training batches on cuda:  39%|███▉      | 194/492 [00:01<00:01, 153.07batch/s]

Training batches on cuda:  43%|████▎     | 211/492 [00:01<00:01, 157.76batch/s]

Training batches on cuda:  46%|████▋     | 228/492 [00:01<00:01, 160.35batch/s]

Training batches on cuda:  50%|████▉     | 245/492 [00:01<00:01, 159.20batch/s]

Training batches on cuda:  53%|█████▎    | 262/492 [00:01<00:01, 162.25batch/s]

Training batches on cuda:  57%|█████▋    | 280/492 [00:01<00:01, 164.84batch/s]

Training batches on cuda:  60%|██████    | 297/492 [00:01<00:01, 164.18batch/s]

Training batches on cuda:  64%|██████▍   | 314/492 [00:02<00:01, 163.76batch/s]

Training batches on cuda:  67%|██████▋   | 331/492 [00:02<00:01, 160.93batch/s]

Training batches on cuda:  71%|███████   | 348/492 [00:02<00:00, 161.64batch/s]

Training batches on cuda:  74%|███████▍  | 365/492 [00:02<00:00, 160.21batch/s]

Training batches on cuda:  78%|███████▊  | 385/492 [00:02<00:00, 168.62batch/s]

Training batches on cuda:  82%|████████▏ | 404/492 [00:02<00:00, 173.80batch/s]

Training batches on cuda:  86%|████████▌ | 422/492 [00:02<00:00, 171.02batch/s]

Training batches on cuda:  89%|████████▉ | 440/492 [00:02<00:00, 155.36batch/s]

Training batches on cuda:  93%|█████████▎| 456/492 [00:02<00:00, 153.65batch/s]

Training batches on cuda:  96%|█████████▌| 472/492 [00:03<00:00, 154.59batch/s]

Training batches on cuda:  99%|█████████▉| 489/492 [00:03<00:00, 157.06batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:20<00:17,  3.47s/epoch, loss=0.0016, prev_loss=0.0016]
Training epochs on cuda:  60%|██████    | 6/10 [00:20<00:13,  3.46s/epoch, loss=0.0016, prev_loss=0.0016]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/492 [00:00<00:06, 74.74batch/s]

Training batches on cuda:   4%|▍         | 20/492 [00:00<00:05, 84.05batch/s]

Training batches on cuda:   7%|▋         | 35/492 [00:00<00:04, 95.61batch/s]

Training batches on cuda:  10%|█         | 51/492 [00:00<00:04, 107.60batch/s]

Training batches on cuda:  13%|█▎        | 66/492 [00:00<00:03, 115.14batch/s]

Training batches on cuda:  16%|█▌        | 79/492 [00:00<00:03, 119.08batch/s]

Training batches on cuda:  19%|█▉        | 95/492 [00:00<00:03, 128.77batch/s]

Training batches on cuda:  23%|██▎       | 112/492 [00:00<00:02, 138.78batch/s]

Training batches on cuda:  26%|██▌       | 129/492 [00:00<00:02, 145.78batch/s]

Training batches on cuda:  30%|██▉       | 146/492 [00:01<00:02, 151.22batch/s]

Training batches on cuda:  33%|███▎      | 163/492 [00:01<00:02, 155.19batch/s]

Training batches on cuda:  36%|███▋      | 179/492 [00:01<00:02, 153.63batch/s]

Training batches on cuda:  40%|███▉      | 195/492 [00:01<00:01, 151.51batch/s]

Training batches on cuda:  43%|████▎     | 211/492 [00:01<00:01, 152.47batch/s]

Training batches on cuda:  47%|████▋     | 229/492 [00:01<00:01, 157.41batch/s]

Training batches on cuda:  50%|████▉     | 245/492 [00:01<00:01, 157.59batch/s]

Training batches on cuda:  53%|█████▎    | 262/492 [00:01<00:01, 158.98batch/s]

Training batches on cuda:  57%|█████▋    | 278/492 [00:01<00:01, 157.63batch/s]

Training batches on cuda:  60%|█████▉    | 295/492 [00:01<00:01, 159.66batch/s]

Training batches on cuda:  63%|██████▎   | 311/492 [00:02<00:01, 155.03batch/s]

Training batches on cuda:  66%|██████▋   | 327/492 [00:02<00:01, 142.65batch/s]

Training batches on cuda:  70%|██████▉   | 343/492 [00:02<00:01, 145.67batch/s]

Training batches on cuda:  73%|███████▎  | 359/492 [00:02<00:00, 148.08batch/s]

Training batches on cuda:  76%|███████▌  | 375/492 [00:02<00:00, 148.79batch/s]

Training batches on cuda:  80%|███████▉  | 392/492 [00:02<00:00, 153.69batch/s]

Training batches on cuda:  83%|████████▎ | 409/492 [00:02<00:00, 156.95batch/s]

Training batches on cuda:  87%|████████▋ | 427/492 [00:02<00:00, 162.09batch/s]

Training batches on cuda:  90%|█████████ | 444/492 [00:02<00:00, 163.36batch/s]

Training batches on cuda:  94%|█████████▎| 461/492 [00:03<00:00, 162.24batch/s]

Training batches on cuda:  97%|█████████▋| 478/492 [00:03<00:00, 161.75batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:24<00:13,  3.46s/epoch, loss=0.0016, prev_loss=0.0016]
Training epochs on cuda:  70%|███████   | 7/10 [00:24<00:10,  3.45s/epoch, loss=0.0016, prev_loss=0.0016]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 10/492 [00:00<00:04, 98.45batch/s]

Training batches on cuda:   5%|▌         | 25/492 [00:00<00:04, 109.68batch/s]

Training batches on cuda:   9%|▊         | 42/492 [00:00<00:03, 121.19batch/s]

Training batches on cuda:  12%|█▏        | 57/492 [00:00<00:03, 128.44batch/s]

Training batches on cuda:  15%|█▍        | 72/492 [00:00<00:03, 133.82batch/s]

Training batches on cuda:  18%|█▊        | 88/492 [00:00<00:02, 139.97batch/s]

Training batches on cuda:  21%|██▏       | 105/492 [00:00<00:02, 147.28batch/s]

Training batches on cuda:  25%|██▍       | 122/492 [00:00<00:02, 151.82batch/s]

Training batches on cuda:  28%|██▊       | 137/492 [00:00<00:02, 150.58batch/s]

Training batches on cuda:  31%|███       | 152/492 [00:01<00:02, 146.56batch/s]

Training batches on cuda:  34%|███▍      | 168/492 [00:01<00:02, 150.15batch/s]

Training batches on cuda:  37%|███▋      | 184/492 [00:01<00:02, 151.63batch/s]

Training batches on cuda:  41%|████      | 200/492 [00:01<00:01, 151.17batch/s]

Training batches on cuda:  44%|████▍     | 216/492 [00:01<00:01, 148.69batch/s]

Training batches on cuda:  48%|████▊     | 234/492 [00:01<00:01, 153.90batch/s]

Training batches on cuda:  51%|█████     | 250/492 [00:01<00:01, 154.49batch/s]

Training batches on cuda:  54%|█████▍    | 266/492 [00:01<00:01, 150.73batch/s]

Training batches on cuda:  58%|█████▊    | 283/492 [00:01<00:01, 155.36batch/s]

Training batches on cuda:  61%|██████    | 299/492 [00:01<00:01, 155.06batch/s]

Training batches on cuda:  64%|██████▍   | 316/492 [00:02<00:01, 157.43batch/s]

Training batches on cuda:  67%|██████▋   | 332/492 [00:02<00:01, 156.92batch/s]

Training batches on cuda:  71%|███████   | 348/492 [00:02<00:01, 137.57batch/s]

Training batches on cuda:  74%|███████▍  | 363/492 [00:02<00:00, 130.40batch/s]

Training batches on cuda:  77%|███████▋  | 377/492 [00:02<00:00, 130.24batch/s]

Training batches on cuda:  79%|███████▉  | 391/492 [00:02<00:00, 127.12batch/s]

Training batches on cuda:  83%|████████▎ | 406/492 [00:02<00:00, 131.86batch/s]

Training batches on cuda:  85%|████████▌ | 420/492 [00:02<00:00, 132.98batch/s]

Training batches on cuda:  89%|████████▉ | 438/492 [00:02<00:00, 142.95batch/s]

Training batches on cuda:  93%|█████████▎| 456/492 [00:03<00:00, 150.82batch/s]

Training batches on cuda:  96%|█████████▌| 473/492 [00:03<00:00, 155.20batch/s]

Training batches on cuda:  99%|█████████▉| 489/492 [00:03<00:00, 152.55batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:27<00:10,  3.45s/epoch, loss=0.0016, prev_loss=0.0016]
Training epochs on cuda:  80%|████████  | 8/10 [00:27<00:07,  3.51s/epoch, loss=0.0016, prev_loss=0.0016]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/492 [00:00<00:04, 106.25batch/s]

Training batches on cuda:   5%|▌         | 27/492 [00:00<00:03, 118.16batch/s]

Training batches on cuda:   9%|▉         | 44/492 [00:00<00:03, 128.50batch/s]

Training batches on cuda:  12%|█▏        | 60/492 [00:00<00:03, 134.18batch/s]

Training batches on cuda:  15%|█▌        | 75/492 [00:00<00:03, 138.51batch/s]

Training batches on cuda:  18%|█▊        | 90/492 [00:00<00:02, 139.50batch/s]

Training batches on cuda:  22%|██▏       | 107/492 [00:00<00:02, 146.60batch/s]

Training batches on cuda:  25%|██▌       | 124/492 [00:00<00:02, 152.38batch/s]

Training batches on cuda:  29%|██▊       | 141/492 [00:00<00:02, 156.71batch/s]

Training batches on cuda:  33%|███▎      | 160/492 [00:01<00:02, 163.50batch/s]

Training batches on cuda:  36%|███▌      | 177/492 [00:01<00:02, 152.36batch/s]

Training batches on cuda:  39%|███▉      | 193/492 [00:01<00:02, 146.19batch/s]

Training batches on cuda:  42%|████▏     | 208/492 [00:01<00:01, 144.05batch/s]

Training batches on cuda:  45%|████▌     | 223/492 [00:01<00:01, 138.49batch/s]

Training batches on cuda:  48%|████▊     | 237/492 [00:01<00:01, 138.50batch/s]

Training batches on cuda:  51%|█████     | 252/492 [00:01<00:01, 139.64batch/s]

Training batches on cuda:  54%|█████▍    | 267/492 [00:01<00:01, 139.03batch/s]

Training batches on cuda:  57%|█████▋    | 281/492 [00:01<00:01, 130.20batch/s]

Training batches on cuda:  60%|██████    | 297/492 [00:02<00:01, 137.71batch/s]

Training batches on cuda:  64%|██████▍   | 314/492 [00:02<00:01, 145.48batch/s]

Training batches on cuda:  67%|██████▋   | 329/492 [00:02<00:01, 144.58batch/s]

Training batches on cuda:  70%|███████   | 346/492 [00:02<00:00, 149.15batch/s]

Training batches on cuda:  74%|███████▍  | 363/492 [00:02<00:00, 154.20batch/s]

Training batches on cuda:  77%|███████▋  | 381/492 [00:02<00:00, 159.32batch/s]

Training batches on cuda:  81%|████████  | 398/492 [00:02<00:00, 150.63batch/s]

Training batches on cuda:  84%|████████▍ | 414/492 [00:02<00:00, 145.50batch/s]

Training batches on cuda:  87%|████████▋ | 429/492 [00:02<00:00, 146.28batch/s]

Training batches on cuda:  90%|█████████ | 445/492 [00:03<00:00, 149.44batch/s]

Training batches on cuda:  94%|█████████▍| 462/492 [00:03<00:00, 153.13batch/s]

Training batches on cuda:  97%|█████████▋| 479/492 [00:03<00:00, 157.66batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:31<00:07,  3.51s/epoch, loss=0.0016, prev_loss=0.0016]
Training epochs on cuda:  90%|█████████ | 9/10 [00:31<00:03,  3.51s/epoch, loss=0.0016, prev_loss=0.0016]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 13/492 [00:00<00:03, 126.37batch/s]

Training batches on cuda:   6%|▋         | 31/492 [00:00<00:03, 138.42batch/s]

Training batches on cuda:  10%|▉         | 49/492 [00:00<00:03, 146.96batch/s]

Training batches on cuda:  14%|█▎        | 67/492 [00:00<00:02, 154.44batch/s]

Training batches on cuda:  18%|█▊        | 87/492 [00:00<00:02, 163.76batch/s]

Training batches on cuda:  22%|██▏       | 106/492 [00:00<00:02, 168.75batch/s]

Training batches on cuda:  25%|██▌       | 125/492 [00:00<00:02, 173.32batch/s]

Training batches on cuda:  29%|██▉       | 144/492 [00:00<00:01, 177.39batch/s]

Training batches on cuda:  33%|███▎      | 162/492 [00:00<00:01, 175.99batch/s]

Training batches on cuda:  37%|███▋      | 180/492 [00:01<00:01, 169.98batch/s]

Training batches on cuda:  40%|████      | 198/492 [00:01<00:01, 171.63batch/s]

Training batches on cuda:  44%|████▍     | 216/492 [00:01<00:01, 171.21batch/s]

Training batches on cuda:  48%|████▊     | 235/492 [00:01<00:01, 173.75batch/s]

Training batches on cuda:  51%|█████▏    | 253/492 [00:01<00:01, 168.91batch/s]

Training batches on cuda:  55%|█████▌    | 271/492 [00:01<00:01, 170.52batch/s]

Training batches on cuda:  59%|█████▊    | 289/492 [00:01<00:01, 171.40batch/s]

Training batches on cuda:  62%|██████▏   | 307/492 [00:01<00:01, 148.57batch/s]

Training batches on cuda:  66%|██████▌   | 323/492 [00:01<00:01, 148.18batch/s]

Training batches on cuda:  69%|██████▉   | 339/492 [00:02<00:01, 147.88batch/s]

Training batches on cuda:  72%|███████▏  | 355/492 [00:02<00:00, 150.21batch/s]

Training batches on cuda:  75%|███████▌  | 371/492 [00:02<00:00, 149.68batch/s]

Training batches on cuda:  79%|███████▊  | 387/492 [00:02<00:00, 147.00batch/s]

Training batches on cuda:  82%|████████▏ | 402/492 [00:02<00:00, 147.21batch/s]

Training batches on cuda:  85%|████████▍ | 417/492 [00:02<00:00, 147.34batch/s]

Training batches on cuda:  88%|████████▊ | 435/492 [00:02<00:00, 154.10batch/s]

Training batches on cuda:  92%|█████████▏| 453/492 [00:02<00:00, 158.84batch/s]

Training batches on cuda:  96%|█████████▌| 471/492 [00:02<00:00, 162.26batch/s]

Training batches on cuda:  99%|█████████▉| 489/492 [00:03<00:00, 165.17batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:34<00:03,  3.51s/epoch, loss=0.0016, prev_loss=0.0016]2021-09-16:17:30:19,350 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:30:20,358 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:30:35,282 INFO     [evaluator.py:657] Evaluation took 14.92s seconds
2021-09-16:17:30:35,400 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:50<00:00,  8.25s/epoch, loss=0.0016, prev_loss=0.0016]
Training epochs on cuda: 100%|██████████| 10/10 [00:50<00:00,  5.08s/epoch, loss=0.0016, prev_loss=0.0016]
2021-09-16:17:30:35,401 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp75i58x68'
2021-09-16:17:30:35,463 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp75i58x68' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.28ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.38ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.62ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.51ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.55ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.57ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.59ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.60ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.62ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:17:30:50,323 INFO     [evaluator.py:657] Evaluation took 14.85s seconds
[I 2021-09-16 17:30:50,329] Trial 6 finished with value: -0.0018272268711376771 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.1927271302345578, 'optimizer.lr': 0.03361354084709356, 'negative_sampler.num_negs_per_pos': 1, 'training.batch_size': 632}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:30:50,331 WARNING  [api.py:823] No random seed is specified. Setting to 679453201.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 8/275 [00:00<00:03, 79.55batch/s]

Training batches on cuda:   7%|▋         | 20/275 [00:00<00:02, 88.48batch/s]

Training batches on cuda:  12%|█▏        | 34/275 [00:00<00:02, 98.42batch/s]

Training batches on cuda:  17%|█▋        | 48/275 [00:00<00:02, 107.03batch/s]

Training batches on cuda:  22%|██▏       | 60/275 [00:00<00:01, 109.63batch/s]

Training batches on cuda:  27%|██▋       | 73/275 [00:00<00:01, 114.89batch/s]

Training batches on cuda:  32%|███▏      | 87/275 [00:00<00:01, 120.57batch/s]

Training batches on cuda:  37%|███▋      | 101/275 [00:00<00:01, 124.46batch/s]

Training batches on cuda:  42%|████▏     | 115/275 [00:00<00:01, 127.26batch/s]

Training batches on cuda:  47%|████▋     | 130/275 [00:01<00:01, 131.20batch/s]

Training batches on cuda:  52%|█████▏    | 144/275 [00:01<00:00, 133.59batch/s]

Training batches on cuda:  57%|█████▋    | 158/275 [00:01<00:00, 133.90batch/s]

Training batches on cuda:  63%|██████▎   | 172/275 [00:01<00:00, 134.68batch/s]

Training batches on cuda:  68%|██████▊   | 186/275 [00:01<00:00, 134.98batch/s]

Training batches on cuda:  73%|███████▎  | 201/275 [00:01<00:00, 137.55batch/s]

Training batches on cuda:  78%|███████▊  | 215/275 [00:01<00:00, 136.12batch/s]

Training batches on cuda:  84%|████████▎ | 230/275 [00:01<00:00, 137.60batch/s]

Training batches on cuda:  89%|████████▊ | 244/275 [00:01<00:00, 137.40batch/s]

Training batches on cuda:  94%|█████████▍| 258/275 [00:01<00:00, 136.23batch/s]

Training batches on cuda:  99%|█████████▉| 273/275 [00:02<00:00, 137.43batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000651, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:20,  2.25s/epoch, loss=0.000651, prev_loss=nan]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 6/275 [00:00<00:04, 56.18batch/s]

Training batches on cuda:   7%|▋         | 19/275 [00:00<00:03, 67.01batch/s]

Training batches on cuda:   9%|▊         | 24/275 [00:00<00:04, 60.38batch/s]

Training batches on cuda:  14%|█▍        | 38/275 [00:00<00:03, 71.94batch/s]

Training batches on cuda:  19%|█▉        | 53/275 [00:00<00:02, 84.37batch/s]

Training batches on cuda:  24%|██▎       | 65/275 [00:00<00:02, 91.19batch/s]

Training batches on cuda:  28%|██▊       | 77/275 [00:00<00:02, 96.01batch/s]

Training batches on cuda:  32%|███▏      | 89/275 [00:00<00:01, 101.78batch/s]

Training batches on cuda:  38%|███▊      | 104/275 [00:00<00:01, 110.95batch/s]

Training batches on cuda:  43%|████▎     | 119/275 [00:01<00:01, 118.65batch/s]

Training batches on cuda:  48%|████▊     | 133/275 [00:01<00:01, 123.09batch/s]

Training batches on cuda:  53%|█████▎    | 147/275 [00:01<00:01, 127.53batch/s]

Training batches on cuda:  59%|█████▊    | 161/275 [00:01<00:00, 129.92batch/s]

Training batches on cuda:  64%|██████▎   | 175/275 [00:01<00:00, 130.01batch/s]

Training batches on cuda:  69%|██████▊   | 189/275 [00:01<00:00, 129.53batch/s]

Training batches on cuda:  74%|███████▍  | 203/275 [00:01<00:00, 130.87batch/s]

Training batches on cuda:  79%|███████▉  | 217/275 [00:01<00:00, 129.37batch/s]

Training batches on cuda:  84%|████████▍ | 231/275 [00:01<00:00, 129.51batch/s]

Training batches on cuda:  89%|████████▉ | 245/275 [00:02<00:00, 130.29batch/s]

Training batches on cuda:  94%|█████████▍| 259/275 [00:02<00:00, 127.06batch/s]

Training batches on cuda:  99%|█████████▉| 272/275 [00:02<00:00, 127.47batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:20,  2.25s/epoch, loss=0.000637, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:18,  2.31s/epoch, loss=0.000637, prev_loss=nan]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/275 [00:00<00:04, 66.17batch/s]

Training batches on cuda:   8%|▊         | 21/275 [00:00<00:03, 77.99batch/s]

Training batches on cuda:  13%|█▎        | 35/275 [00:00<00:02, 89.27batch/s]

Training batches on cuda:  18%|█▊        | 49/275 [00:00<00:02, 99.97batch/s]

Training batches on cuda:  23%|██▎       | 63/275 [00:00<00:01, 108.71batch/s]

Training batches on cuda:  28%|██▊       | 76/275 [00:00<00:01, 113.74batch/s]

Training batches on cuda:  32%|███▏      | 88/275 [00:00<00:01, 105.59batch/s]

Training batches on cuda:  37%|███▋      | 101/275 [00:00<00:01, 111.56batch/s]

Training batches on cuda:  41%|████      | 113/275 [00:00<00:01, 113.95batch/s]

Training batches on cuda:  45%|████▌     | 125/275 [00:01<00:01, 109.67batch/s]

Training batches on cuda:  50%|████▉     | 137/275 [00:01<00:01, 106.31batch/s]

Training batches on cuda:  55%|█████▍    | 151/275 [00:01<00:01, 113.73batch/s]

Training batches on cuda:  60%|█████▉    | 164/275 [00:01<00:00, 117.28batch/s]

Training batches on cuda:  64%|██████▍   | 177/275 [00:01<00:00, 119.83batch/s]

Training batches on cuda:  69%|██████▉   | 190/275 [00:01<00:00, 121.17batch/s]

Training batches on cuda:  74%|███████▍  | 203/275 [00:01<00:00, 120.34batch/s]

Training batches on cuda:  79%|███████▊  | 216/275 [00:01<00:00, 119.97batch/s]

Training batches on cuda:  84%|████████▎ | 230/275 [00:01<00:00, 123.84batch/s]

Training batches on cuda:  89%|████████▊ | 244/275 [00:02<00:00, 127.63batch/s]

Training batches on cuda:  94%|█████████▍| 258/275 [00:02<00:00, 130.77batch/s]

Training batches on cuda:  99%|█████████▉| 272/275 [00:02<00:00, 125.71batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:18,  2.31s/epoch, loss=0.000638, prev_loss=0.000637]
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:16,  2.36s/epoch, loss=0.000638, prev_loss=0.000637]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 9/275 [00:00<00:03, 87.10batch/s]

Training batches on cuda:   8%|▊         | 23/275 [00:00<00:02, 97.15batch/s]

Training batches on cuda:  13%|█▎        | 37/275 [00:00<00:02, 105.88batch/s]

Training batches on cuda:  17%|█▋        | 46/275 [00:00<00:02, 89.74batch/s] 

Training batches on cuda:  21%|██        | 58/275 [00:00<00:02, 95.90batch/s]

Training batches on cuda:  25%|██▌       | 70/275 [00:00<00:02, 101.81batch/s]

Training batches on cuda:  30%|███       | 83/275 [00:00<00:01, 108.83batch/s]

Training batches on cuda:  35%|███▌      | 97/275 [00:00<00:01, 115.08batch/s]

Training batches on cuda:  40%|███▉      | 109/275 [00:00<00:01, 116.36batch/s]

Training batches on cuda:  44%|████▍     | 121/275 [00:01<00:01, 114.32batch/s]

Training batches on cuda:  48%|████▊     | 133/275 [00:01<00:01, 108.00batch/s]

Training batches on cuda:  53%|█████▎    | 146/275 [00:01<00:01, 113.41batch/s]

Training batches on cuda:  58%|█████▊    | 159/275 [00:01<00:00, 117.79batch/s]

Training batches on cuda:  63%|██████▎   | 172/275 [00:01<00:00, 118.27batch/s]

Training batches on cuda:  67%|██████▋   | 184/275 [00:01<00:00, 118.10batch/s]

Training batches on cuda:  71%|███████▏  | 196/275 [00:01<00:00, 117.70batch/s]

Training batches on cuda:  76%|███████▌  | 208/275 [00:01<00:00, 118.38batch/s]

Training batches on cuda:  80%|████████  | 221/275 [00:01<00:00, 119.58batch/s]

Training batches on cuda:  85%|████████▍ | 233/275 [00:02<00:00, 115.65batch/s]

Training batches on cuda:  90%|████████▉ | 247/275 [00:02<00:00, 121.25batch/s]

Training batches on cuda:  95%|█████████▍| 260/275 [00:02<00:00, 120.52batch/s]

Training batches on cuda:  99%|█████████▉| 273/275 [00:02<00:00, 111.05batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:16,  2.36s/epoch, loss=0.000637, prev_loss=0.000638]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:14,  2.45s/epoch, loss=0.000637, prev_loss=0.000638]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 4/275 [00:00<00:07, 38.68batch/s]

Training batches on cuda:   5%|▌         | 14/275 [00:00<00:05, 47.00batch/s]

Training batches on cuda:   9%|▉         | 25/275 [00:00<00:04, 56.14batch/s]

Training batches on cuda:  15%|█▍        | 40/275 [00:00<00:03, 68.30batch/s]

Training batches on cuda:  19%|█▉        | 53/275 [00:00<00:02, 79.19batch/s]

Training batches on cuda:  24%|██▍       | 66/275 [00:00<00:02, 88.62batch/s]

Training batches on cuda:  29%|██▊       | 79/275 [00:00<00:02, 96.22batch/s]

Training batches on cuda:  34%|███▍      | 93/275 [00:00<00:01, 104.91batch/s]

Training batches on cuda:  39%|███▉      | 107/275 [00:00<00:01, 112.72batch/s]

Training batches on cuda:  44%|████▍     | 121/275 [00:01<00:01, 119.44batch/s]

Training batches on cuda:  49%|████▉     | 135/275 [00:01<00:01, 123.98batch/s]

Training batches on cuda:  54%|█████▍    | 149/275 [00:01<00:00, 127.35batch/s]

Training batches on cuda:  59%|█████▉    | 163/275 [00:01<00:00, 129.86batch/s]

Training batches on cuda:  64%|██████▍   | 177/275 [00:01<00:00, 130.02batch/s]

Training batches on cuda:  69%|██████▉   | 191/275 [00:01<00:00, 127.43batch/s]

Training batches on cuda:  74%|███████▍  | 204/275 [00:01<00:00, 125.99batch/s]

Training batches on cuda:  79%|███████▉  | 217/275 [00:01<00:00, 125.13batch/s]

Training batches on cuda:  84%|████████▎ | 230/275 [00:01<00:00, 121.00batch/s]

Training batches on cuda:  88%|████████▊ | 243/275 [00:02<00:00, 118.26batch/s]

Training batches on cuda:  93%|█████████▎| 255/275 [00:02<00:00, 118.38batch/s]

Training batches on cuda:  97%|█████████▋| 267/275 [00:02<00:00, 116.13batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:14,  2.45s/epoch, loss=0.000638, prev_loss=0.000637]
Training epochs on cuda:  50%|█████     | 5/10 [00:12<00:12,  2.50s/epoch, loss=0.000638, prev_loss=0.000637]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/275 [00:00<00:05, 49.14batch/s]

Training batches on cuda:   5%|▌         | 14/275 [00:00<00:04, 56.12batch/s]

Training batches on cuda:   9%|▉         | 25/275 [00:00<00:03, 65.18batch/s]

Training batches on cuda:  13%|█▎        | 36/275 [00:00<00:03, 72.68batch/s]

Training batches on cuda:  17%|█▋        | 46/275 [00:00<00:02, 77.60batch/s]

Training batches on cuda:  21%|██        | 58/275 [00:00<00:02, 85.19batch/s]

Training batches on cuda:  25%|██▌       | 70/275 [00:00<00:02, 92.40batch/s]

Training batches on cuda:  29%|██▉       | 81/275 [00:00<00:02, 96.26batch/s]

Training batches on cuda:  34%|███▍      | 94/275 [00:00<00:01, 102.43batch/s]

Training batches on cuda:  38%|███▊      | 105/275 [00:01<00:01, 104.43batch/s]

Training batches on cuda:  43%|████▎     | 117/275 [00:01<00:01, 107.26batch/s]

Training batches on cuda:  47%|████▋     | 128/275 [00:01<00:01, 106.89batch/s]

Training batches on cuda:  51%|█████     | 139/275 [00:01<00:01, 106.51batch/s]

Training batches on cuda:  55%|█████▍    | 150/275 [00:01<00:01, 107.47batch/s]

Training batches on cuda:  59%|█████▉    | 163/275 [00:01<00:00, 112.30batch/s]

Training batches on cuda:  64%|██████▍   | 177/275 [00:01<00:00, 118.45batch/s]

Training batches on cuda:  69%|██████▉   | 190/275 [00:01<00:00, 120.35batch/s]

Training batches on cuda:  74%|███████▍  | 203/275 [00:01<00:00, 122.21batch/s]

Training batches on cuda:  79%|███████▊  | 216/275 [00:01<00:00, 122.75batch/s]

Training batches on cuda:  83%|████████▎ | 229/275 [00:02<00:00, 122.68batch/s]

Training batches on cuda:  88%|████████▊ | 242/275 [00:02<00:00, 122.58batch/s]

Training batches on cuda:  93%|█████████▎| 255/275 [00:02<00:00, 121.51batch/s]

Training batches on cuda:  97%|█████████▋| 268/275 [00:02<00:00, 115.99batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:12,  2.50s/epoch, loss=0.000638, prev_loss=0.000638]
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:10,  2.59s/epoch, loss=0.000638, prev_loss=0.000638]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 9/275 [00:00<00:03, 86.31batch/s]

Training batches on cuda:   8%|▊         | 22/275 [00:00<00:02, 94.48batch/s]

Training batches on cuda:  13%|█▎        | 36/275 [00:00<00:02, 103.47batch/s]

Training batches on cuda:  18%|█▊        | 50/275 [00:00<00:02, 111.74batch/s]

Training batches on cuda:  24%|██▎       | 65/275 [00:00<00:01, 119.03batch/s]

Training batches on cuda:  29%|██▉       | 80/275 [00:00<00:01, 125.23batch/s]

Training batches on cuda:  34%|███▍      | 94/275 [00:00<00:01, 129.03batch/s]

Training batches on cuda:  39%|███▉      | 107/275 [00:00<00:01, 128.55batch/s]

Training batches on cuda:  44%|████▎     | 120/275 [00:00<00:01, 127.23batch/s]

Training batches on cuda:  48%|████▊     | 133/275 [00:01<00:01, 125.88batch/s]

Training batches on cuda:  53%|█████▎    | 146/275 [00:01<00:01, 125.94batch/s]

Training batches on cuda:  58%|█████▊    | 159/275 [00:01<00:00, 125.49batch/s]

Training batches on cuda:  63%|██████▎   | 172/275 [00:01<00:00, 123.48batch/s]

Training batches on cuda:  68%|██████▊   | 186/275 [00:01<00:00, 126.39batch/s]

Training batches on cuda:  72%|███████▏  | 199/275 [00:01<00:00, 125.80batch/s]

Training batches on cuda:  77%|███████▋  | 212/275 [00:01<00:00, 124.70batch/s]

Training batches on cuda:  82%|████████▏ | 225/275 [00:01<00:00, 125.71batch/s]

Training batches on cuda:  87%|████████▋ | 239/275 [00:01<00:00, 128.23batch/s]

Training batches on cuda:  92%|█████████▏| 253/275 [00:01<00:00, 129.22batch/s]

Training batches on cuda:  97%|█████████▋| 267/275 [00:02<00:00, 129.82batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:10,  2.59s/epoch, loss=0.000638, prev_loss=0.000638]
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:07,  2.52s/epoch, loss=0.000638, prev_loss=0.000638]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 9/275 [00:00<00:03, 83.37batch/s]

Training batches on cuda:   7%|▋         | 20/275 [00:00<00:02, 88.72batch/s]

Training batches on cuda:  12%|█▏        | 32/275 [00:00<00:02, 95.56batch/s]

Training batches on cuda:  16%|█▌        | 44/275 [00:00<00:02, 100.80batch/s]

Training batches on cuda:  20%|██        | 55/275 [00:00<00:02, 102.81batch/s]

Training batches on cuda:  24%|██▍       | 67/275 [00:00<00:01, 106.14batch/s]

Training batches on cuda:  28%|██▊       | 78/275 [00:00<00:01, 106.24batch/s]

Training batches on cuda:  33%|███▎      | 90/275 [00:00<00:01, 108.76batch/s]

Training batches on cuda:  37%|███▋      | 102/275 [00:00<00:01, 109.96batch/s]

Training batches on cuda:  41%|████      | 113/275 [00:01<00:01, 108.44batch/s]

Training batches on cuda:  45%|████▌     | 124/275 [00:01<00:01, 107.71batch/s]

Training batches on cuda:  49%|████▉     | 135/275 [00:01<00:01, 106.77batch/s]

Training batches on cuda:  53%|█████▎    | 147/275 [00:01<00:01, 108.72batch/s]

Training batches on cuda:  58%|█████▊    | 160/275 [00:01<00:01, 112.93batch/s]

Training batches on cuda:  63%|██████▎   | 172/275 [00:01<00:00, 109.39batch/s]

Training batches on cuda:  67%|██████▋   | 184/275 [00:01<00:00, 111.70batch/s]

Training batches on cuda:  71%|███████▏  | 196/275 [00:01<00:00, 112.23batch/s]

Training batches on cuda:  76%|███████▋  | 210/275 [00:01<00:00, 119.17batch/s]

Training batches on cuda:  81%|████████▏ | 224/275 [00:01<00:00, 123.97batch/s]

Training batches on cuda:  87%|████████▋ | 239/275 [00:02<00:00, 130.22batch/s]

Training batches on cuda:  92%|█████████▏| 253/275 [00:02<00:00, 130.50batch/s]

Training batches on cuda:  97%|█████████▋| 267/275 [00:02<00:00, 127.55batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:07,  2.52s/epoch, loss=0.000638, prev_loss=0.000638]
Training epochs on cuda:  80%|████████  | 8/10 [00:20<00:05,  2.53s/epoch, loss=0.000638, prev_loss=0.000638]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/275 [00:00<00:03, 68.67batch/s]

Training batches on cuda:   7%|▋         | 18/275 [00:00<00:03, 77.07batch/s]

Training batches on cuda:  11%|█         | 30/275 [00:00<00:02, 86.15batch/s]

Training batches on cuda:  14%|█▍        | 39/275 [00:00<00:02, 87.18batch/s]

Training batches on cuda:  19%|█▉        | 52/275 [00:00<00:02, 95.91batch/s]

Training batches on cuda:  23%|██▎       | 64/275 [00:00<00:02, 100.99batch/s]

Training batches on cuda:  28%|██▊       | 76/275 [00:00<00:01, 105.82batch/s]

Training batches on cuda:  32%|███▏      | 88/275 [00:00<00:01, 108.43batch/s]

Training batches on cuda:  36%|███▋      | 100/275 [00:00<00:01, 109.54batch/s]

Training batches on cuda:  40%|████      | 111/275 [00:01<00:01, 105.69batch/s]

Training batches on cuda:  45%|████▍     | 123/275 [00:01<00:01, 107.37batch/s]

Training batches on cuda:  49%|████▊     | 134/275 [00:01<00:01, 107.53batch/s]

Training batches on cuda:  53%|█████▎    | 145/275 [00:01<00:01, 105.91batch/s]

Training batches on cuda:  57%|█████▋    | 156/275 [00:01<00:01, 102.48batch/s]

Training batches on cuda:  61%|██████    | 167/275 [00:01<00:01, 98.73batch/s] 

Training batches on cuda:  64%|██████▍   | 177/275 [00:01<00:01, 97.98batch/s]

Training batches on cuda:  68%|██████▊   | 187/275 [00:01<00:00, 98.55batch/s]

Training batches on cuda:  73%|███████▎  | 200/275 [00:01<00:00, 105.25batch/s]

Training batches on cuda:  78%|███████▊  | 214/275 [00:02<00:00, 112.39batch/s]

Training batches on cuda:  83%|████████▎ | 227/275 [00:02<00:00, 116.41batch/s]

Training batches on cuda:  87%|████████▋ | 239/275 [00:02<00:00, 110.46batch/s]

Training batches on cuda:  91%|█████████▏| 251/275 [00:02<00:00, 109.16batch/s]

Training batches on cuda:  96%|█████████▌| 263/275 [00:02<00:00, 109.14batch/s]

Training batches on cuda: 100%|██████████| 275/275 [00:02<00:00, 95.53batch/s] 

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:05,  2.53s/epoch, loss=0.000638, prev_loss=0.000638]
Training epochs on cuda:  90%|█████████ | 9/10 [00:23<00:02,  2.64s/epoch, loss=0.000638, prev_loss=0.000638]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 10/275 [00:00<00:02, 97.37batch/s]

Training batches on cuda:   8%|▊         | 21/275 [00:00<00:02, 98.96batch/s]

Training batches on cuda:  12%|█▏        | 34/275 [00:00<00:02, 106.30batch/s]

Training batches on cuda:  17%|█▋        | 48/275 [00:00<00:01, 114.09batch/s]

Training batches on cuda:  23%|██▎       | 62/275 [00:00<00:01, 118.72batch/s]

Training batches on cuda:  28%|██▊       | 76/275 [00:00<00:01, 123.80batch/s]

Training batches on cuda:  33%|███▎      | 90/275 [00:00<00:01, 126.27batch/s]

Training batches on cuda:  37%|███▋      | 103/275 [00:00<00:01, 125.64batch/s]

Training batches on cuda:  43%|████▎     | 117/275 [00:00<00:01, 128.96batch/s]

Training batches on cuda:  48%|████▊     | 131/275 [00:01<00:01, 131.89batch/s]

Training batches on cuda:  53%|█████▎    | 145/275 [00:01<00:00, 133.51batch/s]

Training batches on cuda:  58%|█████▊    | 159/275 [00:01<00:00, 132.96batch/s]

Training batches on cuda:  63%|██████▎   | 173/275 [00:01<00:00, 121.24batch/s]

Training batches on cuda:  68%|██████▊   | 186/275 [00:01<00:00, 110.36batch/s]

Training batches on cuda:  72%|███████▏  | 198/275 [00:01<00:00, 108.05batch/s]

Training batches on cuda:  76%|███████▋  | 210/275 [00:01<00:00, 107.40batch/s]

Training batches on cuda:  81%|████████  | 222/275 [00:01<00:00, 109.52batch/s]

Training batches on cuda:  85%|████████▌ | 234/275 [00:01<00:00, 112.36batch/s]

Training batches on cuda:  89%|████████▉ | 246/275 [00:02<00:00, 114.07batch/s]

Training batches on cuda:  94%|█████████▍| 258/275 [00:02<00:00, 115.64batch/s]

Training batches on cuda:  99%|█████████▊| 271/275 [00:02<00:00, 118.47batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:25<00:02,  2.64s/epoch, loss=0.000638, prev_loss=0.000638]2021-09-16:17:31:16,286 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:31:17,345 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:31:32,284 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:17:31:32,400 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00,  7.42s/epoch, loss=0.000638, prev_loss=0.000638]
Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00,  4.17s/epoch, loss=0.000638, prev_loss=0.000638]
2021-09-16:17:31:32,401 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp6bxnzszd'
2021-09-16:17:31:32,454 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6bxnzszd' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.14ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.28ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.38ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.55ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.61ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.60ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.62ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:17:31:47,306 INFO     [evaluator.py:657] Evaluation took 14.84s seconds
[I 2021-09-16 17:31:47,311] Trial 7 finished with value: -0.0028888322040463432 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.41334476715518176, 'optimizer.lr': 0.061555779164346514, 'negative_sampler.num_negs_per_pos': 1, 'training.batch_size': 1132}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:31:47,314 WARNING  [api.py:823] No random seed is specified. Setting to 2876964706.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:32,  4.08batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:25,  5.08batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:21,  6.08batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:17,  7.11batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:15,  8.08batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:13,  8.91batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:12,  9.63batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:11, 10.14batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:11,  9.81batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:11, 10.27batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:02<00:10, 10.73batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:10, 11.00batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:09, 11.24batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.34batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:09, 11.47batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.51batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:03<00:08, 11.60batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 11.68batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:08, 11.75batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:08, 11.77batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:07, 11.84batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 11.78batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:04<00:07, 11.75batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 11.56batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 11.72batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:07, 11.84batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:06, 11.81batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 11.87batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:05<00:06, 11.69batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 11.75batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:06, 11.77batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:06, 11.71batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:05, 11.74batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 11.86batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:06<00:05, 11.80batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 11.64batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 11.46batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:05, 11.56batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:05, 10.89batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:07<00:05, 10.85batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:07<00:04, 11.05batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.15batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.32batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:04, 11.37batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.47batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:08<00:03, 11.51batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:08<00:03, 11.50batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.53batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.67batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:02, 11.74batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.85batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:09<00:02, 11.82batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:09<00:02, 11.86batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 11.76batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 11.74batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 11.84batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 11.82batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:10<00:01, 11.77batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:10<00:01, 11.75batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 11.76batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 11.78batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 11.76batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 11.78batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:11<00:00, 11.78batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 11.71batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 11.67batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 11.50batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:12<?, ?epoch/s, loss=0.000333, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:12<01:48, 12.00s/epoch, loss=0.000333, prev_loss=nan]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:26,  4.96batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:21,  6.03batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:18,  7.05batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:15,  8.02batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:14,  8.89batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:12,  9.69batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:11, 10.21batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:11, 10.68batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:10, 10.99batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:10, 11.26batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:01<00:09, 11.47batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:09, 11.58batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:09, 11.70batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.67batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:09, 11.65batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.61batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:02<00:08, 11.62batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 11.67batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:08, 11.63batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:08, 11.67batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:07, 11.67batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 11.62batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:03<00:07, 11.65batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 11.71batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 11.76batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:07, 11.80batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:06, 11.78batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 11.75batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:04<00:06, 11.58batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 11.75batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:06, 11.66batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:06, 11.62batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:05, 11.66batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 11.74batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:05<00:05, 11.66batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 11.73batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 11.81batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:04, 11.86batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:04, 11.94batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:06<00:04, 11.86batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:06<00:04, 11.77batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.75batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.75batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:03, 11.80batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.71batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:07<00:03, 11.61batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:08<00:03, 11.56batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.63batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.56batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:03, 11.61batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.71batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:08<00:02, 11.71batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:09<00:02, 11.43batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 11.43batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 11.55batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 11.52batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 11.52batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:09<00:01, 11.15batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:10<00:01, 11.32batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 11.48batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 11.65batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 11.58batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 11.73batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:10<00:00, 11.80batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 11.95batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 12.08batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 12.14batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:23<01:48, 12.00s/epoch, loss=0.00031, prev_loss=nan] 
Training epochs on cuda:  20%|██        | 2/10 [00:23<01:35, 11.94s/epoch, loss=0.00031, prev_loss=nan]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:21,  6.30batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:17,  7.35batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:15,  8.20batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:14,  8.94batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:13,  9.61batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:12, 10.15batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:11, 10.52batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:10, 10.82batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:10, 11.05batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:10, 11.19batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:01<00:09, 11.36batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:09, 11.48batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:09, 11.47batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.58batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:09, 11.54batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.53batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:02<00:08, 11.57batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 11.56batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:08, 11.55batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:08, 11.56batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:08, 11.61batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 11.64batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:03<00:07, 11.64batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 11.60batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 11.64batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:07, 11.79batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:06, 11.75batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 11.73batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:04<00:06, 11.64batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 11.55batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:06, 11.52batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:06, 11.61batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:05, 11.69batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 11.75batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:05<00:05, 11.83batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 11.84batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 11.87batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:04, 11.92batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:04, 11.84batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:06<00:04, 11.79batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:07<00:04, 11.82batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.84batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.86batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:03, 11.84batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.94batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:07<00:03, 11.89batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:08<00:03, 11.94batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.97batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.22batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:03, 11.41batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.51batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:08<00:02, 11.66batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:09<00:02, 11.75batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 11.80batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 11.84batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 11.87batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 11.90batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:09<00:01, 11.95batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:10<00:01, 11.90batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 11.93batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 11.95batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 11.92batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 11.92batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:10<00:00, 11.96batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 11.95batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 11.98batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 11.93batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:35<01:35, 11.94s/epoch, loss=0.000311, prev_loss=0.00031]
Training epochs on cuda:  30%|███       | 3/10 [00:35<01:23, 11.89s/epoch, loss=0.000311, prev_loss=0.00031]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:25,  5.21batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:21,  6.23batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:17,  7.25batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:15,  8.23batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:13,  9.06batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:12,  9.70batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:11, 10.28batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:11, 10.70batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:10, 11.06batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:10, 11.23batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:01<00:09, 11.33batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:09, 11.53batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:09, 11.58batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.69batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:08, 11.78batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.73batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:02<00:08, 11.69batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 11.78batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:08, 11.83batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:08, 11.78batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:07, 11.82batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 11.84batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:03<00:07, 11.88batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 11.96batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 11.95batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:06, 12.01batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:06, 11.96batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 11.86batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:04<00:06, 11.90batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 11.89batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:06, 11.89batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:05, 11.93batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:05, 11.69batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 11.77batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:05<00:05, 11.80batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 11.85batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 11.68batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:05, 11.69batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:04, 11.60batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:06<00:05, 10.92batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:07<00:04, 10.99batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.25batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.33batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:04, 11.50batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.49batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:07<00:03, 11.50batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:08<00:03, 11.52batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.65batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.61batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:03, 11.59batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.51batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:08<00:02, 11.45batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:09<00:02, 11.61batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 11.66batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 11.62batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 11.73batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 11.80batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:09<00:01, 11.84batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:10<00:01, 11.88batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 11.92batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 11.95batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 11.94batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 11.86batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:10<00:00, 11.93batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 11.98batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 11.91batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 11.98batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:47<01:23, 11.89s/epoch, loss=0.000311, prev_loss=0.000311]
Training epochs on cuda:  40%|████      | 4/10 [00:47<01:11, 11.84s/epoch, loss=0.000311, prev_loss=0.000311]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:31,  4.29batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:24,  5.28batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:20,  6.26batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:17,  7.31batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:15,  8.28batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:13,  9.14batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:12,  9.84batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:11, 10.43batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:10, 10.82batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:10, 11.03batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:01<00:09, 11.31batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:09, 11.49batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:09, 11.61batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.78batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:08, 11.88batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.97batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:02<00:08, 12.03batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 12.12batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:07, 12.18batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:07, 12.13batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:07, 12.12batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 12.14batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:03<00:07, 12.11batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 12.08batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 12.03batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:06, 12.02batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:06, 11.95batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 11.89batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:04<00:06, 11.87batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 11.81batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:06, 11.93batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:05, 11.92batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:05, 11.94batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 11.92batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:05<00:05, 11.83batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 11.80batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 11.86batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:05, 11.75batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:04, 11.77batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:06<00:04, 11.71batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:06<00:04, 11.68batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.62batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.72batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:04, 11.66batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.60batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:07<00:03, 11.67batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:07<00:03, 11.55batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.71batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.73batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:02, 11.82batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.85batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:08<00:02, 11.82batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:08<00:02, 11.84batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 11.85batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 11.76batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 11.79batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 11.87batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:09<00:01, 11.69batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:10<00:01, 11.70batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 11.64batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 11.59batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 11.58batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 11.50batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:10<00:00, 11.59batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 11.57batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 11.67batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 11.69batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:59<01:11, 11.84s/epoch, loss=0.000311, prev_loss=0.000311]
Training epochs on cuda:  50%|█████     | 5/10 [00:59<00:58, 11.80s/epoch, loss=0.000311, prev_loss=0.000311]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:28,  4.71batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:22,  5.72batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:19,  6.75batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:16,  7.76batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:14,  8.67batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:13,  9.36batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:12,  9.92batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:11, 10.45batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:10, 10.88batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:10, 11.21batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:01<00:09, 11.50batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:09, 11.72batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:09, 11.85batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.86batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:08, 11.98batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.99batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:02<00:08, 12.04batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 12.09batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:08, 12.12batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:07, 12.21batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:07, 12.21batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 12.18batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:03<00:07, 12.09batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 12.11batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 12.13batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:06, 12.15batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:06, 12.15batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 12.12batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:04<00:06, 12.13batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 12.19batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:05, 12.18batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:05, 12.08batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:05, 12.01batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 12.03batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:05<00:05, 12.08batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 11.95batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 11.77batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:05, 11.73batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:04, 11.66batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:06<00:04, 11.57batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:06<00:04, 11.50batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.48batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.42batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:04, 11.50batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.57batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:07<00:03, 11.75batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:07<00:03, 11.83batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.81batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.85batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:02, 11.89batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.19batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:08<00:02, 11.43batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:08<00:02, 11.55batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 11.67batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 11.74batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 11.72batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 11.74batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:09<00:01, 11.71batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:09<00:01, 11.65batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 11.63batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 11.66batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 11.74batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 11.76batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:10<00:00, 11.73batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 11.72batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 11.69batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 11.60batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [01:10<00:58, 11.80s/epoch, loss=0.000311, prev_loss=0.000311]
Training epochs on cuda:  60%|██████    | 6/10 [01:10<00:47, 11.79s/epoch, loss=0.000311, prev_loss=0.000311]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:21,  6.14batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:18,  7.15batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:15,  8.16batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:14,  8.99batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:13,  9.61batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:12, 10.12batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:11, 10.57batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:10, 10.87batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:10, 11.10batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:10, 11.28batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:01<00:10, 10.53batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:10, 10.79batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:09, 11.11batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.31batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:09, 11.55batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.60batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:02<00:08, 11.69batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 11.73batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:08, 11.75batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:08, 11.87batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:07, 11.92batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 12.00batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:03<00:07, 11.88batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 11.88batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 11.82batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:07, 11.86batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:06, 11.97batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 12.06batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:04<00:06, 12.03batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 12.03batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:06, 12.00batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:06, 11.81batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:05, 11.88batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 11.91batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:05<00:05, 11.97batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 12.07batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 12.02batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:04, 11.99batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:04, 11.97batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:06<00:04, 11.96batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:06<00:04, 11.21batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.42batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.50batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:04, 11.56batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.62batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:07<00:03, 11.70batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:08<00:03, 11.71batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.78batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.73batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:02, 11.85batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.87batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:08<00:02, 11.91batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:09<00:02, 11.88batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 11.89batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 11.73batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 11.81batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 11.69batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:09<00:01, 11.77batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:10<00:01, 11.80batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 11.80batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 11.86batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 11.87batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 11.79batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:10<00:00, 11.84batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 11.92batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 11.97batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 11.95batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [01:22<00:47, 11.79s/epoch, loss=0.000311, prev_loss=0.000311]
Training epochs on cuda:  70%|███████   | 7/10 [01:22<00:35, 11.75s/epoch, loss=0.000311, prev_loss=0.000311]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:28,  4.68batch/s]

Training batches on cuda:   1%|▏         | 2/134 [00:00<00:24,  5.38batch/s]

Training batches on cuda:   3%|▎         | 4/134 [00:00<00:20,  6.43batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:17,  7.47batch/s]

Training batches on cuda:   6%|▌         | 8/134 [00:00<00:15,  8.36batch/s]

Training batches on cuda:   7%|▋         | 10/134 [00:01<00:13,  9.14batch/s]

Training batches on cuda:   9%|▉         | 12/134 [00:01<00:12,  9.79batch/s]

Training batches on cuda:  10%|█         | 14/134 [00:01<00:11, 10.35batch/s]

Training batches on cuda:  12%|█▏        | 16/134 [00:01<00:11, 10.71batch/s]

Training batches on cuda:  13%|█▎        | 18/134 [00:01<00:10, 11.03batch/s]

Training batches on cuda:  15%|█▍        | 20/134 [00:01<00:10, 11.25batch/s]

Training batches on cuda:  16%|█▋        | 22/134 [00:02<00:09, 11.42batch/s]

Training batches on cuda:  18%|█▊        | 24/134 [00:02<00:09, 11.59batch/s]

Training batches on cuda:  19%|█▉        | 26/134 [00:02<00:09, 11.76batch/s]

Training batches on cuda:  21%|██        | 28/134 [00:02<00:09, 11.67batch/s]

Training batches on cuda:  22%|██▏       | 30/134 [00:02<00:08, 11.70batch/s]

Training batches on cuda:  24%|██▍       | 32/134 [00:02<00:08, 11.75batch/s]

Training batches on cuda:  25%|██▌       | 34/134 [00:03<00:08, 11.80batch/s]

Training batches on cuda:  27%|██▋       | 36/134 [00:03<00:08, 11.79batch/s]

Training batches on cuda:  28%|██▊       | 38/134 [00:03<00:08, 11.82batch/s]

Training batches on cuda:  30%|██▉       | 40/134 [00:03<00:07, 11.80batch/s]

Training batches on cuda:  31%|███▏      | 42/134 [00:03<00:07, 11.76batch/s]

Training batches on cuda:  33%|███▎      | 44/134 [00:03<00:07, 11.90batch/s]

Training batches on cuda:  34%|███▍      | 46/134 [00:04<00:07, 11.98batch/s]

Training batches on cuda:  36%|███▌      | 48/134 [00:04<00:07, 11.93batch/s]

Training batches on cuda:  37%|███▋      | 50/134 [00:04<00:06, 12.02batch/s]

Training batches on cuda:  39%|███▉      | 52/134 [00:04<00:06, 12.05batch/s]

Training batches on cuda:  40%|████      | 54/134 [00:04<00:06, 12.15batch/s]

Training batches on cuda:  42%|████▏     | 56/134 [00:04<00:06, 12.17batch/s]

Training batches on cuda:  43%|████▎     | 58/134 [00:05<00:06, 12.21batch/s]

Training batches on cuda:  45%|████▍     | 60/134 [00:05<00:06, 12.24batch/s]

Training batches on cuda:  46%|████▋     | 62/134 [00:05<00:05, 12.22batch/s]

Training batches on cuda:  48%|████▊     | 64/134 [00:05<00:06, 11.24batch/s]

Training batches on cuda:  49%|████▉     | 66/134 [00:05<00:05, 11.36batch/s]

Training batches on cuda:  51%|█████     | 68/134 [00:05<00:05, 11.56batch/s]

Training batches on cuda:  52%|█████▏    | 70/134 [00:06<00:05, 11.49batch/s]

Training batches on cuda:  54%|█████▎    | 72/134 [00:06<00:05, 11.69batch/s]

Training batches on cuda:  55%|█████▌    | 74/134 [00:06<00:05, 11.84batch/s]

Training batches on cuda:  57%|█████▋    | 76/134 [00:06<00:04, 11.85batch/s]

Training batches on cuda:  58%|█████▊    | 78/134 [00:06<00:04, 11.72batch/s]

Training batches on cuda:  60%|█████▉    | 80/134 [00:06<00:04, 11.73batch/s]

Training batches on cuda:  61%|██████    | 82/134 [00:07<00:04, 11.58batch/s]

Training batches on cuda:  63%|██████▎   | 84/134 [00:07<00:04, 11.69batch/s]

Training batches on cuda:  64%|██████▍   | 86/134 [00:07<00:04, 11.71batch/s]

Training batches on cuda:  66%|██████▌   | 88/134 [00:07<00:03, 11.85batch/s]

Training batches on cuda:  67%|██████▋   | 90/134 [00:07<00:03, 11.89batch/s]

Training batches on cuda:  69%|██████▊   | 92/134 [00:07<00:03, 11.81batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:08<00:03, 11.81batch/s]

Training batches on cuda:  72%|███████▏  | 96/134 [00:08<00:03, 11.79batch/s]

Training batches on cuda:  73%|███████▎  | 98/134 [00:08<00:03, 11.79batch/s]

Training batches on cuda:  75%|███████▍  | 100/134 [00:08<00:02, 11.80batch/s]

Training batches on cuda:  76%|███████▌  | 102/134 [00:08<00:02, 11.85batch/s]

Training batches on cuda:  78%|███████▊  | 104/134 [00:08<00:02, 11.90batch/s]

Training batches on cuda:  79%|███████▉  | 106/134 [00:09<00:02, 11.87batch/s]

Training batches on cuda:  81%|████████  | 108/134 [00:09<00:02, 11.91batch/s]

Training batches on cuda:  82%|████████▏ | 110/134 [00:09<00:02, 11.91batch/s]

Training batches on cuda:  84%|████████▎ | 112/134 [00:09<00:01, 11.76batch/s]

Training batches on cuda:  85%|████████▌ | 114/134 [00:09<00:01, 11.62batch/s]

Training batches on cuda:  87%|████████▋ | 116/134 [00:09<00:01, 11.66batch/s]

Training batches on cuda:  88%|████████▊ | 118/134 [00:10<00:01, 11.74batch/s]

Training batches on cuda:  90%|████████▉ | 120/134 [00:10<00:01, 11.67batch/s]

Training batches on cuda:  91%|█████████ | 122/134 [00:10<00:01, 11.71batch/s]

Training batches on cuda:  93%|█████████▎| 124/134 [00:10<00:00, 10.43batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:10<00:00, 10.79batch/s]

Training batches on cuda:  96%|█████████▌| 128/134 [00:11<00:00, 11.03batch/s]

Training batches on cuda:  97%|█████████▋| 130/134 [00:11<00:00, 11.34batch/s]

Training batches on cuda:  99%|█████████▊| 132/134 [00:11<00:00, 11.44batch/s]

Training batches on cuda: 100%|██████████| 134/134 [00:11<00:00, 13.09batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [01:34<00:35, 11.75s/epoch, loss=0.000311, prev_loss=0.000311]
Training epochs on cuda:  80%|████████  | 8/10 [01:34<00:23, 11.74s/epoch, loss=0.000311, prev_loss=0.000311]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:30,  4.37batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:24,  5.40batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:20,  6.41batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:17,  7.42batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:15,  8.31batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:13,  9.09batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:12,  9.83batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:11, 10.47batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:10, 10.88batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:10, 11.18batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:01<00:09, 11.43batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:09, 11.41batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:09, 11.44batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.40batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:09, 11.49batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.62batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:02<00:08, 11.77batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 11.77batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:08, 11.45batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:08, 11.52batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:08, 11.55batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 11.66batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:04<00:08, 11.06batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 11.18batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 11.34batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:07, 11.47batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:07, 11.51batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 11.41batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:05<00:06, 11.42batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 11.56batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:06, 11.61batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:06, 11.72batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:05, 11.81batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 11.83batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:06<00:05, 11.79batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 11.82batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 11.80batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:05, 11.75batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:04, 11.86batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:06<00:04, 11.71batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:07<00:04, 11.69batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.70batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.76batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:04, 11.68batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.74batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:07<00:03, 11.81batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:08<00:03, 11.60batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.71batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.62batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:02, 11.68batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.69batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:08<00:02, 11.72batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:09<00:02, 11.70batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 11.67batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 11.67batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 11.74batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 11.79batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:09<00:01, 11.84batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:10<00:01, 11.84batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 12.00batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 11.93batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 11.65batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 11.56batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:11<00:00, 11.60batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 11.79batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 11.77batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 11.81batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:46<00:23, 11.74s/epoch, loss=0.000311, prev_loss=0.000311]
Training epochs on cuda:  90%|█████████ | 9/10 [01:46<00:11, 11.78s/epoch, loss=0.000311, prev_loss=0.000311]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:27,  4.83batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:22,  5.84batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:18,  6.82batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:16,  7.77batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:14,  8.50batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:01<00:13,  9.31batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:01<00:12,  9.76batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:01<00:11, 10.23batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:01<00:11, 10.58batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:01<00:10, 10.86batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:01<00:10, 11.16batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:02<00:09, 11.33batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:02<00:10, 10.76batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:02<00:09, 11.01batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:02<00:09, 11.29batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:02<00:08, 11.44batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:03<00:08, 11.56batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:03<00:08, 11.59batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:03<00:08, 11.78batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:03<00:07, 11.89batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:03<00:07, 11.89batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:03<00:07, 11.85batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:04<00:07, 11.94batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:04<00:07, 11.96batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:04<00:07, 11.88batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:04<00:07, 11.70batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:04<00:06, 11.60batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:04<00:06, 11.47batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:05<00:06, 11.41batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:05<00:06, 11.41batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:05<00:06, 11.47batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:05<00:06, 11.47batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:05<00:06, 11.37batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:05<00:05, 11.34batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:06<00:05, 11.19batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:06<00:05, 11.17batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:06<00:05, 11.32batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:06<00:05, 11.44batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:06<00:04, 11.52batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:06<00:04, 11.66batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:07<00:04, 11.76batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:07<00:04, 11.07batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:07<00:04, 11.15batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:07<00:04, 11.36batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:07<00:03, 11.44batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:08<00:03, 11.58batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:08<00:03, 11.74batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:08<00:03, 11.71batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:08<00:03, 11.80batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:08<00:02, 11.82batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:08<00:02, 11.91batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:09<00:02, 11.91batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:09<00:02, 11.99batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:09<00:02, 12.02batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:09<00:02, 12.09batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:09<00:01, 12.14batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:09<00:01, 12.14batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:10<00:01, 12.01batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:10<00:01, 11.97batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:10<00:01, 12.05batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:10<00:01, 12.13batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:10<00:00, 12.16batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:10<00:00, 12.16batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:11<00:00, 12.16batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:11<00:00, 12.08batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:11<00:00, 12.05batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:11<00:00, 12.09batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:57<00:11, 11.78s/epoch, loss=0.000312, prev_loss=0.000311]2021-09-16:17:33:45,895 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:33:46,974 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:34:01,982 INFO     [evaluator.py:657] Evaluation took 15.01s seconds
2021-09-16:17:34:02,116 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 16.67s/epoch, loss=0.000312, prev_loss=0.000311]
Training epochs on cuda: 100%|██████████| 10/10 [02:14<00:00, 13.41s/epoch, loss=0.000312, prev_loss=0.000311]
2021-09-16:17:34:02,117 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp_3auerc5'
2021-09-16:17:34:02,184 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_3auerc5' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.52ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.62ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.60ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.60ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.61ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.62ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.61ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.62ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.54ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.57ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.59ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.60ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.61ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.61ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.62ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.62ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:17:34:17,130 INFO     [evaluator.py:657] Evaluation took 14.92s seconds
[I 2021-09-16 17:34:17,135] Trial 8 finished with value: -0.003864388369607674 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.6022094157229457, 'optimizer.lr': 0.0471048320300294, 'negative_sampler.num_negs_per_pos': 97, 'training.batch_size': 2332}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:34:17,138 WARNING  [api.py:823] No random seed is specified. Setting to 575518866.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:03, 29.92batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:02, 34.84batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:02, 39.60batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:00<00:01, 43.23batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:00<00:01, 46.21batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:00<00:01, 48.65batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:00<00:01, 51.17batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:00<00:01, 52.98batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:00<00:00, 53.94batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:01<00:00, 53.93batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:01<00:00, 50.38batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:01<00:00, 50.77batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:01<00:00, 49.83batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:01<00:00, 50.90batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:01<00:00, 51.52batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:01<00:00, 51.79batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:01<00:00, 52.26batch/s]

                                                                             
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000298, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:19,  2.16s/epoch, loss=0.000298, prev_loss=nan]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 4/103 [00:00<00:02, 35.47batch/s]

Training batches on cuda:  10%|▉         | 10/103 [00:00<00:02, 39.50batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:02, 41.59batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:02, 40.53batch/s]

Training batches on cuda:  23%|██▎       | 24/103 [00:00<00:01, 42.84batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:00<00:01, 43.63batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:00<00:01, 44.93batch/s]

Training batches on cuda:  39%|███▉      | 40/103 [00:00<00:01, 45.54batch/s]

Training batches on cuda:  45%|████▍     | 46/103 [00:00<00:01, 47.81batch/s]

Training batches on cuda:  50%|█████     | 52/103 [00:01<00:01, 49.48batch/s]

Training batches on cuda:  56%|█████▋    | 58/103 [00:01<00:00, 50.92batch/s]

Training batches on cuda:  62%|██████▏   | 64/103 [00:01<00:00, 48.16batch/s]

Training batches on cuda:  68%|██████▊   | 70/103 [00:01<00:00, 48.76batch/s]

Training batches on cuda:  74%|███████▍  | 76/103 [00:01<00:00, 49.28batch/s]

Training batches on cuda:  80%|███████▉  | 82/103 [00:01<00:00, 49.65batch/s]

Training batches on cuda:  85%|████████▌ | 88/103 [00:01<00:00, 49.85batch/s]

Training batches on cuda:  91%|█████████▏| 94/103 [00:01<00:00, 50.19batch/s]

Training batches on cuda:  97%|█████████▋| 100/103 [00:02<00:00, 48.53batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:19,  2.16s/epoch, loss=0.000259, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:17,  2.23s/epoch, loss=0.000259, prev_loss=nan]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 4/103 [00:00<00:02, 35.14batch/s]

Training batches on cuda:  10%|▉         | 10/103 [00:00<00:02, 38.99batch/s]

Training batches on cuda:  16%|█▌        | 16/103 [00:00<00:02, 42.11batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:00<00:01, 43.56batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:00<00:01, 45.37batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:00<00:01, 46.96batch/s]

Training batches on cuda:  37%|███▋      | 38/103 [00:00<00:01, 47.50batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:00<00:01, 47.58batch/s]

Training batches on cuda:  47%|████▋     | 48/103 [00:00<00:01, 47.97batch/s]

Training batches on cuda:  52%|█████▏    | 54/103 [00:01<00:00, 49.01batch/s]

Training batches on cuda:  58%|█████▊    | 60/103 [00:01<00:00, 50.14batch/s]

Training batches on cuda:  64%|██████▍   | 66/103 [00:01<00:00, 51.91batch/s]

Training batches on cuda:  70%|██████▉   | 72/103 [00:01<00:00, 50.68batch/s]

Training batches on cuda:  76%|███████▌  | 78/103 [00:01<00:00, 49.01batch/s]

Training batches on cuda:  82%|████████▏ | 84/103 [00:01<00:00, 49.65batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:01<00:00, 48.88batch/s]

Training batches on cuda:  91%|█████████▏| 94/103 [00:01<00:00, 49.16batch/s]

Training batches on cuda:  97%|█████████▋| 100/103 [00:02<00:00, 49.79batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:17,  2.23s/epoch, loss=0.000259, prev_loss=0.000259]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:15,  2.25s/epoch, loss=0.000259, prev_loss=0.000259]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:03, 26.07batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:03, 28.98batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:02, 33.22batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:02, 36.99batch/s]

Training batches on cuda:  23%|██▎       | 24/103 [00:00<00:01, 39.81batch/s]

Training batches on cuda:  29%|██▉       | 30/103 [00:00<00:01, 42.60batch/s]

Training batches on cuda:  35%|███▍      | 36/103 [00:00<00:01, 44.62batch/s]

Training batches on cuda:  41%|████      | 42/103 [00:00<00:01, 46.15batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:01<00:01, 47.03batch/s]

Training batches on cuda:  50%|█████     | 52/103 [00:01<00:01, 47.00batch/s]

Training batches on cuda:  56%|█████▋    | 58/103 [00:01<00:00, 49.47batch/s]

Training batches on cuda:  62%|██████▏   | 64/103 [00:01<00:00, 50.87batch/s]

Training batches on cuda:  68%|██████▊   | 70/103 [00:01<00:00, 51.46batch/s]

Training batches on cuda:  74%|███████▍  | 76/103 [00:01<00:00, 52.09batch/s]

Training batches on cuda:  80%|███████▉  | 82/103 [00:01<00:00, 53.09batch/s]

Training batches on cuda:  85%|████████▌ | 88/103 [00:01<00:00, 54.16batch/s]

Training batches on cuda:  91%|█████████▏| 94/103 [00:01<00:00, 54.30batch/s]

Training batches on cuda:  97%|█████████▋| 100/103 [00:02<00:00, 54.22batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:15,  2.25s/epoch, loss=0.00026, prev_loss=0.000259] 
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:13,  2.27s/epoch, loss=0.00026, prev_loss=0.000259]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/103 [00:00<00:05, 19.99batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:04, 23.88batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:03, 28.71batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:02, 33.51batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:00<00:02, 38.08batch/s]

Training batches on cuda:  29%|██▉       | 30/103 [00:00<00:01, 40.63batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:00<00:01, 41.74batch/s]

Training batches on cuda:  39%|███▉      | 40/103 [00:00<00:01, 42.29batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:00<00:01, 42.17batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:01<00:01, 44.42batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:01<00:01, 45.99batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:01<00:00, 47.39batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:01<00:00, 48.39batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:01<00:00, 50.59batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:01<00:00, 51.71batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:01<00:00, 53.01batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:01<00:00, 53.86batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:02<00:00, 52.38batch/s]

                                                                             
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:13,  2.27s/epoch, loss=0.000261, prev_loss=0.00026]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.29s/epoch, loss=0.000261, prev_loss=0.00026]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 4/103 [00:00<00:02, 36.79batch/s]

Training batches on cuda:  10%|▉         | 10/103 [00:00<00:02, 40.33batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:02, 42.62batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:00<00:01, 44.96batch/s]

Training batches on cuda:  25%|██▌       | 26/103 [00:00<00:01, 46.11batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:00<00:01, 46.36batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:00<00:01, 47.71batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:00<00:01, 48.48batch/s]

Training batches on cuda:  47%|████▋     | 48/103 [00:00<00:01, 48.70batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:01<00:01, 48.93batch/s]

Training batches on cuda:  56%|█████▋    | 58/103 [00:01<00:00, 48.38batch/s]

Training batches on cuda:  62%|██████▏   | 64/103 [00:01<00:00, 49.29batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:01<00:00, 49.41batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:01<00:00, 50.72batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:01<00:00, 51.62batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:01<00:00, 52.29batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:01<00:00, 52.82batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:01<00:00, 52.40batch/s]

                                                                             
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:11,  2.29s/epoch, loss=0.000261, prev_loss=0.000261]
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:09,  2.31s/epoch, loss=0.000261, prev_loss=0.000261]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:04, 23.54batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:03, 28.50batch/s]

Training batches on cuda:  14%|█▎        | 14/103 [00:00<00:02, 32.72batch/s]

Training batches on cuda:  19%|█▉        | 20/103 [00:00<00:02, 37.03batch/s]

Training batches on cuda:  23%|██▎       | 24/103 [00:00<00:02, 35.86batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:00<00:01, 37.53batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:00<00:01, 40.68batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:00<00:01, 43.52batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:01<00:01, 46.26batch/s]

Training batches on cuda:  50%|█████     | 52/103 [00:01<00:01, 46.63batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:01<00:01, 45.40batch/s]

Training batches on cuda:  60%|██████    | 62/103 [00:01<00:00, 46.07batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:01<00:00, 46.60batch/s]

Training batches on cuda:  70%|██████▉   | 72/103 [00:01<00:00, 43.82batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:01<00:00, 42.68batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:01<00:00, 44.75batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:01<00:00, 46.19batch/s]

Training batches on cuda:  91%|█████████▏| 94/103 [00:02<00:00, 46.14batch/s]

Training batches on cuda:  97%|█████████▋| 100/103 [00:02<00:00, 48.31batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:16<00:09,  2.31s/epoch, loss=0.000262, prev_loss=0.000261]
Training epochs on cuda:  70%|███████   | 7/10 [00:16<00:07,  2.35s/epoch, loss=0.000262, prev_loss=0.000261]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:03, 25.01batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:03, 29.71batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:02, 34.26batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:00<00:02, 38.89batch/s]

Training batches on cuda:  25%|██▌       | 26/103 [00:00<00:01, 40.50batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:00<00:01, 40.40batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:00<00:01, 42.54batch/s]

Training batches on cuda:  41%|████      | 42/103 [00:00<00:01, 44.09batch/s]

Training batches on cuda:  47%|████▋     | 48/103 [00:01<00:01, 45.88batch/s]

Training batches on cuda:  52%|█████▏    | 54/103 [00:01<00:01, 48.27batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:01<00:00, 47.46batch/s]

Training batches on cuda:  62%|██████▏   | 64/103 [00:01<00:00, 43.63batch/s]

Training batches on cuda:  68%|██████▊   | 70/103 [00:01<00:00, 45.39batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:01<00:00, 46.60batch/s]

Training batches on cuda:  78%|███████▊  | 80/103 [00:01<00:00, 45.42batch/s]

Training batches on cuda:  83%|████████▎ | 86/103 [00:01<00:00, 48.09batch/s]

Training batches on cuda:  89%|████████▉ | 92/103 [00:01<00:00, 49.69batch/s]

Training batches on cuda:  95%|█████████▌| 98/103 [00:02<00:00, 50.69batch/s]

                                                                             
Training epochs on cuda:  70%|███████   | 7/10 [00:18<00:07,  2.35s/epoch, loss=0.000261, prev_loss=0.000262]
Training epochs on cuda:  80%|████████  | 8/10 [00:18<00:04,  2.38s/epoch, loss=0.000261, prev_loss=0.000262]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:03, 26.75batch/s]

Training batches on cuda:   8%|▊         | 8/103 [00:00<00:03, 29.93batch/s]

Training batches on cuda:  14%|█▎        | 14/103 [00:00<00:02, 33.49batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:02, 37.04batch/s]

Training batches on cuda:  23%|██▎       | 24/103 [00:00<00:01, 40.09batch/s]

Training batches on cuda:  27%|██▋       | 28/103 [00:00<00:02, 36.84batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:00<00:01, 39.95batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:00<00:01, 42.83batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:01<00:01, 44.86batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:01<00:01, 47.36batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:01<00:00, 49.21batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:01<00:00, 50.91batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:01<00:00, 51.58batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:01<00:00, 52.94batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:01<00:00, 52.52batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:01<00:00, 52.82batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:01<00:00, 52.49batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:02<00:00, 52.28batch/s]

                                                                             
Training epochs on cuda:  80%|████████  | 8/10 [00:21<00:04,  2.38s/epoch, loss=0.000262, prev_loss=0.000261]
Training epochs on cuda:  90%|█████████ | 9/10 [00:21<00:02,  2.38s/epoch, loss=0.000262, prev_loss=0.000261]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/103 [00:00<00:05, 19.55batch/s]

Training batches on cuda:   8%|▊         | 8/103 [00:00<00:03, 24.07batch/s]

Training batches on cuda:  14%|█▎        | 14/103 [00:00<00:03, 28.83batch/s]

Training batches on cuda:  19%|█▉        | 20/103 [00:00<00:02, 33.37batch/s]

Training batches on cuda:  25%|██▌       | 26/103 [00:00<00:02, 37.55batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:00<00:01, 39.77batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:00<00:01, 43.20batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:00<00:01, 47.00batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:00<00:01, 48.59batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:01<00:00, 51.07batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:01<00:00, 52.25batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:01<00:00, 53.60batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:01<00:00, 54.31batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:01<00:00, 54.88batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:01<00:00, 54.84batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:01<00:00, 55.23batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:01<00:00, 55.26batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:01<00:00, 55.83batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:23<00:02,  2.38s/epoch, loss=0.000261, prev_loss=0.000262]2021-09-16:17:34:40,900 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:34:41,996 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:34:57,48 INFO     [evaluator.py:657] Evaluation took 15.05s seconds
2021-09-16:17:34:57,159 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  7.19s/epoch, loss=0.000261, prev_loss=0.000262]
Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  3.95s/epoch, loss=0.000261, prev_loss=0.000262]
2021-09-16:17:34:57,160 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpioae44tt'
2021-09-16:17:34:57,214 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpioae44tt' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.31ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.40ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.50ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.60ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:17:35:12,106 INFO     [evaluator.py:657] Evaluation took 14.87s seconds
[I 2021-09-16 17:35:12,112] Trial 9 finished with value: -0.0012999890345517517 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.9871166498659066, 'optimizer.lr': 0.0993287613866088, 'negative_sampler.num_negs_per_pos': 4, 'training.batch_size': 3032}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:35:12,133 WARNING  [api.py:823] No random seed is specified. Setting to 959141333.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 16/9708 [00:00<01:01, 156.34batch/s]

Training batches on cuda:   0%|          | 34/9708 [00:00<01:00, 160.94batch/s]

Training batches on cuda:   1%|          | 54/9708 [00:00<00:56, 170.77batch/s]

Training batches on cuda:   1%|          | 77/9708 [00:00<00:52, 183.56batch/s]

Training batches on cuda:   1%|          | 99/9708 [00:00<00:50, 190.39batch/s]

Training batches on cuda:   1%|▏         | 123/9708 [00:00<00:47, 201.48batch/s]

Training batches on cuda:   1%|▏         | 145/9708 [00:00<00:46, 205.96batch/s]

Training batches on cuda:   2%|▏         | 168/9708 [00:00<00:45, 211.03batch/s]

Training batches on cuda:   2%|▏         | 190/9708 [00:00<00:44, 211.69batch/s]

Training batches on cuda:   2%|▏         | 213/9708 [00:01<00:43, 216.82batch/s]

Training batches on cuda:   2%|▏         | 235/9708 [00:01<00:44, 213.94batch/s]

Training batches on cuda:   3%|▎         | 258/9708 [00:01<00:43, 217.24batch/s]

Training batches on cuda:   3%|▎         | 280/9708 [00:01<00:43, 216.79batch/s]

Training batches on cuda:   3%|▎         | 303/9708 [00:01<00:42, 220.13batch/s]

Training batches on cuda:   3%|▎         | 327/9708 [00:01<00:41, 223.60batch/s]

Training batches on cuda:   4%|▎         | 350/9708 [00:01<00:41, 225.26batch/s]

Training batches on cuda:   4%|▍         | 374/9708 [00:01<00:41, 226.84batch/s]

Training batches on cuda:   4%|▍         | 398/9708 [00:01<00:40, 227.80batch/s]

Training batches on cuda:   4%|▍         | 421/9708 [00:01<00:40, 227.18batch/s]

Training batches on cuda:   5%|▍         | 444/9708 [00:02<00:40, 226.44batch/s]

Training batches on cuda:   5%|▍         | 467/9708 [00:02<00:40, 225.68batch/s]

Training batches on cuda:   5%|▌         | 490/9708 [00:02<00:41, 224.12batch/s]

Training batches on cuda:   5%|▌         | 513/9708 [00:02<00:41, 222.73batch/s]

Training batches on cuda:   6%|▌         | 536/9708 [00:02<00:40, 224.12batch/s]

Training batches on cuda:   6%|▌         | 559/9708 [00:02<00:41, 222.49batch/s]

Training batches on cuda:   6%|▌         | 582/9708 [00:02<00:41, 217.43batch/s]

Training batches on cuda:   6%|▌         | 604/9708 [00:02<00:44, 205.75batch/s]

Training batches on cuda:   6%|▋         | 625/9708 [00:02<00:47, 191.38batch/s]

Training batches on cuda:   7%|▋         | 645/9708 [00:03<00:46, 192.91batch/s]

Training batches on cuda:   7%|▋         | 668/9708 [00:03<00:44, 201.36batch/s]

Training batches on cuda:   7%|▋         | 690/9708 [00:03<00:43, 205.96batch/s]

Training batches on cuda:   7%|▋         | 713/9708 [00:03<00:42, 210.03batch/s]

Training batches on cuda:   8%|▊         | 736/9708 [00:03<00:42, 213.01batch/s]

Training batches on cuda:   8%|▊         | 759/9708 [00:03<00:41, 215.52batch/s]

Training batches on cuda:   8%|▊         | 782/9708 [00:03<00:41, 217.15batch/s]

Training batches on cuda:   8%|▊         | 804/9708 [00:03<00:40, 217.52batch/s]

Training batches on cuda:   9%|▊         | 826/9708 [00:03<00:40, 217.37batch/s]

Training batches on cuda:   9%|▊         | 849/9708 [00:03<00:40, 218.88batch/s]

Training batches on cuda:   9%|▉         | 871/9708 [00:04<00:40, 215.75batch/s]

Training batches on cuda:   9%|▉         | 893/9708 [00:04<00:42, 206.43batch/s]

Training batches on cuda:   9%|▉         | 914/9708 [00:04<00:45, 195.39batch/s]

Training batches on cuda:  10%|▉         | 934/9708 [00:04<00:46, 189.41batch/s]

Training batches on cuda:  10%|▉         | 954/9708 [00:04<00:46, 186.92batch/s]

Training batches on cuda:  10%|█         | 973/9708 [00:04<00:47, 185.41batch/s]

Training batches on cuda:  10%|█         | 994/9708 [00:04<00:45, 190.57batch/s]

Training batches on cuda:  10%|█         | 1016/9708 [00:04<00:44, 197.21batch/s]

Training batches on cuda:  11%|█         | 1038/9708 [00:04<00:43, 201.19batch/s]

Training batches on cuda:  11%|█         | 1059/9708 [00:05<00:42, 203.70batch/s]

Training batches on cuda:  11%|█         | 1080/9708 [00:05<00:43, 196.36batch/s]

Training batches on cuda:  11%|█▏        | 1100/9708 [00:05<00:45, 190.02batch/s]

Training batches on cuda:  12%|█▏        | 1121/9708 [00:05<00:43, 195.50batch/s]

Training batches on cuda:  12%|█▏        | 1142/9708 [00:05<00:42, 199.41batch/s]

Training batches on cuda:  12%|█▏        | 1163/9708 [00:05<00:43, 197.58batch/s]

Training batches on cuda:  12%|█▏        | 1184/9708 [00:05<00:42, 200.90batch/s]

Training batches on cuda:  12%|█▏        | 1205/9708 [00:05<00:42, 201.57batch/s]

Training batches on cuda:  13%|█▎        | 1226/9708 [00:05<00:41, 203.52batch/s]

Training batches on cuda:  13%|█▎        | 1247/9708 [00:06<00:45, 187.44batch/s]

Training batches on cuda:  13%|█▎        | 1269/9708 [00:06<00:43, 194.11batch/s]

Training batches on cuda:  13%|█▎        | 1291/9708 [00:06<00:42, 198.80batch/s]

Training batches on cuda:  14%|█▎        | 1313/9708 [00:06<00:41, 202.15batch/s]

Training batches on cuda:  14%|█▍        | 1335/9708 [00:06<00:40, 204.98batch/s]

Training batches on cuda:  14%|█▍        | 1356/9708 [00:06<00:40, 206.33batch/s]

Training batches on cuda:  14%|█▍        | 1378/9708 [00:06<00:39, 209.07batch/s]

Training batches on cuda:  14%|█▍        | 1399/9708 [00:06<00:39, 208.48batch/s]

Training batches on cuda:  15%|█▍        | 1420/9708 [00:06<00:39, 208.19batch/s]

Training batches on cuda:  15%|█▍        | 1442/9708 [00:06<00:39, 210.22batch/s]

Training batches on cuda:  15%|█▌        | 1465/9708 [00:07<00:38, 214.35batch/s]

Training batches on cuda:  15%|█▌        | 1489/9708 [00:07<00:37, 218.98batch/s]

Training batches on cuda:  16%|█▌        | 1512/9708 [00:07<00:37, 221.12batch/s]

Training batches on cuda:  16%|█▌        | 1536/9708 [00:07<00:36, 223.73batch/s]

Training batches on cuda:  16%|█▌        | 1559/9708 [00:07<00:36, 224.60batch/s]

Training batches on cuda:  16%|█▋        | 1583/9708 [00:07<00:35, 226.53batch/s]

Training batches on cuda:  17%|█▋        | 1606/9708 [00:07<00:35, 226.95batch/s]

Training batches on cuda:  17%|█▋        | 1630/9708 [00:07<00:35, 228.60batch/s]

Training batches on cuda:  17%|█▋        | 1653/9708 [00:07<00:35, 226.01batch/s]

Training batches on cuda:  17%|█▋        | 1677/9708 [00:07<00:35, 228.50batch/s]

Training batches on cuda:  18%|█▊        | 1700/9708 [00:08<00:35, 225.41batch/s]

Training batches on cuda:  18%|█▊        | 1724/9708 [00:08<00:35, 226.72batch/s]

Training batches on cuda:  18%|█▊        | 1747/9708 [00:08<00:35, 225.99batch/s]

Training batches on cuda:  18%|█▊        | 1771/9708 [00:08<00:34, 226.85batch/s]

Training batches on cuda:  18%|█▊        | 1794/9708 [00:08<00:35, 223.57batch/s]

Training batches on cuda:  19%|█▊        | 1817/9708 [00:08<00:36, 216.01batch/s]

Training batches on cuda:  19%|█▉        | 1839/9708 [00:08<00:36, 214.69batch/s]

Training batches on cuda:  19%|█▉        | 1861/9708 [00:08<00:38, 203.38batch/s]

Training batches on cuda:  19%|█▉        | 1882/9708 [00:08<00:40, 192.39batch/s]

Training batches on cuda:  20%|█▉        | 1902/9708 [00:09<00:40, 192.54batch/s]

Training batches on cuda:  20%|█▉        | 1923/9708 [00:09<00:39, 195.16batch/s]

Training batches on cuda:  20%|██        | 1943/9708 [00:09<00:39, 195.14batch/s]

Training batches on cuda:  20%|██        | 1963/9708 [00:09<00:39, 194.85batch/s]

Training batches on cuda:  20%|██        | 1983/9708 [00:09<00:40, 192.71batch/s]

Training batches on cuda:  21%|██        | 2003/9708 [00:09<00:40, 189.80batch/s]

Training batches on cuda:  21%|██        | 2024/9708 [00:09<00:39, 194.72batch/s]

Training batches on cuda:  21%|██        | 2045/9708 [00:09<00:38, 198.18batch/s]

Training batches on cuda:  21%|██▏       | 2065/9708 [00:09<00:38, 197.82batch/s]

Training batches on cuda:  21%|██▏       | 2086/9708 [00:09<00:37, 200.62batch/s]

Training batches on cuda:  22%|██▏       | 2107/9708 [00:10<00:38, 197.59batch/s]

Training batches on cuda:  22%|██▏       | 2127/9708 [00:10<00:38, 194.48batch/s]

Training batches on cuda:  22%|██▏       | 2147/9708 [00:10<00:39, 190.22batch/s]

Training batches on cuda:  22%|██▏       | 2167/9708 [00:10<00:39, 189.01batch/s]

Training batches on cuda:  23%|██▎       | 2186/9708 [00:10<00:40, 187.87batch/s]

Training batches on cuda:  23%|██▎       | 2206/9708 [00:10<00:39, 190.85batch/s]

Training batches on cuda:  23%|██▎       | 2226/9708 [00:10<00:39, 191.83batch/s]

Training batches on cuda:  23%|██▎       | 2246/9708 [00:10<00:38, 193.24batch/s]

Training batches on cuda:  23%|██▎       | 2266/9708 [00:10<00:38, 194.67batch/s]

Training batches on cuda:  24%|██▎       | 2287/9708 [00:11<00:37, 196.53batch/s]

Training batches on cuda:  24%|██▍       | 2307/9708 [00:11<00:38, 194.09batch/s]

Training batches on cuda:  24%|██▍       | 2327/9708 [00:11<00:38, 193.82batch/s]

Training batches on cuda:  24%|██▍       | 2348/9708 [00:11<00:37, 195.62batch/s]

Training batches on cuda:  24%|██▍       | 2368/9708 [00:11<00:37, 196.43batch/s]

Training batches on cuda:  25%|██▍       | 2388/9708 [00:11<00:38, 191.90batch/s]

Training batches on cuda:  25%|██▍       | 2409/9708 [00:11<00:37, 194.88batch/s]

Training batches on cuda:  25%|██▌       | 2429/9708 [00:11<00:37, 195.32batch/s]

Training batches on cuda:  25%|██▌       | 2449/9708 [00:11<00:37, 194.55batch/s]

Training batches on cuda:  25%|██▌       | 2469/9708 [00:11<00:37, 194.12batch/s]

Training batches on cuda:  26%|██▌       | 2490/9708 [00:12<00:36, 197.76batch/s]

Training batches on cuda:  26%|██▌       | 2510/9708 [00:12<00:36, 197.97batch/s]

Training batches on cuda:  26%|██▌       | 2531/9708 [00:12<00:35, 199.83batch/s]

Training batches on cuda:  26%|██▋       | 2551/9708 [00:12<00:37, 191.06batch/s]

Training batches on cuda:  26%|██▋       | 2571/9708 [00:12<00:38, 187.51batch/s]

Training batches on cuda:  27%|██▋       | 2593/9708 [00:12<00:36, 195.05batch/s]

Training batches on cuda:  27%|██▋       | 2616/9708 [00:12<00:35, 202.23batch/s]

Training batches on cuda:  27%|██▋       | 2640/9708 [00:12<00:33, 210.45batch/s]

Training batches on cuda:  27%|██▋       | 2663/9708 [00:12<00:32, 214.79batch/s]

Training batches on cuda:  28%|██▊       | 2686/9708 [00:13<00:32, 215.98batch/s]

Training batches on cuda:  28%|██▊       | 2709/9708 [00:13<00:31, 219.94batch/s]

Training batches on cuda:  28%|██▊       | 2732/9708 [00:13<00:31, 222.82batch/s]

Training batches on cuda:  28%|██▊       | 2755/9708 [00:13<00:30, 224.79batch/s]

Training batches on cuda:  29%|██▊       | 2778/9708 [00:13<00:30, 226.01batch/s]

Training batches on cuda:  29%|██▉       | 2802/9708 [00:13<00:30, 227.63batch/s]

Training batches on cuda:  29%|██▉       | 2825/9708 [00:13<00:30, 225.95batch/s]

Training batches on cuda:  29%|██▉       | 2848/9708 [00:13<00:31, 217.95batch/s]

Training batches on cuda:  30%|██▉       | 2870/9708 [00:13<00:32, 212.03batch/s]

Training batches on cuda:  30%|██▉       | 2892/9708 [00:13<00:32, 208.48batch/s]

Training batches on cuda:  30%|███       | 2913/9708 [00:14<00:33, 205.67batch/s]

Training batches on cuda:  30%|███       | 2934/9708 [00:14<00:32, 206.43batch/s]

Training batches on cuda:  30%|███       | 2955/9708 [00:14<00:32, 207.06batch/s]

Training batches on cuda:  31%|███       | 2977/9708 [00:14<00:32, 208.83batch/s]

Training batches on cuda:  31%|███       | 2999/9708 [00:14<00:31, 211.22batch/s]

Training batches on cuda:  31%|███       | 3022/9708 [00:14<00:31, 214.91batch/s]

Training batches on cuda:  31%|███▏      | 3045/9708 [00:14<00:30, 216.88batch/s]

Training batches on cuda:  32%|███▏      | 3068/9708 [00:14<00:30, 218.50batch/s]

Training batches on cuda:  32%|███▏      | 3091/9708 [00:14<00:29, 221.76batch/s]

Training batches on cuda:  32%|███▏      | 3114/9708 [00:15<00:31, 206.58batch/s]

Training batches on cuda:  32%|███▏      | 3135/9708 [00:15<00:31, 206.45batch/s]

Training batches on cuda:  33%|███▎      | 3158/9708 [00:15<00:31, 211.00batch/s]

Training batches on cuda:  33%|███▎      | 3181/9708 [00:15<00:30, 214.41batch/s]

Training batches on cuda:  33%|███▎      | 3204/9708 [00:15<00:29, 218.61batch/s]

Training batches on cuda:  33%|███▎      | 3228/9708 [00:15<00:28, 224.25batch/s]

Training batches on cuda:  34%|███▎      | 3253/9708 [00:15<00:28, 229.21batch/s]

Training batches on cuda:  34%|███▍      | 3278/9708 [00:15<00:27, 232.58batch/s]

Training batches on cuda:  34%|███▍      | 3303/9708 [00:15<00:27, 235.10batch/s]

Training batches on cuda:  34%|███▍      | 3328/9708 [00:15<00:26, 236.96batch/s]

Training batches on cuda:  35%|███▍      | 3352/9708 [00:16<00:31, 204.38batch/s]

Training batches on cuda:  35%|███▍      | 3374/9708 [00:16<00:31, 200.15batch/s]

Training batches on cuda:  35%|███▍      | 3395/9708 [00:16<00:32, 192.11batch/s]

Training batches on cuda:  35%|███▌      | 3418/9708 [00:16<00:31, 201.41batch/s]

Training batches on cuda:  35%|███▌      | 3441/9708 [00:16<00:30, 206.76batch/s]

Training batches on cuda:  36%|███▌      | 3463/9708 [00:16<00:30, 205.68batch/s]

Training batches on cuda:  36%|███▌      | 3484/9708 [00:16<00:30, 203.96batch/s]

Training batches on cuda:  36%|███▌      | 3505/9708 [00:16<00:30, 201.16batch/s]

Training batches on cuda:  36%|███▋      | 3527/9708 [00:16<00:30, 203.88batch/s]

Training batches on cuda:  37%|███▋      | 3548/9708 [00:17<00:30, 204.89batch/s]

Training batches on cuda:  37%|███▋      | 3570/9708 [00:17<00:29, 208.97batch/s]

Training batches on cuda:  37%|███▋      | 3593/9708 [00:17<00:28, 212.55batch/s]

Training batches on cuda:  37%|███▋      | 3615/9708 [00:17<00:28, 214.42batch/s]

Training batches on cuda:  37%|███▋      | 3640/9708 [00:17<00:27, 223.64batch/s]

Training batches on cuda:  38%|███▊      | 3664/9708 [00:17<00:26, 227.21batch/s]

Training batches on cuda:  38%|███▊      | 3687/9708 [00:17<00:26, 223.85batch/s]

Training batches on cuda:  38%|███▊      | 3710/9708 [00:17<00:26, 223.12batch/s]

Training batches on cuda:  38%|███▊      | 3734/9708 [00:17<00:26, 226.22batch/s]

Training batches on cuda:  39%|███▊      | 3757/9708 [00:18<00:28, 211.61batch/s]

Training batches on cuda:  39%|███▉      | 3779/9708 [00:18<00:29, 198.69batch/s]

Training batches on cuda:  39%|███▉      | 3800/9708 [00:18<00:30, 196.52batch/s]

Training batches on cuda:  39%|███▉      | 3820/9708 [00:18<00:30, 192.19batch/s]

Training batches on cuda:  40%|███▉      | 3844/9708 [00:18<00:28, 203.56batch/s]

Training batches on cuda:  40%|███▉      | 3870/9708 [00:18<00:27, 216.19batch/s]

Training batches on cuda:  40%|████      | 3893/9708 [00:18<00:26, 218.72batch/s]

Training batches on cuda:  40%|████      | 3916/9708 [00:18<00:26, 218.47batch/s]

Training batches on cuda:  41%|████      | 3939/9708 [00:18<00:26, 218.87batch/s]

Training batches on cuda:  41%|████      | 3962/9708 [00:18<00:25, 221.46batch/s]

Training batches on cuda:  41%|████      | 3985/9708 [00:19<00:25, 222.10batch/s]

Training batches on cuda:  41%|████▏     | 4008/9708 [00:19<00:25, 223.13batch/s]

Training batches on cuda:  42%|████▏     | 4031/9708 [00:19<00:25, 220.50batch/s]

Training batches on cuda:  42%|████▏     | 4054/9708 [00:19<00:25, 221.11batch/s]

Training batches on cuda:  42%|████▏     | 4077/9708 [00:19<00:25, 222.65batch/s]

Training batches on cuda:  42%|████▏     | 4100/9708 [00:19<00:25, 223.99batch/s]

Training batches on cuda:  42%|████▏     | 4123/9708 [00:19<00:24, 223.45batch/s]

Training batches on cuda:  43%|████▎     | 4146/9708 [00:19<00:25, 219.05batch/s]

Training batches on cuda:  43%|████▎     | 4168/9708 [00:19<00:26, 208.08batch/s]

Training batches on cuda:  43%|████▎     | 4190/9708 [00:20<00:26, 210.29batch/s]

Training batches on cuda:  43%|████▎     | 4212/9708 [00:20<00:25, 212.63batch/s]

Training batches on cuda:  44%|████▎     | 4235/9708 [00:20<00:25, 217.22batch/s]

Training batches on cuda:  44%|████▍     | 4258/9708 [00:20<00:24, 220.46batch/s]

Training batches on cuda:  44%|████▍     | 4282/9708 [00:20<00:24, 224.42batch/s]

Training batches on cuda:  44%|████▍     | 4306/9708 [00:20<00:23, 226.07batch/s]

Training batches on cuda:  45%|████▍     | 4330/9708 [00:20<00:23, 227.98batch/s]

Training batches on cuda:  45%|████▍     | 4354/9708 [00:20<00:23, 229.61batch/s]

Training batches on cuda:  45%|████▌     | 4377/9708 [00:20<00:23, 224.57batch/s]

Training batches on cuda:  45%|████▌     | 4400/9708 [00:20<00:23, 222.01batch/s]

Training batches on cuda:  46%|████▌     | 4423/9708 [00:21<00:26, 200.22batch/s]

Training batches on cuda:  46%|████▌     | 4447/9708 [00:21<00:25, 210.23batch/s]

Training batches on cuda:  46%|████▌     | 4470/9708 [00:21<00:24, 215.68batch/s]

Training batches on cuda:  46%|████▋     | 4494/9708 [00:21<00:23, 221.05batch/s]

Training batches on cuda:  47%|████▋     | 4518/9708 [00:21<00:22, 225.87batch/s]

Training batches on cuda:  47%|████▋     | 4541/9708 [00:21<00:23, 219.96batch/s]

Training batches on cuda:  47%|████▋     | 4564/9708 [00:21<00:23, 215.06batch/s]

Training batches on cuda:  47%|████▋     | 4586/9708 [00:21<00:24, 212.19batch/s]

Training batches on cuda:  47%|████▋     | 4608/9708 [00:21<00:25, 199.74batch/s]

Training batches on cuda:  48%|████▊     | 4629/9708 [00:22<00:26, 189.85batch/s]

Training batches on cuda:  48%|████▊     | 4651/9708 [00:22<00:25, 197.17batch/s]

Training batches on cuda:  48%|████▊     | 4674/9708 [00:22<00:24, 203.62batch/s]

Training batches on cuda:  48%|████▊     | 4695/9708 [00:22<00:24, 201.26batch/s]

Training batches on cuda:  49%|████▊     | 4716/9708 [00:22<00:24, 203.35batch/s]

Training batches on cuda:  49%|████▉     | 4738/9708 [00:22<00:24, 205.91batch/s]

Training batches on cuda:  49%|████▉     | 4761/9708 [00:22<00:23, 210.64batch/s]

Training batches on cuda:  49%|████▉     | 4783/9708 [00:22<00:23, 210.27batch/s]

Training batches on cuda:  49%|████▉     | 4805/9708 [00:22<00:23, 208.36batch/s]

Training batches on cuda:  50%|████▉     | 4826/9708 [00:23<00:23, 205.50batch/s]

Training batches on cuda:  50%|████▉     | 4847/9708 [00:23<00:24, 202.03batch/s]

Training batches on cuda:  50%|█████     | 4868/9708 [00:23<00:24, 200.63batch/s]

Training batches on cuda:  50%|█████     | 4889/9708 [00:23<00:23, 201.29batch/s]

Training batches on cuda:  51%|█████     | 4910/9708 [00:23<00:24, 197.18batch/s]

Training batches on cuda:  51%|█████     | 4930/9708 [00:23<00:24, 196.50batch/s]

Training batches on cuda:  51%|█████     | 4952/9708 [00:23<00:23, 201.60batch/s]

Training batches on cuda:  51%|█████     | 4973/9708 [00:23<00:23, 200.12batch/s]

Training batches on cuda:  51%|█████▏    | 4994/9708 [00:23<00:23, 201.32batch/s]

Training batches on cuda:  52%|█████▏    | 5015/9708 [00:23<00:23, 199.56batch/s]

Training batches on cuda:  52%|█████▏    | 5035/9708 [00:24<00:23, 198.81batch/s]

Training batches on cuda:  52%|█████▏    | 5056/9708 [00:24<00:23, 200.69batch/s]

Training batches on cuda:  52%|█████▏    | 5078/9708 [00:24<00:22, 204.03batch/s]

Training batches on cuda:  53%|█████▎    | 5100/9708 [00:24<00:22, 207.90batch/s]

Training batches on cuda:  53%|█████▎    | 5122/9708 [00:24<00:21, 211.07batch/s]

Training batches on cuda:  53%|█████▎    | 5145/9708 [00:24<00:21, 214.31batch/s]

Training batches on cuda:  53%|█████▎    | 5168/9708 [00:24<00:20, 216.72batch/s]

Training batches on cuda:  53%|█████▎    | 5190/9708 [00:24<00:20, 216.69batch/s]

Training batches on cuda:  54%|█████▎    | 5212/9708 [00:24<00:21, 213.80batch/s]

Training batches on cuda:  54%|█████▍    | 5234/9708 [00:25<00:21, 205.86batch/s]

Training batches on cuda:  54%|█████▍    | 5255/9708 [00:25<00:21, 205.36batch/s]

Training batches on cuda:  54%|█████▍    | 5276/9708 [00:25<00:22, 200.89batch/s]

Training batches on cuda:  55%|█████▍    | 5298/9708 [00:25<00:21, 205.94batch/s]

Training batches on cuda:  55%|█████▍    | 5319/9708 [00:25<00:21, 205.43batch/s]

Training batches on cuda:  55%|█████▌    | 5340/9708 [00:25<00:21, 206.67batch/s]

Training batches on cuda:  55%|█████▌    | 5361/9708 [00:25<00:21, 205.92batch/s]

Training batches on cuda:  55%|█████▌    | 5382/9708 [00:25<00:21, 205.37batch/s]

Training batches on cuda:  56%|█████▌    | 5403/9708 [00:25<00:21, 204.24batch/s]

Training batches on cuda:  56%|█████▌    | 5424/9708 [00:25<00:20, 205.05batch/s]

Training batches on cuda:  56%|█████▌    | 5445/9708 [00:26<00:21, 199.04batch/s]

Training batches on cuda:  56%|█████▋    | 5465/9708 [00:26<00:23, 180.48batch/s]

Training batches on cuda:  56%|█████▋    | 5484/9708 [00:26<00:23, 181.61batch/s]

Training batches on cuda:  57%|█████▋    | 5505/9708 [00:26<00:22, 188.00batch/s]

Training batches on cuda:  57%|█████▋    | 5526/9708 [00:26<00:21, 192.89batch/s]

Training batches on cuda:  57%|█████▋    | 5547/9708 [00:26<00:21, 196.33batch/s]

Training batches on cuda:  57%|█████▋    | 5568/9708 [00:26<00:20, 199.22batch/s]

Training batches on cuda:  58%|█████▊    | 5589/9708 [00:26<00:20, 200.87batch/s]

Training batches on cuda:  58%|█████▊    | 5610/9708 [00:26<00:20, 198.81batch/s]

Training batches on cuda:  58%|█████▊    | 5630/9708 [00:27<00:21, 192.78batch/s]

Training batches on cuda:  58%|█████▊    | 5650/9708 [00:27<00:21, 191.87batch/s]

Training batches on cuda:  58%|█████▊    | 5670/9708 [00:27<00:20, 192.82batch/s]

Training batches on cuda:  59%|█████▊    | 5690/9708 [00:27<00:20, 191.50batch/s]

Training batches on cuda:  59%|█████▉    | 5710/9708 [00:27<00:20, 193.56batch/s]

Training batches on cuda:  59%|█████▉    | 5730/9708 [00:27<00:20, 190.49batch/s]

Training batches on cuda:  59%|█████▉    | 5750/9708 [00:27<00:21, 186.96batch/s]

Training batches on cuda:  59%|█████▉    | 5770/9708 [00:27<00:20, 188.77batch/s]

Training batches on cuda:  60%|█████▉    | 5789/9708 [00:27<00:20, 188.06batch/s]

Training batches on cuda:  60%|█████▉    | 5808/9708 [00:27<00:20, 186.26batch/s]

Training batches on cuda:  60%|██████    | 5827/9708 [00:28<00:21, 183.81batch/s]

Training batches on cuda:  60%|██████    | 5846/9708 [00:28<00:21, 183.79batch/s]

Training batches on cuda:  60%|██████    | 5865/9708 [00:28<00:20, 183.20batch/s]

Training batches on cuda:  61%|██████    | 5887/9708 [00:28<00:19, 191.38batch/s]

Training batches on cuda:  61%|██████    | 5908/9708 [00:28<00:19, 196.33batch/s]

Training batches on cuda:  61%|██████    | 5929/9708 [00:28<00:19, 197.72batch/s]

Training batches on cuda:  61%|██████▏   | 5951/9708 [00:28<00:18, 201.47batch/s]

Training batches on cuda:  62%|██████▏   | 5972/9708 [00:28<00:18, 202.94batch/s]

Training batches on cuda:  62%|██████▏   | 5995/9708 [00:28<00:17, 209.39batch/s]

Training batches on cuda:  62%|██████▏   | 6017/9708 [00:28<00:17, 211.04batch/s]

Training batches on cuda:  62%|██████▏   | 6040/9708 [00:29<00:17, 215.36batch/s]

Training batches on cuda:  62%|██████▏   | 6062/9708 [00:29<00:17, 208.55batch/s]

Training batches on cuda:  63%|██████▎   | 6083/9708 [00:29<00:18, 193.93batch/s]

Training batches on cuda:  63%|██████▎   | 6105/9708 [00:29<00:18, 198.09batch/s]

Training batches on cuda:  63%|██████▎   | 6127/9708 [00:29<00:17, 201.72batch/s]

Training batches on cuda:  63%|██████▎   | 6148/9708 [00:29<00:17, 203.69batch/s]

Training batches on cuda:  64%|██████▎   | 6170/9708 [00:29<00:17, 206.88batch/s]

Training batches on cuda:  64%|██████▍   | 6192/9708 [00:29<00:16, 209.14batch/s]

Training batches on cuda:  64%|██████▍   | 6214/9708 [00:29<00:16, 211.01batch/s]

Training batches on cuda:  64%|██████▍   | 6236/9708 [00:30<00:16, 209.58batch/s]

Training batches on cuda:  64%|██████▍   | 6258/9708 [00:30<00:16, 211.66batch/s]

Training batches on cuda:  65%|██████▍   | 6280/9708 [00:30<00:16, 207.24batch/s]

Training batches on cuda:  65%|██████▍   | 6301/9708 [00:30<00:16, 206.45batch/s]

Training batches on cuda:  65%|██████▌   | 6322/9708 [00:30<00:17, 198.86batch/s]

Training batches on cuda:  65%|██████▌   | 6343/9708 [00:30<00:16, 201.43batch/s]

Training batches on cuda:  66%|██████▌   | 6365/9708 [00:30<00:16, 205.79batch/s]

Training batches on cuda:  66%|██████▌   | 6386/9708 [00:30<00:16, 204.06batch/s]

Training batches on cuda:  66%|██████▌   | 6409/9708 [00:30<00:15, 209.54batch/s]

Training batches on cuda:  66%|██████▌   | 6431/9708 [00:31<00:15, 209.83batch/s]

Training batches on cuda:  66%|██████▋   | 6454/9708 [00:31<00:15, 214.64batch/s]

Training batches on cuda:  67%|██████▋   | 6476/9708 [00:31<00:16, 194.13batch/s]

Training batches on cuda:  67%|██████▋   | 6499/9708 [00:31<00:15, 202.48batch/s]

Training batches on cuda:  67%|██████▋   | 6521/9708 [00:31<00:15, 205.25batch/s]

Training batches on cuda:  67%|██████▋   | 6542/9708 [00:31<00:15, 204.13batch/s]

Training batches on cuda:  68%|██████▊   | 6564/9708 [00:31<00:15, 206.70batch/s]

Training batches on cuda:  68%|██████▊   | 6585/9708 [00:31<00:15, 200.70batch/s]

Training batches on cuda:  68%|██████▊   | 6606/9708 [00:31<00:15, 195.23batch/s]

Training batches on cuda:  68%|██████▊   | 6626/9708 [00:31<00:16, 190.01batch/s]

Training batches on cuda:  68%|██████▊   | 6646/9708 [00:32<00:16, 187.79batch/s]

Training batches on cuda:  69%|██████▊   | 6665/9708 [00:32<00:16, 186.30batch/s]

Training batches on cuda:  69%|██████▉   | 6686/9708 [00:32<00:15, 191.86batch/s]

Training batches on cuda:  69%|██████▉   | 6707/9708 [00:32<00:15, 196.08batch/s]

Training batches on cuda:  69%|██████▉   | 6728/9708 [00:32<00:15, 198.27batch/s]

Training batches on cuda:  70%|██████▉   | 6749/9708 [00:32<00:14, 200.49batch/s]

Training batches on cuda:  70%|██████▉   | 6770/9708 [00:32<00:14, 202.21batch/s]

Training batches on cuda:  70%|██████▉   | 6791/9708 [00:32<00:14, 202.95batch/s]

Training batches on cuda:  70%|███████   | 6812/9708 [00:32<00:14, 203.36batch/s]

Training batches on cuda:  70%|███████   | 6833/9708 [00:33<00:14, 204.54batch/s]

Training batches on cuda:  71%|███████   | 6854/9708 [00:33<00:13, 205.35batch/s]

Training batches on cuda:  71%|███████   | 6875/9708 [00:33<00:13, 203.78batch/s]

Training batches on cuda:  71%|███████   | 6896/9708 [00:33<00:14, 199.14batch/s]

Training batches on cuda:  71%|███████▏  | 6917/9708 [00:33<00:13, 200.68batch/s]

Training batches on cuda:  71%|███████▏  | 6938/9708 [00:33<00:13, 200.76batch/s]

Training batches on cuda:  72%|███████▏  | 6959/9708 [00:33<00:13, 199.66batch/s]

Training batches on cuda:  72%|███████▏  | 6979/9708 [00:33<00:13, 199.31batch/s]

Training batches on cuda:  72%|███████▏  | 7000/9708 [00:33<00:13, 200.41batch/s]

Training batches on cuda:  72%|███████▏  | 7021/9708 [00:33<00:13, 200.54batch/s]

Training batches on cuda:  73%|███████▎  | 7042/9708 [00:34<00:13, 200.45batch/s]

Training batches on cuda:  73%|███████▎  | 7063/9708 [00:34<00:13, 201.11batch/s]

Training batches on cuda:  73%|███████▎  | 7084/9708 [00:34<00:13, 200.36batch/s]

Training batches on cuda:  73%|███████▎  | 7105/9708 [00:34<00:13, 198.53batch/s]

Training batches on cuda:  73%|███████▎  | 7125/9708 [00:34<00:13, 197.45batch/s]

Training batches on cuda:  74%|███████▎  | 7146/9708 [00:34<00:12, 199.77batch/s]

Training batches on cuda:  74%|███████▍  | 7167/9708 [00:34<00:12, 201.66batch/s]

Training batches on cuda:  74%|███████▍  | 7188/9708 [00:34<00:12, 201.89batch/s]

Training batches on cuda:  74%|███████▍  | 7209/9708 [00:34<00:12, 204.15batch/s]

Training batches on cuda:  74%|███████▍  | 7230/9708 [00:35<00:12, 202.65batch/s]

Training batches on cuda:  75%|███████▍  | 7252/9708 [00:35<00:11, 205.07batch/s]

Training batches on cuda:  75%|███████▍  | 7273/9708 [00:35<00:11, 205.35batch/s]

Training batches on cuda:  75%|███████▌  | 7294/9708 [00:35<00:11, 204.53batch/s]

Training batches on cuda:  75%|███████▌  | 7315/9708 [00:35<00:11, 205.09batch/s]

Training batches on cuda:  76%|███████▌  | 7336/9708 [00:35<00:11, 201.39batch/s]

Training batches on cuda:  76%|███████▌  | 7357/9708 [00:35<00:11, 202.98batch/s]

Training batches on cuda:  76%|███████▌  | 7379/9708 [00:35<00:11, 205.12batch/s]

Training batches on cuda:  76%|███████▌  | 7401/9708 [00:35<00:11, 207.32batch/s]

Training batches on cuda:  76%|███████▋  | 7422/9708 [00:35<00:11, 207.05batch/s]

Training batches on cuda:  77%|███████▋  | 7444/9708 [00:36<00:10, 208.31batch/s]

Training batches on cuda:  77%|███████▋  | 7465/9708 [00:36<00:10, 208.62batch/s]

Training batches on cuda:  77%|███████▋  | 7486/9708 [00:36<00:11, 188.21batch/s]

Training batches on cuda:  77%|███████▋  | 7507/9708 [00:36<00:11, 192.80batch/s]

Training batches on cuda:  78%|███████▊  | 7528/9708 [00:36<00:11, 195.06batch/s]

Training batches on cuda:  78%|███████▊  | 7548/9708 [00:36<00:11, 180.13batch/s]

Training batches on cuda:  78%|███████▊  | 7568/9708 [00:36<00:11, 184.11batch/s]

Training batches on cuda:  78%|███████▊  | 7590/9708 [00:36<00:11, 192.11batch/s]

Training batches on cuda:  78%|███████▊  | 7611/9708 [00:36<00:10, 196.08batch/s]

Training batches on cuda:  79%|███████▊  | 7633/9708 [00:37<00:10, 200.16batch/s]

Training batches on cuda:  79%|███████▉  | 7654/9708 [00:37<00:10, 201.35batch/s]

Training batches on cuda:  79%|███████▉  | 7677/9708 [00:37<00:09, 207.52batch/s]

Training batches on cuda:  79%|███████▉  | 7700/9708 [00:37<00:09, 212.09batch/s]

Training batches on cuda:  80%|███████▉  | 7723/9708 [00:37<00:09, 215.18batch/s]

Training batches on cuda:  80%|███████▉  | 7745/9708 [00:37<00:09, 213.11batch/s]

Training batches on cuda:  80%|████████  | 7767/9708 [00:37<00:09, 210.29batch/s]

Training batches on cuda:  80%|████████  | 7789/9708 [00:37<00:09, 202.02batch/s]

Training batches on cuda:  80%|████████  | 7811/9708 [00:37<00:09, 205.53batch/s]

Training batches on cuda:  81%|████████  | 7833/9708 [00:37<00:09, 207.48batch/s]

Training batches on cuda:  81%|████████  | 7855/9708 [00:38<00:08, 209.46batch/s]

Training batches on cuda:  81%|████████  | 7877/9708 [00:38<00:08, 210.66batch/s]

Training batches on cuda:  81%|████████▏ | 7899/9708 [00:38<00:08, 211.68batch/s]

Training batches on cuda:  82%|████████▏ | 7921/9708 [00:38<00:08, 212.65batch/s]

Training batches on cuda:  82%|████████▏ | 7943/9708 [00:38<00:08, 211.23batch/s]

Training batches on cuda:  82%|████████▏ | 7965/9708 [00:38<00:08, 212.71batch/s]

Training batches on cuda:  82%|████████▏ | 7987/9708 [00:38<00:08, 211.82batch/s]

Training batches on cuda:  82%|████████▏ | 8009/9708 [00:38<00:08, 212.31batch/s]

Training batches on cuda:  83%|████████▎ | 8031/9708 [00:38<00:07, 213.71batch/s]

Training batches on cuda:  83%|████████▎ | 8055/9708 [00:39<00:07, 219.09batch/s]

Training batches on cuda:  83%|████████▎ | 8079/9708 [00:39<00:07, 222.96batch/s]

Training batches on cuda:  83%|████████▎ | 8103/9708 [00:39<00:07, 225.29batch/s]

Training batches on cuda:  84%|████████▎ | 8126/9708 [00:39<00:06, 226.51batch/s]

Training batches on cuda:  84%|████████▍ | 8149/9708 [00:39<00:06, 227.01batch/s]

Training batches on cuda:  84%|████████▍ | 8172/9708 [00:39<00:06, 226.23batch/s]

Training batches on cuda:  84%|████████▍ | 8195/9708 [00:39<00:06, 227.23batch/s]

Training batches on cuda:  85%|████████▍ | 8219/9708 [00:39<00:06, 228.44batch/s]

Training batches on cuda:  85%|████████▍ | 8243/9708 [00:39<00:06, 230.03batch/s]

Training batches on cuda:  85%|████████▌ | 8267/9708 [00:39<00:06, 230.23batch/s]

Training batches on cuda:  85%|████████▌ | 8291/9708 [00:40<00:06, 221.82batch/s]

Training batches on cuda:  86%|████████▌ | 8314/9708 [00:40<00:06, 218.07batch/s]

Training batches on cuda:  86%|████████▌ | 8336/9708 [00:40<00:06, 206.73batch/s]

Training batches on cuda:  86%|████████▌ | 8360/9708 [00:40<00:06, 213.32batch/s]

Training batches on cuda:  86%|████████▋ | 8383/9708 [00:40<00:06, 217.31batch/s]

Training batches on cuda:  87%|████████▋ | 8405/9708 [00:40<00:06, 215.47batch/s]

Training batches on cuda:  87%|████████▋ | 8428/9708 [00:40<00:05, 218.94batch/s]

Training batches on cuda:  87%|████████▋ | 8451/9708 [00:40<00:05, 221.66batch/s]

Training batches on cuda:  87%|████████▋ | 8475/9708 [00:40<00:05, 225.15batch/s]

Training batches on cuda:  88%|████████▊ | 8498/9708 [00:40<00:05, 225.92batch/s]

Training batches on cuda:  88%|████████▊ | 8522/9708 [00:41<00:05, 227.96batch/s]

Training batches on cuda:  88%|████████▊ | 8545/9708 [00:41<00:05, 228.44batch/s]

Training batches on cuda:  88%|████████▊ | 8568/9708 [00:41<00:05, 208.09batch/s]

Training batches on cuda:  88%|████████▊ | 8591/9708 [00:41<00:05, 213.50batch/s]

Training batches on cuda:  89%|████████▉ | 8616/9708 [00:41<00:04, 221.94batch/s]

Training batches on cuda:  89%|████████▉ | 8639/9708 [00:41<00:04, 224.28batch/s]

Training batches on cuda:  89%|████████▉ | 8664/9708 [00:41<00:04, 230.83batch/s]

Training batches on cuda:  89%|████████▉ | 8688/9708 [00:41<00:04, 228.36batch/s]

Training batches on cuda:  90%|████████▉ | 8712/9708 [00:41<00:04, 229.71batch/s]

Training batches on cuda:  90%|████████▉ | 8736/9708 [00:42<00:04, 227.72batch/s]

Training batches on cuda:  90%|█████████ | 8760/9708 [00:42<00:04, 230.62batch/s]

Training batches on cuda:  90%|█████████ | 8784/9708 [00:42<00:04, 227.40batch/s]

Training batches on cuda:  91%|█████████ | 8807/9708 [00:42<00:03, 226.38batch/s]

Training batches on cuda:  91%|█████████ | 8830/9708 [00:42<00:03, 226.16batch/s]

Training batches on cuda:  91%|█████████ | 8853/9708 [00:42<00:03, 222.38batch/s]

Training batches on cuda:  91%|█████████▏| 8876/9708 [00:42<00:03, 219.65batch/s]

Training batches on cuda:  92%|█████████▏| 8899/9708 [00:42<00:03, 221.43batch/s]

Training batches on cuda:  92%|█████████▏| 8922/9708 [00:42<00:03, 222.80batch/s]

Training batches on cuda:  92%|█████████▏| 8945/9708 [00:42<00:03, 223.95batch/s]

Training batches on cuda:  92%|█████████▏| 8968/9708 [00:43<00:03, 223.93batch/s]

Training batches on cuda:  93%|█████████▎| 8991/9708 [00:43<00:03, 224.17batch/s]

Training batches on cuda:  93%|█████████▎| 9014/9708 [00:43<00:03, 224.48batch/s]

Training batches on cuda:  93%|█████████▎| 9037/9708 [00:43<00:03, 218.46batch/s]

Training batches on cuda:  93%|█████████▎| 9060/9708 [00:43<00:02, 221.62batch/s]

Training batches on cuda:  94%|█████████▎| 9084/9708 [00:43<00:02, 224.79batch/s]

Training batches on cuda:  94%|█████████▍| 9107/9708 [00:43<00:02, 223.42batch/s]

Training batches on cuda:  94%|█████████▍| 9130/9708 [00:43<00:02, 218.13batch/s]

Training batches on cuda:  94%|█████████▍| 9152/9708 [00:43<00:02, 215.35batch/s]

Training batches on cuda:  94%|█████████▍| 9174/9708 [00:44<00:02, 211.38batch/s]

Training batches on cuda:  95%|█████████▍| 9196/9708 [00:44<00:02, 207.32batch/s]

Training batches on cuda:  95%|█████████▍| 9218/9708 [00:44<00:02, 209.13batch/s]

Training batches on cuda:  95%|█████████▌| 9240/9708 [00:44<00:02, 211.96batch/s]

Training batches on cuda:  95%|█████████▌| 9262/9708 [00:44<00:02, 214.27batch/s]

Training batches on cuda:  96%|█████████▌| 9284/9708 [00:44<00:02, 210.29batch/s]

Training batches on cuda:  96%|█████████▌| 9306/9708 [00:44<00:01, 210.59batch/s]

Training batches on cuda:  96%|█████████▌| 9328/9708 [00:44<00:01, 208.35batch/s]

Training batches on cuda:  96%|█████████▋| 9350/9708 [00:44<00:01, 209.62batch/s]

Training batches on cuda:  97%|█████████▋| 9371/9708 [00:44<00:01, 206.33batch/s]

Training batches on cuda:  97%|█████████▋| 9392/9708 [00:45<00:01, 206.17batch/s]

Training batches on cuda:  97%|█████████▋| 9414/9708 [00:45<00:01, 208.05batch/s]

Training batches on cuda:  97%|█████████▋| 9435/9708 [00:45<00:01, 201.69batch/s]

Training batches on cuda:  97%|█████████▋| 9456/9708 [00:45<00:01, 196.15batch/s]

Training batches on cuda:  98%|█████████▊| 9476/9708 [00:45<00:01, 191.21batch/s]

Training batches on cuda:  98%|█████████▊| 9497/9708 [00:45<00:01, 196.44batch/s]

Training batches on cuda:  98%|█████████▊| 9518/9708 [00:45<00:00, 200.14batch/s]

Training batches on cuda:  98%|█████████▊| 9540/9708 [00:45<00:00, 203.80batch/s]

Training batches on cuda:  98%|█████████▊| 9562/9708 [00:45<00:00, 206.11batch/s]

Training batches on cuda:  99%|█████████▊| 9583/9708 [00:46<00:00, 206.97batch/s]

Training batches on cuda:  99%|█████████▉| 9606/9708 [00:46<00:00, 212.27batch/s]

Training batches on cuda:  99%|█████████▉| 9629/9708 [00:46<00:00, 216.73batch/s]

Training batches on cuda:  99%|█████████▉| 9651/9708 [00:46<00:00, 185.31batch/s]

Training batches on cuda: 100%|█████████▉| 9675/9708 [00:46<00:00, 197.29batch/s]

Training batches on cuda: 100%|█████████▉| 9696/9708 [00:46<00:00, 200.70batch/s]

                                                                                 
Training epochs on cuda:   0%|          | 0/10 [00:46<?, ?epoch/s, loss=0.096, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:46<07:02, 46.92s/epoch, loss=0.096, prev_loss=nan]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 17/9708 [00:00<00:58, 165.65batch/s]

Training batches on cuda:   0%|          | 39/9708 [00:00<00:54, 178.75batch/s]

Training batches on cuda:   1%|          | 62/9708 [00:00<00:50, 189.72batch/s]

Training batches on cuda:   1%|          | 83/9708 [00:00<00:49, 194.50batch/s]

Training batches on cuda:   1%|          | 104/9708 [00:00<00:48, 197.13batch/s]

Training batches on cuda:   1%|▏         | 124/9708 [00:00<00:48, 197.49batch/s]

Training batches on cuda:   2%|▏         | 146/9708 [00:00<00:47, 202.00batch/s]

Training batches on cuda:   2%|▏         | 168/9708 [00:00<00:46, 206.59batch/s]

Training batches on cuda:   2%|▏         | 189/9708 [00:00<00:46, 206.33batch/s]

Training batches on cuda:   2%|▏         | 212/9708 [00:01<00:45, 210.34batch/s]

Training batches on cuda:   2%|▏         | 234/9708 [00:01<00:44, 211.10batch/s]

Training batches on cuda:   3%|▎         | 257/9708 [00:01<00:44, 214.46batch/s]

Training batches on cuda:   3%|▎         | 279/9708 [00:01<00:44, 213.89batch/s]

Training batches on cuda:   3%|▎         | 301/9708 [00:01<00:43, 214.85batch/s]

Training batches on cuda:   3%|▎         | 324/9708 [00:01<00:43, 217.50batch/s]

Training batches on cuda:   4%|▎         | 347/9708 [00:01<00:42, 220.03batch/s]

Training batches on cuda:   4%|▍         | 369/9708 [00:01<00:42, 219.34batch/s]

Training batches on cuda:   4%|▍         | 393/9708 [00:01<00:41, 222.59batch/s]

Training batches on cuda:   4%|▍         | 416/9708 [00:01<00:41, 222.16batch/s]

Training batches on cuda:   5%|▍         | 439/9708 [00:02<00:41, 224.17batch/s]

Training batches on cuda:   5%|▍         | 462/9708 [00:02<00:42, 217.26batch/s]

Training batches on cuda:   5%|▍         | 484/9708 [00:02<00:42, 214.65batch/s]

Training batches on cuda:   5%|▌         | 506/9708 [00:02<00:43, 211.28batch/s]

Training batches on cuda:   5%|▌         | 528/9708 [00:02<00:43, 209.31batch/s]

Training batches on cuda:   6%|▌         | 549/9708 [00:02<00:44, 206.81batch/s]

Training batches on cuda:   6%|▌         | 570/9708 [00:02<00:44, 205.79batch/s]

Training batches on cuda:   6%|▌         | 591/9708 [00:02<00:44, 203.16batch/s]

Training batches on cuda:   6%|▋         | 612/9708 [00:02<00:44, 204.24batch/s]

Training batches on cuda:   7%|▋         | 633/9708 [00:03<00:44, 203.72batch/s]

Training batches on cuda:   7%|▋         | 654/9708 [00:03<00:44, 205.25batch/s]

Training batches on cuda:   7%|▋         | 675/9708 [00:03<00:44, 204.43batch/s]

Training batches on cuda:   7%|▋         | 696/9708 [00:03<00:44, 204.12batch/s]

Training batches on cuda:   7%|▋         | 718/9708 [00:03<00:43, 206.13batch/s]

Training batches on cuda:   8%|▊         | 739/9708 [00:03<00:44, 203.36batch/s]

Training batches on cuda:   8%|▊         | 760/9708 [00:03<00:45, 197.52batch/s]

Training batches on cuda:   8%|▊         | 781/9708 [00:03<00:45, 198.31batch/s]

Training batches on cuda:   8%|▊         | 801/9708 [00:03<00:45, 197.03batch/s]

Training batches on cuda:   8%|▊         | 823/9708 [00:03<00:43, 202.37batch/s]

Training batches on cuda:   9%|▊         | 844/9708 [00:04<00:43, 202.03batch/s]

Training batches on cuda:   9%|▉         | 867/9708 [00:04<00:42, 207.50batch/s]

Training batches on cuda:   9%|▉         | 888/9708 [00:04<00:42, 207.80batch/s]

Training batches on cuda:   9%|▉         | 909/9708 [00:04<00:42, 207.38batch/s]

Training batches on cuda:  10%|▉         | 930/9708 [00:04<00:44, 197.19batch/s]

Training batches on cuda:  10%|▉         | 951/9708 [00:04<00:44, 198.07batch/s]

Training batches on cuda:  10%|█         | 972/9708 [00:04<00:43, 200.82batch/s]

Training batches on cuda:  10%|█         | 993/9708 [00:04<00:45, 193.01batch/s]

Training batches on cuda:  10%|█         | 1015/9708 [00:04<00:43, 198.46batch/s]

Training batches on cuda:  11%|█         | 1037/9708 [00:05<00:42, 202.34batch/s]

Training batches on cuda:  11%|█         | 1058/9708 [00:05<00:42, 203.92batch/s]

Training batches on cuda:  11%|█         | 1079/9708 [00:05<00:42, 204.99batch/s]

Training batches on cuda:  11%|█▏        | 1101/9708 [00:05<00:41, 207.20batch/s]

Training batches on cuda:  12%|█▏        | 1122/9708 [00:05<00:42, 201.77batch/s]

Training batches on cuda:  12%|█▏        | 1143/9708 [00:05<00:42, 200.29batch/s]

Training batches on cuda:  12%|█▏        | 1165/9708 [00:05<00:41, 205.33batch/s]

Training batches on cuda:  12%|█▏        | 1186/9708 [00:05<00:42, 199.79batch/s]

Training batches on cuda:  12%|█▏        | 1208/9708 [00:05<00:41, 204.36batch/s]

Training batches on cuda:  13%|█▎        | 1229/9708 [00:05<00:44, 190.31batch/s]

Training batches on cuda:  13%|█▎        | 1249/9708 [00:06<00:45, 187.67batch/s]

Training batches on cuda:  13%|█▎        | 1268/9708 [00:06<00:47, 179.24batch/s]

Training batches on cuda:  13%|█▎        | 1288/9708 [00:06<00:45, 184.26batch/s]

Training batches on cuda:  13%|█▎        | 1308/9708 [00:06<00:45, 186.62batch/s]

Training batches on cuda:  14%|█▎        | 1327/9708 [00:06<00:46, 180.92batch/s]

Training batches on cuda:  14%|█▍        | 1348/9708 [00:06<00:44, 186.55batch/s]

Training batches on cuda:  14%|█▍        | 1368/9708 [00:06<00:43, 190.24batch/s]

Training batches on cuda:  14%|█▍        | 1389/9708 [00:06<00:42, 195.31batch/s]

Training batches on cuda:  15%|█▍        | 1411/9708 [00:06<00:41, 200.68batch/s]

Training batches on cuda:  15%|█▍        | 1434/9708 [00:07<00:39, 206.91batch/s]

Training batches on cuda:  15%|█▍        | 1456/9708 [00:07<00:39, 208.29batch/s]

Training batches on cuda:  15%|█▌        | 1477/9708 [00:07<00:39, 207.39batch/s]

Training batches on cuda:  15%|█▌        | 1499/9708 [00:07<00:39, 207.07batch/s]

Training batches on cuda:  16%|█▌        | 1520/9708 [00:07<00:39, 206.45batch/s]

Training batches on cuda:  16%|█▌        | 1542/9708 [00:07<00:39, 208.29batch/s]

Training batches on cuda:  16%|█▌        | 1565/9708 [00:07<00:38, 212.48batch/s]

Training batches on cuda:  16%|█▋        | 1587/9708 [00:07<00:38, 213.66batch/s]

Training batches on cuda:  17%|█▋        | 1610/9708 [00:07<00:37, 217.49batch/s]

Training batches on cuda:  17%|█▋        | 1633/9708 [00:07<00:36, 220.73batch/s]

Training batches on cuda:  17%|█▋        | 1656/9708 [00:08<00:36, 219.87batch/s]

Training batches on cuda:  17%|█▋        | 1679/9708 [00:08<00:36, 220.68batch/s]

Training batches on cuda:  18%|█▊        | 1702/9708 [00:08<00:36, 220.48batch/s]

Training batches on cuda:  18%|█▊        | 1726/9708 [00:08<00:35, 223.55batch/s]

Training batches on cuda:  18%|█▊        | 1749/9708 [00:08<00:35, 221.30batch/s]

Training batches on cuda:  18%|█▊        | 1772/9708 [00:08<00:36, 219.10batch/s]

Training batches on cuda:  18%|█▊        | 1795/9708 [00:08<00:35, 221.54batch/s]

Training batches on cuda:  19%|█▊        | 1818/9708 [00:08<00:35, 222.56batch/s]

Training batches on cuda:  19%|█▉        | 1841/9708 [00:08<00:36, 218.01batch/s]

Training batches on cuda:  19%|█▉        | 1863/9708 [00:09<00:36, 212.51batch/s]

Training batches on cuda:  19%|█▉        | 1885/9708 [00:09<00:37, 208.01batch/s]

Training batches on cuda:  20%|█▉        | 1907/9708 [00:09<00:36, 211.35batch/s]

Training batches on cuda:  20%|█▉        | 1929/9708 [00:09<00:38, 203.47batch/s]

Training batches on cuda:  20%|██        | 1950/9708 [00:09<00:41, 187.73batch/s]

Training batches on cuda:  20%|██        | 1971/9708 [00:09<00:39, 193.59batch/s]

Training batches on cuda:  21%|██        | 1992/9708 [00:09<00:39, 197.45batch/s]

Training batches on cuda:  21%|██        | 2013/9708 [00:09<00:38, 200.84batch/s]

Training batches on cuda:  21%|██        | 2036/9708 [00:09<00:37, 207.17batch/s]

Training batches on cuda:  21%|██        | 2057/9708 [00:09<00:37, 203.08batch/s]

Training batches on cuda:  21%|██▏       | 2079/9708 [00:10<00:36, 207.03batch/s]

Training batches on cuda:  22%|██▏       | 2102/9708 [00:10<00:36, 211.18batch/s]

Training batches on cuda:  22%|██▏       | 2124/9708 [00:10<00:35, 213.14batch/s]

Training batches on cuda:  22%|██▏       | 2146/9708 [00:10<00:35, 214.56batch/s]

Training batches on cuda:  22%|██▏       | 2168/9708 [00:10<00:35, 213.93batch/s]

Training batches on cuda:  23%|██▎       | 2190/9708 [00:10<00:35, 212.94batch/s]

Training batches on cuda:  23%|██▎       | 2212/9708 [00:10<00:35, 211.00batch/s]

Training batches on cuda:  23%|██▎       | 2234/9708 [00:10<00:35, 209.25batch/s]

Training batches on cuda:  23%|██▎       | 2256/9708 [00:10<00:35, 210.19batch/s]

Training batches on cuda:  23%|██▎       | 2278/9708 [00:11<00:35, 208.86batch/s]

Training batches on cuda:  24%|██▎       | 2301/9708 [00:11<00:34, 212.31batch/s]

Training batches on cuda:  24%|██▍       | 2324/9708 [00:11<00:34, 216.83batch/s]

Training batches on cuda:  24%|██▍       | 2346/9708 [00:11<00:33, 217.76batch/s]

Training batches on cuda:  24%|██▍       | 2369/9708 [00:11<00:33, 220.35batch/s]

Training batches on cuda:  25%|██▍       | 2392/9708 [00:11<00:34, 213.69batch/s]

Training batches on cuda:  25%|██▍       | 2414/9708 [00:11<00:36, 200.58batch/s]

Training batches on cuda:  25%|██▌       | 2435/9708 [00:11<00:37, 195.40batch/s]

Training batches on cuda:  25%|██▌       | 2456/9708 [00:11<00:36, 197.53batch/s]

Training batches on cuda:  26%|██▌       | 2479/9708 [00:11<00:35, 206.19batch/s]

Training batches on cuda:  26%|██▌       | 2502/9708 [00:12<00:33, 212.34batch/s]

Training batches on cuda:  26%|██▌       | 2524/9708 [00:12<00:33, 214.01batch/s]

Training batches on cuda:  26%|██▌       | 2546/9708 [00:12<00:33, 214.29batch/s]

Training batches on cuda:  26%|██▋       | 2568/9708 [00:12<00:33, 213.56batch/s]

Training batches on cuda:  27%|██▋       | 2590/9708 [00:12<00:34, 208.51batch/s]

Training batches on cuda:  27%|██▋       | 2611/9708 [00:12<00:34, 206.64batch/s]

Training batches on cuda:  27%|██▋       | 2634/9708 [00:12<00:33, 212.39batch/s]

Training batches on cuda:  27%|██▋       | 2657/9708 [00:12<00:32, 215.25batch/s]

Training batches on cuda:  28%|██▊       | 2680/9708 [00:12<00:32, 216.83batch/s]

Training batches on cuda:  28%|██▊       | 2704/9708 [00:13<00:31, 221.12batch/s]

Training batches on cuda:  28%|██▊       | 2727/9708 [00:13<00:31, 221.39batch/s]

Training batches on cuda:  28%|██▊       | 2751/9708 [00:13<00:31, 223.83batch/s]

Training batches on cuda:  29%|██▊       | 2774/9708 [00:13<00:30, 225.53batch/s]

Training batches on cuda:  29%|██▉       | 2797/9708 [00:13<00:30, 226.70batch/s]

Training batches on cuda:  29%|██▉       | 2820/9708 [00:13<00:30, 226.28batch/s]

Training batches on cuda:  29%|██▉       | 2844/9708 [00:13<00:30, 227.25batch/s]

Training batches on cuda:  30%|██▉       | 2867/9708 [00:13<00:30, 226.99batch/s]

Training batches on cuda:  30%|██▉       | 2890/9708 [00:13<00:30, 224.65batch/s]

Training batches on cuda:  30%|███       | 2913/9708 [00:13<00:30, 224.28batch/s]

Training batches on cuda:  30%|███       | 2936/9708 [00:14<00:30, 225.54batch/s]

Training batches on cuda:  30%|███       | 2959/9708 [00:14<00:30, 224.01batch/s]

Training batches on cuda:  31%|███       | 2982/9708 [00:14<00:29, 224.42batch/s]

Training batches on cuda:  31%|███       | 3005/9708 [00:14<00:29, 224.23batch/s]

Training batches on cuda:  31%|███       | 3028/9708 [00:14<00:32, 205.12batch/s]

Training batches on cuda:  31%|███▏      | 3049/9708 [00:14<00:32, 205.26batch/s]

Training batches on cuda:  32%|███▏      | 3071/9708 [00:14<00:32, 207.22batch/s]

Training batches on cuda:  32%|███▏      | 3092/9708 [00:14<00:33, 198.32batch/s]

Training batches on cuda:  32%|███▏      | 3113/9708 [00:14<00:34, 190.27batch/s]

Training batches on cuda:  32%|███▏      | 3133/9708 [00:15<00:34, 188.28batch/s]

Training batches on cuda:  32%|███▏      | 3153/9708 [00:15<00:34, 189.25batch/s]

Training batches on cuda:  33%|███▎      | 3173/9708 [00:15<00:34, 190.89batch/s]

Training batches on cuda:  33%|███▎      | 3193/9708 [00:15<00:34, 187.86batch/s]

Training batches on cuda:  33%|███▎      | 3213/9708 [00:15<00:34, 189.66batch/s]

Training batches on cuda:  33%|███▎      | 3233/9708 [00:15<00:34, 190.36batch/s]

Training batches on cuda:  34%|███▎      | 3253/9708 [00:15<00:34, 187.13batch/s]

Training batches on cuda:  34%|███▎      | 3276/9708 [00:15<00:32, 197.18batch/s]

Training batches on cuda:  34%|███▍      | 3298/9708 [00:15<00:31, 203.20batch/s]

Training batches on cuda:  34%|███▍      | 3321/9708 [00:15<00:30, 209.80batch/s]

Training batches on cuda:  34%|███▍      | 3346/9708 [00:16<00:29, 218.68batch/s]

Training batches on cuda:  35%|███▍      | 3371/9708 [00:16<00:28, 225.65batch/s]

Training batches on cuda:  35%|███▍      | 3394/9708 [00:16<00:28, 225.16batch/s]

Training batches on cuda:  35%|███▌      | 3418/9708 [00:16<00:27, 227.48batch/s]

Training batches on cuda:  35%|███▌      | 3441/9708 [00:16<00:27, 226.15batch/s]

Training batches on cuda:  36%|███▌      | 3464/9708 [00:16<00:27, 226.08batch/s]

Training batches on cuda:  36%|███▌      | 3487/9708 [00:16<00:28, 217.46batch/s]

Training batches on cuda:  36%|███▌      | 3509/9708 [00:16<00:28, 217.28batch/s]

Training batches on cuda:  36%|███▋      | 3531/9708 [00:16<00:28, 215.67batch/s]

Training batches on cuda:  37%|███▋      | 3554/9708 [00:17<00:28, 218.57batch/s]

Training batches on cuda:  37%|███▋      | 3577/9708 [00:17<00:27, 220.55batch/s]

Training batches on cuda:  37%|███▋      | 3600/9708 [00:17<00:27, 222.33batch/s]

Training batches on cuda:  37%|███▋      | 3623/9708 [00:17<00:27, 222.19batch/s]

Training batches on cuda:  38%|███▊      | 3646/9708 [00:17<00:27, 220.73batch/s]

Training batches on cuda:  38%|███▊      | 3669/9708 [00:17<00:27, 218.37batch/s]

Training batches on cuda:  38%|███▊      | 3691/9708 [00:17<00:28, 213.12batch/s]

Training batches on cuda:  38%|███▊      | 3713/9708 [00:17<00:28, 212.54batch/s]

Training batches on cuda:  38%|███▊      | 3735/9708 [00:17<00:29, 202.24batch/s]

Training batches on cuda:  39%|███▊      | 3756/9708 [00:17<00:30, 196.74batch/s]

Training batches on cuda:  39%|███▉      | 3776/9708 [00:18<00:30, 192.74batch/s]

Training batches on cuda:  39%|███▉      | 3797/9708 [00:18<00:30, 194.90batch/s]

Training batches on cuda:  39%|███▉      | 3818/9708 [00:18<00:29, 196.63batch/s]

Training batches on cuda:  40%|███▉      | 3839/9708 [00:18<00:29, 198.75batch/s]

Training batches on cuda:  40%|███▉      | 3860/9708 [00:18<00:29, 199.60batch/s]

Training batches on cuda:  40%|███▉      | 3880/9708 [00:18<00:29, 197.04batch/s]

Training batches on cuda:  40%|████      | 3900/9708 [00:18<00:29, 195.66batch/s]

Training batches on cuda:  40%|████      | 3920/9708 [00:18<00:29, 196.56batch/s]

Training batches on cuda:  41%|████      | 3941/9708 [00:18<00:29, 198.65batch/s]

Training batches on cuda:  41%|████      | 3962/9708 [00:19<00:28, 199.88batch/s]

Training batches on cuda:  41%|████      | 3983/9708 [00:19<00:28, 201.18batch/s]

Training batches on cuda:  41%|████      | 4004/9708 [00:19<00:28, 202.45batch/s]

Training batches on cuda:  41%|████▏     | 4025/9708 [00:19<00:28, 201.50batch/s]

Training batches on cuda:  42%|████▏     | 4046/9708 [00:19<00:31, 182.14batch/s]

Training batches on cuda:  42%|████▏     | 4066/9708 [00:19<00:30, 185.22batch/s]

Training batches on cuda:  42%|████▏     | 4087/9708 [00:19<00:29, 191.16batch/s]

Training batches on cuda:  42%|████▏     | 4107/9708 [00:19<00:29, 191.73batch/s]

Training batches on cuda:  43%|████▎     | 4127/9708 [00:19<00:30, 182.36batch/s]

Training batches on cuda:  43%|████▎     | 4146/9708 [00:20<00:30, 182.77batch/s]

Training batches on cuda:  43%|████▎     | 4167/9708 [00:20<00:29, 188.51batch/s]

Training batches on cuda:  43%|████▎     | 4188/9708 [00:20<00:28, 193.03batch/s]

Training batches on cuda:  43%|████▎     | 4209/9708 [00:20<00:28, 195.22batch/s]

Training batches on cuda:  44%|████▎     | 4230/9708 [00:20<00:27, 197.48batch/s]

Training batches on cuda:  44%|████▍     | 4251/9708 [00:20<00:27, 199.58batch/s]

Training batches on cuda:  44%|████▍     | 4272/9708 [00:20<00:27, 197.87batch/s]

Training batches on cuda:  44%|████▍     | 4293/9708 [00:20<00:27, 199.23batch/s]

Training batches on cuda:  44%|████▍     | 4313/9708 [00:20<00:27, 198.08batch/s]

Training batches on cuda:  45%|████▍     | 4334/9708 [00:20<00:26, 199.65batch/s]

Training batches on cuda:  45%|████▍     | 4355/9708 [00:21<00:26, 201.89batch/s]

Training batches on cuda:  45%|████▌     | 4376/9708 [00:21<00:26, 202.70batch/s]

Training batches on cuda:  45%|████▌     | 4397/9708 [00:21<00:26, 200.65batch/s]

Training batches on cuda:  46%|████▌     | 4418/9708 [00:21<00:26, 201.81batch/s]

Training batches on cuda:  46%|████▌     | 4439/9708 [00:21<00:26, 200.21batch/s]

Training batches on cuda:  46%|████▌     | 4460/9708 [00:21<00:26, 200.06batch/s]

Training batches on cuda:  46%|████▌     | 4481/9708 [00:21<00:26, 200.13batch/s]

Training batches on cuda:  46%|████▋     | 4502/9708 [00:21<00:26, 199.16batch/s]

Training batches on cuda:  47%|████▋     | 4522/9708 [00:21<00:26, 195.71batch/s]

Training batches on cuda:  47%|████▋     | 4542/9708 [00:21<00:26, 194.35batch/s]

Training batches on cuda:  47%|████▋     | 4562/9708 [00:22<00:26, 193.36batch/s]

Training batches on cuda:  47%|████▋     | 4582/9708 [00:22<00:26, 192.87batch/s]

Training batches on cuda:  47%|████▋     | 4603/9708 [00:22<00:26, 195.95batch/s]

Training batches on cuda:  48%|████▊     | 4624/9708 [00:22<00:25, 198.91batch/s]

Training batches on cuda:  48%|████▊     | 4645/9708 [00:22<00:25, 201.42batch/s]

Training batches on cuda:  48%|████▊     | 4666/9708 [00:22<00:25, 195.51batch/s]

Training batches on cuda:  48%|████▊     | 4688/9708 [00:22<00:25, 200.74batch/s]

Training batches on cuda:  49%|████▊     | 4709/9708 [00:22<00:24, 201.53batch/s]

Training batches on cuda:  49%|████▊     | 4730/9708 [00:22<00:26, 190.57batch/s]

Training batches on cuda:  49%|████▉     | 4752/9708 [00:23<00:25, 197.45batch/s]

Training batches on cuda:  49%|████▉     | 4773/9708 [00:23<00:24, 197.94batch/s]

Training batches on cuda:  49%|████▉     | 4793/9708 [00:23<00:25, 190.28batch/s]

Training batches on cuda:  50%|████▉     | 4813/9708 [00:23<00:25, 191.43batch/s]

Training batches on cuda:  50%|████▉     | 4833/9708 [00:23<00:25, 190.91batch/s]

Training batches on cuda:  50%|████▉     | 4853/9708 [00:23<00:25, 190.13batch/s]

Training batches on cuda:  50%|█████     | 4874/9708 [00:23<00:24, 194.81batch/s]

Training batches on cuda:  50%|█████     | 4894/9708 [00:23<00:24, 193.98batch/s]

Training batches on cuda:  51%|█████     | 4914/9708 [00:23<00:25, 191.32batch/s]

Training batches on cuda:  51%|█████     | 4934/9708 [00:24<00:25, 187.82batch/s]

Training batches on cuda:  51%|█████     | 4953/9708 [00:24<00:26, 181.31batch/s]

Training batches on cuda:  51%|█████     | 4972/9708 [00:24<00:26, 177.18batch/s]

Training batches on cuda:  51%|█████▏    | 4993/9708 [00:24<00:25, 183.89batch/s]

Training batches on cuda:  52%|█████▏    | 5014/9708 [00:24<00:24, 190.77batch/s]

Training batches on cuda:  52%|█████▏    | 5037/9708 [00:24<00:23, 199.26batch/s]

Training batches on cuda:  52%|█████▏    | 5058/9708 [00:24<00:23, 196.30batch/s]

Training batches on cuda:  52%|█████▏    | 5078/9708 [00:24<00:24, 186.90batch/s]

Training batches on cuda:  53%|█████▎    | 5097/9708 [00:24<00:25, 184.20batch/s]

Training batches on cuda:  53%|█████▎    | 5117/9708 [00:24<00:24, 185.61batch/s]

Training batches on cuda:  53%|█████▎    | 5136/9708 [00:25<00:24, 183.21batch/s]

Training batches on cuda:  53%|█████▎    | 5155/9708 [00:25<00:24, 184.85batch/s]

Training batches on cuda:  53%|█████▎    | 5176/9708 [00:25<00:23, 189.24batch/s]

Training batches on cuda:  54%|█████▎    | 5195/9708 [00:25<00:24, 188.01batch/s]

Training batches on cuda:  54%|█████▎    | 5217/9708 [00:25<00:23, 194.49batch/s]

Training batches on cuda:  54%|█████▍    | 5238/9708 [00:25<00:22, 196.01batch/s]

Training batches on cuda:  54%|█████▍    | 5259/9708 [00:25<00:22, 199.65batch/s]

Training batches on cuda:  54%|█████▍    | 5281/9708 [00:25<00:21, 203.23batch/s]

Training batches on cuda:  55%|█████▍    | 5302/9708 [00:25<00:21, 204.76batch/s]

Training batches on cuda:  55%|█████▍    | 5326/9708 [00:26<00:20, 212.15batch/s]

Training batches on cuda:  55%|█████▌    | 5348/9708 [00:26<00:20, 214.41batch/s]

Training batches on cuda:  55%|█████▌    | 5372/9708 [00:26<00:19, 220.64batch/s]

Training batches on cuda:  56%|█████▌    | 5395/9708 [00:26<00:19, 219.47batch/s]

Training batches on cuda:  56%|█████▌    | 5418/9708 [00:26<00:19, 218.54batch/s]

Training batches on cuda:  56%|█████▌    | 5440/9708 [00:26<00:19, 218.97batch/s]

Training batches on cuda:  56%|█████▋    | 5462/9708 [00:26<00:19, 217.98batch/s]

Training batches on cuda:  56%|█████▋    | 5485/9708 [00:26<00:19, 218.59batch/s]

Training batches on cuda:  57%|█████▋    | 5509/9708 [00:26<00:18, 224.23batch/s]

Training batches on cuda:  57%|█████▋    | 5532/9708 [00:26<00:18, 225.20batch/s]

Training batches on cuda:  57%|█████▋    | 5557/9708 [00:27<00:17, 231.39batch/s]

Training batches on cuda:  58%|█████▊    | 5584/9708 [00:27<00:17, 240.57batch/s]

Training batches on cuda:  58%|█████▊    | 5610/9708 [00:27<00:16, 244.31batch/s]

Training batches on cuda:  58%|█████▊    | 5635/9708 [00:27<00:16, 242.90batch/s]

Training batches on cuda:  58%|█████▊    | 5660/9708 [00:27<00:16, 244.36batch/s]

Training batches on cuda:  59%|█████▊    | 5685/9708 [00:27<00:16, 241.76batch/s]

Training batches on cuda:  59%|█████▉    | 5710/9708 [00:27<00:16, 237.49batch/s]

Training batches on cuda:  59%|█████▉    | 5734/9708 [00:27<00:16, 236.13batch/s]

Training batches on cuda:  59%|█████▉    | 5758/9708 [00:27<00:17, 227.06batch/s]

Training batches on cuda:  60%|█████▉    | 5781/9708 [00:28<00:17, 223.29batch/s]

Training batches on cuda:  60%|█████▉    | 5804/9708 [00:28<00:17, 221.58batch/s]

Training batches on cuda:  60%|██████    | 5827/9708 [00:28<00:17, 221.01batch/s]

Training batches on cuda:  60%|██████    | 5850/9708 [00:28<00:17, 216.86batch/s]

Training batches on cuda:  60%|██████    | 5872/9708 [00:28<00:17, 217.51batch/s]

Training batches on cuda:  61%|██████    | 5897/9708 [00:28<00:17, 223.67batch/s]

Training batches on cuda:  61%|██████    | 5920/9708 [00:28<00:16, 224.44batch/s]

Training batches on cuda:  61%|██████    | 5943/9708 [00:28<00:17, 215.14batch/s]

Training batches on cuda:  61%|██████▏   | 5965/9708 [00:28<00:17, 213.16batch/s]

Training batches on cuda:  62%|██████▏   | 5988/9708 [00:28<00:17, 217.29batch/s]

Training batches on cuda:  62%|██████▏   | 6012/9708 [00:29<00:16, 221.88batch/s]

Training batches on cuda:  62%|██████▏   | 6035/9708 [00:29<00:16, 218.62batch/s]

Training batches on cuda:  62%|██████▏   | 6057/9708 [00:29<00:17, 208.55batch/s]

Training batches on cuda:  63%|██████▎   | 6082/9708 [00:29<00:16, 217.24batch/s]

Training batches on cuda:  63%|██████▎   | 6104/9708 [00:29<00:16, 216.17batch/s]

Training batches on cuda:  63%|██████▎   | 6126/9708 [00:29<00:16, 210.94batch/s]

Training batches on cuda:  63%|██████▎   | 6150/9708 [00:29<00:16, 217.60batch/s]

Training batches on cuda:  64%|██████▎   | 6173/9708 [00:29<00:16, 218.79batch/s]

Training batches on cuda:  64%|██████▍   | 6196/9708 [00:29<00:15, 220.70batch/s]

Training batches on cuda:  64%|██████▍   | 6220/9708 [00:30<00:15, 224.64batch/s]

Training batches on cuda:  64%|██████▍   | 6243/9708 [00:30<00:15, 222.98batch/s]

Training batches on cuda:  65%|██████▍   | 6266/9708 [00:30<00:15, 216.92batch/s]

Training batches on cuda:  65%|██████▍   | 6288/9708 [00:30<00:16, 212.96batch/s]

Training batches on cuda:  65%|██████▍   | 6310/9708 [00:30<00:16, 210.12batch/s]

Training batches on cuda:  65%|██████▌   | 6332/9708 [00:30<00:16, 208.49batch/s]

Training batches on cuda:  65%|██████▌   | 6354/9708 [00:30<00:16, 209.17batch/s]

Training batches on cuda:  66%|██████▌   | 6376/9708 [00:30<00:15, 210.56batch/s]

Training batches on cuda:  66%|██████▌   | 6398/9708 [00:30<00:15, 210.74batch/s]

Training batches on cuda:  66%|██████▌   | 6420/9708 [00:30<00:15, 211.62batch/s]

Training batches on cuda:  66%|██████▋   | 6443/9708 [00:31<00:15, 216.12batch/s]

Training batches on cuda:  67%|██████▋   | 6467/9708 [00:31<00:14, 221.28batch/s]

Training batches on cuda:  67%|██████▋   | 6491/9708 [00:31<00:14, 225.93batch/s]

Training batches on cuda:  67%|██████▋   | 6514/9708 [00:31<00:14, 219.10batch/s]

Training batches on cuda:  67%|██████▋   | 6536/9708 [00:31<00:15, 202.89batch/s]

Training batches on cuda:  68%|██████▊   | 6557/9708 [00:31<00:15, 199.78batch/s]

Training batches on cuda:  68%|██████▊   | 6580/9708 [00:31<00:15, 207.31batch/s]

Training batches on cuda:  68%|██████▊   | 6604/9708 [00:31<00:14, 214.34batch/s]

Training batches on cuda:  68%|██████▊   | 6628/9708 [00:31<00:13, 220.02batch/s]

Training batches on cuda:  69%|██████▊   | 6651/9708 [00:32<00:14, 217.57batch/s]

Training batches on cuda:  69%|██████▊   | 6673/9708 [00:32<00:14, 214.56batch/s]

Training batches on cuda:  69%|██████▉   | 6695/9708 [00:32<00:14, 214.47batch/s]

Training batches on cuda:  69%|██████▉   | 6717/9708 [00:32<00:14, 207.42batch/s]

Training batches on cuda:  69%|██████▉   | 6738/9708 [00:32<00:14, 204.51batch/s]

Training batches on cuda:  70%|██████▉   | 6761/9708 [00:32<00:14, 210.22batch/s]

Training batches on cuda:  70%|██████▉   | 6783/9708 [00:32<00:13, 209.21batch/s]

Training batches on cuda:  70%|███████   | 6804/9708 [00:32<00:14, 206.67batch/s]

Training batches on cuda:  70%|███████   | 6829/9708 [00:32<00:13, 217.06batch/s]

Training batches on cuda:  71%|███████   | 6854/9708 [00:32<00:12, 224.82batch/s]

Training batches on cuda:  71%|███████   | 6878/9708 [00:33<00:12, 228.20batch/s]

Training batches on cuda:  71%|███████   | 6904/9708 [00:33<00:11, 234.23batch/s]

Training batches on cuda:  71%|███████▏  | 6930/9708 [00:33<00:11, 239.04batch/s]

Training batches on cuda:  72%|███████▏  | 6955/9708 [00:33<00:11, 238.39batch/s]

Training batches on cuda:  72%|███████▏  | 6980/9708 [00:33<00:11, 241.63batch/s]

Training batches on cuda:  72%|███████▏  | 7005/9708 [00:33<00:11, 240.74batch/s]

Training batches on cuda:  72%|███████▏  | 7031/9708 [00:33<00:10, 245.28batch/s]

Training batches on cuda:  73%|███████▎  | 7056/9708 [00:33<00:11, 240.02batch/s]

Training batches on cuda:  73%|███████▎  | 7081/9708 [00:33<00:10, 241.45batch/s]

Training batches on cuda:  73%|███████▎  | 7106/9708 [00:34<00:10, 238.23batch/s]

Training batches on cuda:  73%|███████▎  | 7130/9708 [00:34<00:11, 220.58batch/s]

Training batches on cuda:  74%|███████▎  | 7153/9708 [00:34<00:11, 217.72batch/s]

Training batches on cuda:  74%|███████▍  | 7176/9708 [00:34<00:11, 221.06batch/s]

Training batches on cuda:  74%|███████▍  | 7199/9708 [00:34<00:11, 220.61batch/s]

Training batches on cuda:  74%|███████▍  | 7222/9708 [00:34<00:11, 218.27batch/s]

Training batches on cuda:  75%|███████▍  | 7249/9708 [00:34<00:10, 229.33batch/s]

Training batches on cuda:  75%|███████▍  | 7273/9708 [00:34<00:10, 223.59batch/s]

Training batches on cuda:  75%|███████▌  | 7297/9708 [00:34<00:10, 226.56batch/s]

Training batches on cuda:  75%|███████▌  | 7320/9708 [00:35<00:11, 216.71batch/s]

Training batches on cuda:  76%|███████▌  | 7342/9708 [00:35<00:11, 209.39batch/s]

Training batches on cuda:  76%|███████▌  | 7364/9708 [00:35<00:11, 199.51batch/s]

Training batches on cuda:  76%|███████▌  | 7385/9708 [00:35<00:11, 200.47batch/s]

Training batches on cuda:  76%|███████▋  | 7406/9708 [00:35<00:12, 190.50batch/s]

Training batches on cuda:  77%|███████▋  | 7429/9708 [00:35<00:11, 200.47batch/s]

Training batches on cuda:  77%|███████▋  | 7450/9708 [00:35<00:11, 201.91batch/s]

Training batches on cuda:  77%|███████▋  | 7471/9708 [00:35<00:11, 201.61batch/s]

Training batches on cuda:  77%|███████▋  | 7494/9708 [00:35<00:10, 207.95batch/s]

Training batches on cuda:  77%|███████▋  | 7517/9708 [00:35<00:10, 212.65batch/s]

Training batches on cuda:  78%|███████▊  | 7540/9708 [00:36<00:10, 216.51batch/s]

Training batches on cuda:  78%|███████▊  | 7564/9708 [00:36<00:09, 220.40batch/s]

Training batches on cuda:  78%|███████▊  | 7587/9708 [00:36<00:10, 206.33batch/s]

Training batches on cuda:  78%|███████▊  | 7610/9708 [00:36<00:09, 209.89batch/s]

Training batches on cuda:  79%|███████▊  | 7635/9708 [00:36<00:09, 219.72batch/s]

Training batches on cuda:  79%|███████▉  | 7660/9708 [00:36<00:08, 227.91batch/s]

Training batches on cuda:  79%|███████▉  | 7684/9708 [00:36<00:08, 230.37batch/s]

Training batches on cuda:  79%|███████▉  | 7710/9708 [00:36<00:08, 236.00batch/s]

Training batches on cuda:  80%|███████▉  | 7734/9708 [00:36<00:08, 229.41batch/s]

Training batches on cuda:  80%|███████▉  | 7758/9708 [00:37<00:08, 219.80batch/s]

Training batches on cuda:  80%|████████  | 7781/9708 [00:37<00:09, 213.33batch/s]

Training batches on cuda:  80%|████████  | 7803/9708 [00:37<00:09, 206.85batch/s]

Training batches on cuda:  81%|████████  | 7826/9708 [00:37<00:08, 211.90batch/s]

Training batches on cuda:  81%|████████  | 7848/9708 [00:37<00:08, 209.48batch/s]

Training batches on cuda:  81%|████████  | 7870/9708 [00:37<00:08, 204.54batch/s]

Training batches on cuda:  81%|████████▏ | 7892/9708 [00:37<00:08, 208.43batch/s]

Training batches on cuda:  82%|████████▏ | 7914/9708 [00:37<00:08, 210.74batch/s]

Training batches on cuda:  82%|████████▏ | 7936/9708 [00:37<00:08, 210.36batch/s]

Training batches on cuda:  82%|████████▏ | 7958/9708 [00:38<00:08, 212.08batch/s]

Training batches on cuda:  82%|████████▏ | 7982/9708 [00:38<00:07, 217.27batch/s]

Training batches on cuda:  82%|████████▏ | 8005/9708 [00:38<00:07, 220.92batch/s]

Training batches on cuda:  83%|████████▎ | 8028/9708 [00:38<00:07, 221.90batch/s]

Training batches on cuda:  83%|████████▎ | 8051/9708 [00:38<00:07, 213.76batch/s]

Training batches on cuda:  83%|████████▎ | 8073/9708 [00:38<00:07, 211.16batch/s]

Training batches on cuda:  83%|████████▎ | 8095/9708 [00:38<00:07, 207.26batch/s]

Training batches on cuda:  84%|████████▎ | 8116/9708 [00:38<00:07, 201.98batch/s]

Training batches on cuda:  84%|████████▍ | 8137/9708 [00:38<00:07, 197.78batch/s]

Training batches on cuda:  84%|████████▍ | 8159/9708 [00:38<00:07, 202.59batch/s]

Training batches on cuda:  84%|████████▍ | 8181/9708 [00:39<00:07, 204.96batch/s]

Training batches on cuda:  84%|████████▍ | 8202/9708 [00:39<00:07, 206.39batch/s]

Training batches on cuda:  85%|████████▍ | 8224/9708 [00:39<00:07, 207.83batch/s]

Training batches on cuda:  85%|████████▍ | 8246/9708 [00:39<00:06, 211.22batch/s]

Training batches on cuda:  85%|████████▌ | 8268/9708 [00:39<00:07, 198.51batch/s]

Training batches on cuda:  85%|████████▌ | 8291/9708 [00:39<00:06, 205.78batch/s]

Training batches on cuda:  86%|████████▌ | 8312/9708 [00:39<00:06, 206.35batch/s]

Training batches on cuda:  86%|████████▌ | 8335/9708 [00:39<00:06, 211.08batch/s]

Training batches on cuda:  86%|████████▌ | 8358/9708 [00:39<00:06, 213.73batch/s]

Training batches on cuda:  86%|████████▋ | 8380/9708 [00:40<00:06, 214.95batch/s]

Training batches on cuda:  87%|████████▋ | 8403/9708 [00:40<00:05, 218.45batch/s]

Training batches on cuda:  87%|████████▋ | 8425/9708 [00:40<00:05, 216.82batch/s]

Training batches on cuda:  87%|████████▋ | 8447/9708 [00:40<00:05, 216.02batch/s]

Training batches on cuda:  87%|████████▋ | 8470/9708 [00:40<00:05, 219.31batch/s]

Training batches on cuda:  87%|████████▋ | 8492/9708 [00:40<00:05, 214.10batch/s]

Training batches on cuda:  88%|████████▊ | 8514/9708 [00:40<00:05, 214.41batch/s]

Training batches on cuda:  88%|████████▊ | 8536/9708 [00:40<00:05, 211.23batch/s]

Training batches on cuda:  88%|████████▊ | 8558/9708 [00:40<00:05, 208.93batch/s]

Training batches on cuda:  88%|████████▊ | 8579/9708 [00:40<00:05, 207.58batch/s]

Training batches on cuda:  89%|████████▊ | 8601/9708 [00:41<00:05, 209.71batch/s]

Training batches on cuda:  89%|████████▉ | 8622/9708 [00:41<00:05, 209.22batch/s]

Training batches on cuda:  89%|████████▉ | 8644/9708 [00:41<00:05, 211.39batch/s]

Training batches on cuda:  89%|████████▉ | 8666/9708 [00:41<00:05, 208.01batch/s]

Training batches on cuda:  89%|████████▉ | 8687/9708 [00:41<00:04, 206.48batch/s]

Training batches on cuda:  90%|████████▉ | 8712/9708 [00:41<00:04, 215.96batch/s]

Training batches on cuda:  90%|████████▉ | 8737/9708 [00:41<00:04, 223.96batch/s]

Training batches on cuda:  90%|█████████ | 8763/9708 [00:41<00:04, 233.13batch/s]

Training batches on cuda:  91%|█████████ | 8787/9708 [00:41<00:04, 225.33batch/s]

Training batches on cuda:  91%|█████████ | 8810/9708 [00:42<00:04, 224.46batch/s]

Training batches on cuda:  91%|█████████ | 8834/9708 [00:42<00:03, 226.69batch/s]

Training batches on cuda:  91%|█████████ | 8857/9708 [00:42<00:03, 226.41batch/s]

Training batches on cuda:  91%|█████████▏| 8880/9708 [00:42<00:03, 222.74batch/s]

Training batches on cuda:  92%|█████████▏| 8904/9708 [00:42<00:03, 226.97batch/s]

Training batches on cuda:  92%|█████████▏| 8927/9708 [00:42<00:03, 211.79batch/s]

Training batches on cuda:  92%|█████████▏| 8949/9708 [00:42<00:03, 208.18batch/s]

Training batches on cuda:  92%|█████████▏| 8971/9708 [00:42<00:03, 206.22batch/s]

Training batches on cuda:  93%|█████████▎| 8992/9708 [00:42<00:03, 206.78batch/s]

Training batches on cuda:  93%|█████████▎| 9013/9708 [00:42<00:03, 206.59batch/s]

Training batches on cuda:  93%|█████████▎| 9035/9708 [00:43<00:03, 208.29batch/s]

Training batches on cuda:  93%|█████████▎| 9057/9708 [00:43<00:03, 210.20batch/s]

Training batches on cuda:  94%|█████████▎| 9081/9708 [00:43<00:02, 215.76batch/s]

Training batches on cuda:  94%|█████████▍| 9103/9708 [00:43<00:02, 212.88batch/s]

Training batches on cuda:  94%|█████████▍| 9128/9708 [00:43<00:02, 220.92batch/s]

Training batches on cuda:  94%|█████████▍| 9152/9708 [00:43<00:02, 226.23batch/s]

Training batches on cuda:  95%|█████████▍| 9175/9708 [00:43<00:02, 225.14batch/s]

Training batches on cuda:  95%|█████████▍| 9201/9708 [00:43<00:02, 233.34batch/s]

Training batches on cuda:  95%|█████████▌| 9228/9708 [00:43<00:01, 242.27batch/s]

Training batches on cuda:  95%|█████████▌| 9255/9708 [00:44<00:01, 248.07batch/s]

Training batches on cuda:  96%|█████████▌| 9280/9708 [00:44<00:01, 239.77batch/s]

Training batches on cuda:  96%|█████████▌| 9305/9708 [00:44<00:01, 237.44batch/s]

Training batches on cuda:  96%|█████████▌| 9329/9708 [00:44<00:01, 233.97batch/s]

Training batches on cuda:  96%|█████████▋| 9353/9708 [00:44<00:01, 232.25batch/s]

Training batches on cuda:  97%|█████████▋| 9378/9708 [00:44<00:01, 234.65batch/s]

Training batches on cuda:  97%|█████████▋| 9402/9708 [00:44<00:01, 202.42batch/s]

Training batches on cuda:  97%|█████████▋| 9425/9708 [00:44<00:01, 208.34batch/s]

Training batches on cuda:  97%|█████████▋| 9449/9708 [00:44<00:01, 215.30batch/s]

Training batches on cuda:  98%|█████████▊| 9472/9708 [00:45<00:01, 217.05batch/s]

Training batches on cuda:  98%|█████████▊| 9495/9708 [00:45<00:00, 220.46batch/s]

Training batches on cuda:  98%|█████████▊| 9518/9708 [00:45<00:00, 220.49batch/s]

Training batches on cuda:  98%|█████████▊| 9543/9708 [00:45<00:00, 227.45batch/s]

Training batches on cuda:  99%|█████████▊| 9566/9708 [00:45<00:00, 223.30batch/s]

Training batches on cuda:  99%|█████████▉| 9589/9708 [00:45<00:00, 219.08batch/s]

Training batches on cuda:  99%|█████████▉| 9612/9708 [00:45<00:00, 214.77batch/s]

Training batches on cuda:  99%|█████████▉| 9634/9708 [00:45<00:00, 212.92batch/s]

Training batches on cuda:  99%|█████████▉| 9657/9708 [00:45<00:00, 216.42batch/s]

Training batches on cuda: 100%|█████████▉| 9682/9708 [00:45<00:00, 224.37batch/s]

Training batches on cuda: 100%|██████████| 9708/9708 [00:46<00:00, 228.94batch/s]

                                                                                 
Training epochs on cuda:  10%|█         | 1/10 [01:33<07:02, 46.92s/epoch, loss=0.096, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [01:33<06:13, 46.74s/epoch, loss=0.096, prev_loss=nan]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 18/9708 [00:00<00:53, 179.99batch/s]

Training batches on cuda:   0%|          | 42/9708 [00:00<00:49, 193.40batch/s]

Training batches on cuda:   1%|          | 64/9708 [00:00<00:48, 199.22batch/s]

Training batches on cuda:   1%|          | 87/9708 [00:00<00:46, 207.21batch/s]

Training batches on cuda:   1%|          | 110/9708 [00:00<00:45, 210.82batch/s]

Training batches on cuda:   1%|▏         | 130/9708 [00:00<00:46, 205.07batch/s]

Training batches on cuda:   2%|▏         | 153/9708 [00:00<00:45, 209.40batch/s]

Training batches on cuda:   2%|▏         | 175/9708 [00:00<00:45, 209.38batch/s]

Training batches on cuda:   2%|▏         | 198/9708 [00:00<00:44, 213.76batch/s]

Training batches on cuda:   2%|▏         | 221/9708 [00:01<00:43, 217.20batch/s]

Training batches on cuda:   3%|▎         | 244/9708 [00:01<00:42, 220.20batch/s]

Training batches on cuda:   3%|▎         | 267/9708 [00:01<00:42, 221.76batch/s]

Training batches on cuda:   3%|▎         | 289/9708 [00:01<00:44, 213.76batch/s]

Training batches on cuda:   3%|▎         | 311/9708 [00:01<00:45, 206.93batch/s]

Training batches on cuda:   3%|▎         | 332/9708 [00:01<00:46, 203.19batch/s]

Training batches on cuda:   4%|▎         | 353/9708 [00:01<00:46, 200.12batch/s]

Training batches on cuda:   4%|▍         | 374/9708 [00:01<00:48, 192.48batch/s]

Training batches on cuda:   4%|▍         | 394/9708 [00:01<00:50, 185.53batch/s]

Training batches on cuda:   4%|▍         | 416/9708 [00:02<00:47, 193.91batch/s]

Training batches on cuda:   5%|▍         | 437/9708 [00:02<00:47, 195.47batch/s]

Training batches on cuda:   5%|▍         | 457/9708 [00:02<00:48, 190.07batch/s]

Training batches on cuda:   5%|▍         | 477/9708 [00:02<00:49, 186.42batch/s]

Training batches on cuda:   5%|▌         | 499/9708 [00:02<00:47, 192.47batch/s]

Training batches on cuda:   5%|▌         | 519/9708 [00:02<00:48, 188.66batch/s]

Training batches on cuda:   6%|▌         | 539/9708 [00:02<00:47, 191.45batch/s]

Training batches on cuda:   6%|▌         | 561/9708 [00:02<00:46, 197.66batch/s]

Training batches on cuda:   6%|▌         | 582/9708 [00:02<00:45, 198.68batch/s]

Training batches on cuda:   6%|▌         | 604/9708 [00:02<00:44, 202.81batch/s]

Training batches on cuda:   6%|▋         | 625/9708 [00:03<00:45, 201.74batch/s]

Training batches on cuda:   7%|▋         | 646/9708 [00:03<00:46, 194.55batch/s]

Training batches on cuda:   7%|▋         | 666/9708 [00:03<00:48, 185.06batch/s]

Training batches on cuda:   7%|▋         | 685/9708 [00:03<00:53, 169.46batch/s]

Training batches on cuda:   7%|▋         | 706/9708 [00:03<00:50, 179.33batch/s]

Training batches on cuda:   7%|▋         | 726/9708 [00:03<00:48, 184.78batch/s]

Training batches on cuda:   8%|▊         | 747/9708 [00:03<00:47, 189.39batch/s]

Training batches on cuda:   8%|▊         | 767/9708 [00:03<00:49, 182.32batch/s]

Training batches on cuda:   8%|▊         | 786/9708 [00:03<00:49, 181.28batch/s]

Training batches on cuda:   8%|▊         | 805/9708 [00:04<00:49, 179.44batch/s]

Training batches on cuda:   9%|▊         | 828/9708 [00:04<00:46, 190.99batch/s]

Training batches on cuda:   9%|▉         | 855/9708 [00:04<00:42, 207.82batch/s]

Training batches on cuda:   9%|▉         | 880/9708 [00:04<00:40, 218.47batch/s]

Training batches on cuda:   9%|▉         | 903/9708 [00:04<00:39, 221.00batch/s]

Training batches on cuda:  10%|▉         | 926/9708 [00:04<00:39, 221.04batch/s]

Training batches on cuda:  10%|▉         | 949/9708 [00:04<00:39, 220.00batch/s]

Training batches on cuda:  10%|█         | 972/9708 [00:04<00:40, 218.25batch/s]

Training batches on cuda:  10%|█         | 994/9708 [00:04<00:40, 212.75batch/s]

Training batches on cuda:  10%|█         | 1016/9708 [00:05<00:42, 204.57batch/s]

Training batches on cuda:  11%|█         | 1037/9708 [00:05<00:42, 205.25batch/s]

Training batches on cuda:  11%|█         | 1058/9708 [00:05<00:43, 200.38batch/s]

Training batches on cuda:  11%|█         | 1080/9708 [00:05<00:41, 205.59batch/s]

Training batches on cuda:  11%|█▏        | 1103/9708 [00:05<00:40, 211.86batch/s]

Training batches on cuda:  12%|█▏        | 1127/9708 [00:05<00:39, 218.46batch/s]

Training batches on cuda:  12%|█▏        | 1150/9708 [00:05<00:38, 219.46batch/s]

Training batches on cuda:  12%|█▏        | 1175/9708 [00:05<00:37, 225.95batch/s]

Training batches on cuda:  12%|█▏        | 1200/9708 [00:05<00:36, 230.60batch/s]

Training batches on cuda:  13%|█▎        | 1224/9708 [00:05<00:36, 232.06batch/s]

Training batches on cuda:  13%|█▎        | 1248/9708 [00:06<00:36, 232.97batch/s]

Training batches on cuda:  13%|█▎        | 1273/9708 [00:06<00:35, 236.88batch/s]

Training batches on cuda:  13%|█▎        | 1297/9708 [00:06<00:36, 229.83batch/s]

Training batches on cuda:  14%|█▎        | 1321/9708 [00:06<00:37, 225.94batch/s]

Training batches on cuda:  14%|█▍        | 1344/9708 [00:06<00:38, 218.74batch/s]

Training batches on cuda:  14%|█▍        | 1366/9708 [00:06<00:39, 211.20batch/s]

Training batches on cuda:  14%|█▍        | 1388/9708 [00:06<00:40, 206.77batch/s]

Training batches on cuda:  15%|█▍        | 1409/9708 [00:06<00:40, 202.96batch/s]

Training batches on cuda:  15%|█▍        | 1431/9708 [00:06<00:40, 205.66batch/s]

Training batches on cuda:  15%|█▍        | 1453/9708 [00:07<00:39, 207.23batch/s]

Training batches on cuda:  15%|█▌        | 1474/9708 [00:07<00:39, 207.26batch/s]

Training batches on cuda:  15%|█▌        | 1496/9708 [00:07<00:39, 208.21batch/s]

Training batches on cuda:  16%|█▌        | 1518/9708 [00:07<00:39, 209.61batch/s]

Training batches on cuda:  16%|█▌        | 1539/9708 [00:07<00:38, 209.58batch/s]

Training batches on cuda:  16%|█▌        | 1560/9708 [00:07<00:39, 208.75batch/s]

Training batches on cuda:  16%|█▋        | 1583/9708 [00:07<00:37, 214.47batch/s]

Training batches on cuda:  17%|█▋        | 1607/9708 [00:07<00:36, 219.15batch/s]

Training batches on cuda:  17%|█▋        | 1630/9708 [00:07<00:36, 221.99batch/s]

Training batches on cuda:  17%|█▋        | 1654/9708 [00:07<00:35, 224.31batch/s]

Training batches on cuda:  17%|█▋        | 1677/9708 [00:08<00:37, 215.01batch/s]

Training batches on cuda:  18%|█▊        | 1700/9708 [00:08<00:36, 218.09batch/s]

Training batches on cuda:  18%|█▊        | 1723/9708 [00:08<00:36, 220.00batch/s]

Training batches on cuda:  18%|█▊        | 1746/9708 [00:08<00:35, 221.67batch/s]

Training batches on cuda:  18%|█▊        | 1769/9708 [00:08<00:39, 198.75batch/s]

Training batches on cuda:  18%|█▊        | 1791/9708 [00:08<00:38, 203.87batch/s]

Training batches on cuda:  19%|█▊        | 1812/9708 [00:08<00:38, 205.49batch/s]

Training batches on cuda:  19%|█▉        | 1835/9708 [00:08<00:37, 211.62batch/s]

Training batches on cuda:  19%|█▉        | 1857/9708 [00:08<00:36, 212.41batch/s]

Training batches on cuda:  19%|█▉        | 1879/9708 [00:09<00:37, 211.49batch/s]

Training batches on cuda:  20%|█▉        | 1901/9708 [00:09<00:36, 211.49batch/s]

Training batches on cuda:  20%|█▉        | 1923/9708 [00:09<00:36, 210.85batch/s]

Training batches on cuda:  20%|██        | 1945/9708 [00:09<00:37, 207.82batch/s]

Training batches on cuda:  20%|██        | 1966/9708 [00:09<00:37, 206.68batch/s]

Training batches on cuda:  20%|██        | 1988/9708 [00:09<00:36, 210.49batch/s]

Training batches on cuda:  21%|██        | 2010/9708 [00:09<00:36, 209.88batch/s]

Training batches on cuda:  21%|██        | 2032/9708 [00:09<00:36, 208.73batch/s]

Training batches on cuda:  21%|██        | 2053/9708 [00:09<00:37, 201.68batch/s]

Training batches on cuda:  21%|██▏       | 2074/9708 [00:09<00:37, 203.40batch/s]

Training batches on cuda:  22%|██▏       | 2095/9708 [00:10<00:37, 202.75batch/s]

Training batches on cuda:  22%|██▏       | 2116/9708 [00:10<00:37, 204.14batch/s]

Training batches on cuda:  22%|██▏       | 2138/9708 [00:10<00:36, 208.46batch/s]

Training batches on cuda:  22%|██▏       | 2162/9708 [00:10<00:35, 214.11batch/s]

Training batches on cuda:  22%|██▏       | 2184/9708 [00:10<00:34, 215.62batch/s]

Training batches on cuda:  23%|██▎       | 2207/9708 [00:10<00:34, 216.87batch/s]

Training batches on cuda:  23%|██▎       | 2230/9708 [00:10<00:34, 218.57batch/s]

Training batches on cuda:  23%|██▎       | 2253/9708 [00:10<00:33, 220.69batch/s]

Training batches on cuda:  23%|██▎       | 2276/9708 [00:10<00:33, 223.13batch/s]

Training batches on cuda:  24%|██▎       | 2299/9708 [00:11<00:33, 224.34batch/s]

Training batches on cuda:  24%|██▍       | 2322/9708 [00:11<00:33, 223.30batch/s]

Training batches on cuda:  24%|██▍       | 2345/9708 [00:11<00:32, 223.61batch/s]

Training batches on cuda:  24%|██▍       | 2370/9708 [00:11<00:32, 228.17batch/s]

Training batches on cuda:  25%|██▍       | 2394/9708 [00:11<00:31, 230.28batch/s]

Training batches on cuda:  25%|██▍       | 2419/9708 [00:11<00:31, 233.84batch/s]

Training batches on cuda:  25%|██▌       | 2443/9708 [00:11<00:31, 230.36batch/s]

Training batches on cuda:  25%|██▌       | 2467/9708 [00:11<00:32, 222.53batch/s]

Training batches on cuda:  26%|██▌       | 2490/9708 [00:11<00:33, 217.81batch/s]

Training batches on cuda:  26%|██▌       | 2512/9708 [00:11<00:34, 209.52batch/s]

Training batches on cuda:  26%|██▌       | 2535/9708 [00:12<00:33, 213.15batch/s]

Training batches on cuda:  26%|██▋       | 2557/9708 [00:12<00:33, 210.90batch/s]

Training batches on cuda:  27%|██▋       | 2579/9708 [00:12<00:34, 207.68batch/s]

Training batches on cuda:  27%|██▋       | 2600/9708 [00:12<00:35, 201.91batch/s]

Training batches on cuda:  27%|██▋       | 2621/9708 [00:12<00:35, 201.19batch/s]

Training batches on cuda:  27%|██▋       | 2642/9708 [00:12<00:34, 203.24batch/s]

Training batches on cuda:  27%|██▋       | 2663/9708 [00:12<00:34, 202.52batch/s]

Training batches on cuda:  28%|██▊       | 2686/9708 [00:12<00:33, 209.20batch/s]

Training batches on cuda:  28%|██▊       | 2708/9708 [00:12<00:33, 210.88batch/s]

Training batches on cuda:  28%|██▊       | 2730/9708 [00:13<00:33, 210.29batch/s]

Training batches on cuda:  28%|██▊       | 2752/9708 [00:13<00:32, 210.84batch/s]

Training batches on cuda:  29%|██▊       | 2774/9708 [00:13<00:33, 208.60batch/s]

Training batches on cuda:  29%|██▉       | 2795/9708 [00:13<00:33, 208.02batch/s]

Training batches on cuda:  29%|██▉       | 2816/9708 [00:13<00:33, 207.07batch/s]

Training batches on cuda:  29%|██▉       | 2837/9708 [00:13<00:37, 182.06batch/s]

Training batches on cuda:  29%|██▉       | 2857/9708 [00:13<00:36, 186.35batch/s]

Training batches on cuda:  30%|██▉       | 2878/9708 [00:13<00:35, 192.03batch/s]

Training batches on cuda:  30%|██▉       | 2900/9708 [00:13<00:34, 199.20batch/s]

Training batches on cuda:  30%|███       | 2921/9708 [00:13<00:33, 201.96batch/s]

Training batches on cuda:  30%|███       | 2944/9708 [00:14<00:32, 207.68batch/s]

Training batches on cuda:  31%|███       | 2966/9708 [00:14<00:32, 209.92batch/s]

Training batches on cuda:  31%|███       | 2988/9708 [00:14<00:31, 211.34batch/s]

Training batches on cuda:  31%|███       | 3010/9708 [00:14<00:31, 212.43batch/s]

Training batches on cuda:  31%|███       | 3032/9708 [00:14<00:31, 209.35batch/s]

Training batches on cuda:  31%|███▏      | 3054/9708 [00:14<00:31, 211.37batch/s]

Training batches on cuda:  32%|███▏      | 3077/9708 [00:14<00:30, 215.87batch/s]

Training batches on cuda:  32%|███▏      | 3099/9708 [00:14<00:30, 213.91batch/s]

Training batches on cuda:  32%|███▏      | 3121/9708 [00:14<00:31, 210.89batch/s]

Training batches on cuda:  32%|███▏      | 3143/9708 [00:15<00:31, 211.05batch/s]

Training batches on cuda:  33%|███▎      | 3165/9708 [00:15<00:31, 208.21batch/s]

Training batches on cuda:  33%|███▎      | 3186/9708 [00:15<00:31, 204.07batch/s]

Training batches on cuda:  33%|███▎      | 3208/9708 [00:15<00:31, 207.58batch/s]

Training batches on cuda:  33%|███▎      | 3229/9708 [00:15<00:31, 205.85batch/s]

Training batches on cuda:  33%|███▎      | 3252/9708 [00:15<00:30, 210.25batch/s]

Training batches on cuda:  34%|███▎      | 3275/9708 [00:15<00:29, 214.78batch/s]

Training batches on cuda:  34%|███▍      | 3297/9708 [00:15<00:31, 201.83batch/s]

Training batches on cuda:  34%|███▍      | 3318/9708 [00:15<00:31, 201.41batch/s]

Training batches on cuda:  34%|███▍      | 3341/9708 [00:15<00:30, 207.70batch/s]

Training batches on cuda:  35%|███▍      | 3364/9708 [00:16<00:29, 211.50batch/s]

Training batches on cuda:  35%|███▍      | 3387/9708 [00:16<00:29, 216.02batch/s]

Training batches on cuda:  35%|███▌      | 3409/9708 [00:16<00:29, 216.71batch/s]

Training batches on cuda:  35%|███▌      | 3431/9708 [00:16<00:30, 204.48batch/s]

Training batches on cuda:  36%|███▌      | 3453/9708 [00:16<00:30, 208.12batch/s]

Training batches on cuda:  36%|███▌      | 3474/9708 [00:16<00:30, 201.93batch/s]

Training batches on cuda:  36%|███▌      | 3495/9708 [00:16<00:30, 203.38batch/s]

Training batches on cuda:  36%|███▌      | 3518/9708 [00:16<00:29, 209.76batch/s]

Training batches on cuda:  36%|███▋      | 3540/9708 [00:16<00:30, 204.90batch/s]

Training batches on cuda:  37%|███▋      | 3561/9708 [00:17<00:31, 198.21batch/s]

Training batches on cuda:  37%|███▋      | 3581/9708 [00:17<00:31, 195.67batch/s]

Training batches on cuda:  37%|███▋      | 3602/9708 [00:17<00:30, 197.62batch/s]

Training batches on cuda:  37%|███▋      | 3623/9708 [00:17<00:30, 199.32batch/s]

Training batches on cuda:  38%|███▊      | 3645/9708 [00:17<00:29, 204.07batch/s]

Training batches on cuda:  38%|███▊      | 3666/9708 [00:17<00:29, 205.30batch/s]

Training batches on cuda:  38%|███▊      | 3687/9708 [00:17<00:30, 199.17batch/s]

Training batches on cuda:  38%|███▊      | 3707/9708 [00:17<00:30, 197.39batch/s]

Training batches on cuda:  38%|███▊      | 3727/9708 [00:17<00:30, 197.83batch/s]

Training batches on cuda:  39%|███▊      | 3747/9708 [00:17<00:30, 197.92batch/s]

Training batches on cuda:  39%|███▉      | 3769/9708 [00:18<00:29, 202.00batch/s]

Training batches on cuda:  39%|███▉      | 3790/9708 [00:18<00:29, 199.29batch/s]

Training batches on cuda:  39%|███▉      | 3811/9708 [00:18<00:29, 202.12batch/s]

Training batches on cuda:  39%|███▉      | 3833/9708 [00:18<00:28, 205.55batch/s]

Training batches on cuda:  40%|███▉      | 3854/9708 [00:18<00:28, 204.54batch/s]

Training batches on cuda:  40%|███▉      | 3875/9708 [00:18<00:33, 176.37batch/s]

Training batches on cuda:  40%|████      | 3894/9708 [00:18<00:32, 178.37batch/s]

Training batches on cuda:  40%|████      | 3913/9708 [00:18<00:32, 180.01batch/s]

Training batches on cuda:  41%|████      | 3932/9708 [00:18<00:32, 178.99batch/s]

Training batches on cuda:  41%|████      | 3952/9708 [00:19<00:31, 183.67batch/s]

Training batches on cuda:  41%|████      | 3973/9708 [00:19<00:30, 189.91batch/s]

Training batches on cuda:  41%|████      | 3993/9708 [00:19<00:29, 192.11batch/s]

Training batches on cuda:  41%|████▏     | 4015/9708 [00:19<00:28, 197.57batch/s]

Training batches on cuda:  42%|████▏     | 4036/9708 [00:19<00:28, 200.95batch/s]

Training batches on cuda:  42%|████▏     | 4059/9708 [00:19<00:27, 208.13batch/s]

Training batches on cuda:  42%|████▏     | 4081/9708 [00:19<00:26, 210.08batch/s]

Training batches on cuda:  42%|████▏     | 4103/9708 [00:19<00:26, 210.42batch/s]

Training batches on cuda:  42%|████▏     | 4125/9708 [00:19<00:26, 212.65batch/s]

Training batches on cuda:  43%|████▎     | 4147/9708 [00:20<00:26, 211.70batch/s]

Training batches on cuda:  43%|████▎     | 4169/9708 [00:20<00:26, 212.53batch/s]

Training batches on cuda:  43%|████▎     | 4191/9708 [00:20<00:25, 212.98batch/s]

Training batches on cuda:  43%|████▎     | 4213/9708 [00:20<00:25, 212.61batch/s]

Training batches on cuda:  44%|████▎     | 4235/9708 [00:20<00:25, 212.90batch/s]

Training batches on cuda:  44%|████▍     | 4257/9708 [00:20<00:25, 212.74batch/s]

Training batches on cuda:  44%|████▍     | 4280/9708 [00:20<00:25, 215.66batch/s]

Training batches on cuda:  44%|████▍     | 4302/9708 [00:20<00:24, 216.65batch/s]

Training batches on cuda:  45%|████▍     | 4326/9708 [00:20<00:24, 221.70batch/s]

Training batches on cuda:  45%|████▍     | 4350/9708 [00:20<00:23, 225.12batch/s]

Training batches on cuda:  45%|████▌     | 4373/9708 [00:21<00:23, 224.75batch/s]

Training batches on cuda:  45%|████▌     | 4396/9708 [00:21<00:23, 225.43batch/s]

Training batches on cuda:  46%|████▌     | 4419/9708 [00:21<00:23, 220.64batch/s]

Training batches on cuda:  46%|████▌     | 4442/9708 [00:21<00:24, 218.23batch/s]

Training batches on cuda:  46%|████▌     | 4464/9708 [00:21<00:24, 216.74batch/s]

Training batches on cuda:  46%|████▌     | 4486/9708 [00:21<00:24, 211.42batch/s]

Training batches on cuda:  46%|████▋     | 4508/9708 [00:21<00:24, 209.63batch/s]

Training batches on cuda:  47%|████▋     | 4529/9708 [00:21<00:24, 209.23batch/s]

Training batches on cuda:  47%|████▋     | 4550/9708 [00:21<00:25, 201.16batch/s]

Training batches on cuda:  47%|████▋     | 4573/9708 [00:21<00:24, 207.74batch/s]

Training batches on cuda:  47%|████▋     | 4595/9708 [00:22<00:24, 209.71batch/s]

Training batches on cuda:  48%|████▊     | 4617/9708 [00:22<00:24, 204.61batch/s]

Training batches on cuda:  48%|████▊     | 4639/9708 [00:22<00:24, 207.39batch/s]

Training batches on cuda:  48%|████▊     | 4661/9708 [00:22<00:24, 209.13batch/s]

Training batches on cuda:  48%|████▊     | 4682/9708 [00:22<00:24, 208.91batch/s]

Training batches on cuda:  48%|████▊     | 4703/9708 [00:22<00:25, 199.95batch/s]

Training batches on cuda:  49%|████▊     | 4724/9708 [00:22<00:25, 197.17batch/s]

Training batches on cuda:  49%|████▉     | 4744/9708 [00:22<00:25, 195.27batch/s]

Training batches on cuda:  49%|████▉     | 4764/9708 [00:22<00:25, 194.97batch/s]

Training batches on cuda:  49%|████▉     | 4786/9708 [00:23<00:24, 200.93batch/s]

Training batches on cuda:  50%|████▉     | 4810/9708 [00:23<00:23, 209.07batch/s]

Training batches on cuda:  50%|████▉     | 4832/9708 [00:23<00:23, 210.18batch/s]

Training batches on cuda:  50%|█████     | 4854/9708 [00:23<00:23, 202.37batch/s]

Training batches on cuda:  50%|█████     | 4875/9708 [00:23<00:23, 201.89batch/s]

Training batches on cuda:  50%|█████     | 4896/9708 [00:23<00:24, 199.37batch/s]

Training batches on cuda:  51%|█████     | 4917/9708 [00:23<00:27, 172.59batch/s]

Training batches on cuda:  51%|█████     | 4937/9708 [00:23<00:26, 177.82batch/s]

Training batches on cuda:  51%|█████     | 4958/9708 [00:23<00:25, 184.38batch/s]

Training batches on cuda:  51%|█████▏    | 4978/9708 [00:24<00:25, 186.84batch/s]

Training batches on cuda:  52%|█████▏    | 5000/9708 [00:24<00:24, 193.88batch/s]

Training batches on cuda:  52%|█████▏    | 5021/9708 [00:24<00:23, 197.20batch/s]

Training batches on cuda:  52%|█████▏    | 5043/9708 [00:24<00:23, 202.62batch/s]

Training batches on cuda:  52%|█████▏    | 5069/9708 [00:24<00:21, 215.12batch/s]

Training batches on cuda:  52%|█████▏    | 5093/9708 [00:24<00:20, 221.77batch/s]

Training batches on cuda:  53%|█████▎    | 5117/9708 [00:24<00:20, 225.84batch/s]

Training batches on cuda:  53%|█████▎    | 5140/9708 [00:24<00:20, 226.01batch/s]

Training batches on cuda:  53%|█████▎    | 5163/9708 [00:24<00:20, 227.00batch/s]

Training batches on cuda:  53%|█████▎    | 5186/9708 [00:24<00:19, 227.88batch/s]

Training batches on cuda:  54%|█████▎    | 5209/9708 [00:25<00:19, 228.42batch/s]

Training batches on cuda:  54%|█████▍    | 5233/9708 [00:25<00:19, 229.09batch/s]

Training batches on cuda:  54%|█████▍    | 5256/9708 [00:25<00:19, 225.55batch/s]

Training batches on cuda:  54%|█████▍    | 5280/9708 [00:25<00:19, 226.77batch/s]

Training batches on cuda:  55%|█████▍    | 5303/9708 [00:25<00:19, 225.16batch/s]

Training batches on cuda:  55%|█████▍    | 5326/9708 [00:25<00:19, 221.78batch/s]

Training batches on cuda:  55%|█████▌    | 5349/9708 [00:25<00:20, 212.89batch/s]

Training batches on cuda:  55%|█████▌    | 5371/9708 [00:25<00:20, 213.42batch/s]

Training batches on cuda:  56%|█████▌    | 5394/9708 [00:25<00:19, 216.29batch/s]

Training batches on cuda:  56%|█████▌    | 5416/9708 [00:26<00:19, 215.25batch/s]

Training batches on cuda:  56%|█████▌    | 5438/9708 [00:26<00:19, 214.72batch/s]

Training batches on cuda:  56%|█████▌    | 5460/9708 [00:26<00:20, 208.84batch/s]

Training batches on cuda:  56%|█████▋    | 5481/9708 [00:26<00:20, 206.65batch/s]

Training batches on cuda:  57%|█████▋    | 5502/9708 [00:26<00:21, 196.73batch/s]

Training batches on cuda:  57%|█████▋    | 5524/9708 [00:26<00:20, 202.53batch/s]

Training batches on cuda:  57%|█████▋    | 5545/9708 [00:26<00:20, 204.11batch/s]

Training batches on cuda:  57%|█████▋    | 5566/9708 [00:26<00:20, 199.91batch/s]

Training batches on cuda:  58%|█████▊    | 5589/9708 [00:26<00:19, 206.31batch/s]

Training batches on cuda:  58%|█████▊    | 5610/9708 [00:26<00:20, 198.05batch/s]

Training batches on cuda:  58%|█████▊    | 5631/9708 [00:27<00:20, 201.03batch/s]

Training batches on cuda:  58%|█████▊    | 5655/9708 [00:27<00:19, 210.35batch/s]

Training batches on cuda:  58%|█████▊    | 5679/9708 [00:27<00:18, 217.47batch/s]

Training batches on cuda:  59%|█████▊    | 5702/9708 [00:27<00:18, 218.81batch/s]

Training batches on cuda:  59%|█████▉    | 5726/9708 [00:27<00:17, 224.43batch/s]

Training batches on cuda:  59%|█████▉    | 5750/9708 [00:27<00:17, 226.49batch/s]

Training batches on cuda:  59%|█████▉    | 5774/9708 [00:27<00:17, 229.05batch/s]

Training batches on cuda:  60%|█████▉    | 5800/9708 [00:27<00:16, 237.22batch/s]

Training batches on cuda:  60%|██████    | 5825/9708 [00:27<00:16, 240.02batch/s]

Training batches on cuda:  60%|██████    | 5851/9708 [00:28<00:15, 243.83batch/s]

Training batches on cuda:  61%|██████    | 5876/9708 [00:28<00:15, 244.23batch/s]

Training batches on cuda:  61%|██████    | 5901/9708 [00:28<00:18, 205.57batch/s]

Training batches on cuda:  61%|██████    | 5923/9708 [00:28<00:18, 208.06batch/s]

Training batches on cuda:  61%|██████▏   | 5948/9708 [00:28<00:17, 218.37batch/s]

Training batches on cuda:  62%|██████▏   | 5972/9708 [00:28<00:16, 221.52batch/s]

Training batches on cuda:  62%|██████▏   | 5995/9708 [00:28<00:19, 193.53batch/s]

Training batches on cuda:  62%|██████▏   | 6019/9708 [00:28<00:17, 204.99batch/s]

Training batches on cuda:  62%|██████▏   | 6044/9708 [00:28<00:17, 214.61batch/s]

Training batches on cuda:  62%|██████▏   | 6067/9708 [00:29<00:16, 217.65batch/s]

Training batches on cuda:  63%|██████▎   | 6090/9708 [00:29<00:16, 220.19batch/s]

Training batches on cuda:  63%|██████▎   | 6113/9708 [00:29<00:16, 215.19batch/s]

Training batches on cuda:  63%|██████▎   | 6135/9708 [00:29<00:16, 214.26batch/s]

Training batches on cuda:  63%|██████▎   | 6157/9708 [00:29<00:16, 213.73batch/s]

Training batches on cuda:  64%|██████▎   | 6180/9708 [00:29<00:16, 216.70batch/s]

Training batches on cuda:  64%|██████▍   | 6203/9708 [00:29<00:16, 218.38batch/s]

Training batches on cuda:  64%|██████▍   | 6225/9708 [00:29<00:16, 216.15batch/s]

Training batches on cuda:  64%|██████▍   | 6247/9708 [00:29<00:15, 216.94batch/s]

Training batches on cuda:  65%|██████▍   | 6271/9708 [00:29<00:15, 223.02batch/s]

Training batches on cuda:  65%|██████▍   | 6296/9708 [00:30<00:14, 230.31batch/s]

Training batches on cuda:  65%|██████▌   | 6321/9708 [00:30<00:14, 235.66batch/s]

Training batches on cuda:  65%|██████▌   | 6345/9708 [00:30<00:15, 222.43batch/s]

Training batches on cuda:  66%|██████▌   | 6368/9708 [00:30<00:15, 217.28batch/s]

Training batches on cuda:  66%|██████▌   | 6391/9708 [00:30<00:15, 219.49batch/s]

Training batches on cuda:  66%|██████▌   | 6414/9708 [00:30<00:14, 222.01batch/s]

Training batches on cuda:  66%|██████▋   | 6437/9708 [00:30<00:14, 221.52batch/s]

Training batches on cuda:  67%|██████▋   | 6461/9708 [00:30<00:14, 224.12batch/s]

Training batches on cuda:  67%|██████▋   | 6485/9708 [00:30<00:14, 226.31batch/s]

Training batches on cuda:  67%|██████▋   | 6509/9708 [00:31<00:14, 227.45batch/s]

Training batches on cuda:  67%|██████▋   | 6532/9708 [00:31<00:14, 221.19batch/s]

Training batches on cuda:  68%|██████▊   | 6555/9708 [00:31<00:14, 219.34batch/s]

Training batches on cuda:  68%|██████▊   | 6578/9708 [00:31<00:14, 220.50batch/s]

Training batches on cuda:  68%|██████▊   | 6601/9708 [00:31<00:13, 223.09batch/s]

Training batches on cuda:  68%|██████▊   | 6624/9708 [00:31<00:13, 223.36batch/s]

Training batches on cuda:  68%|██████▊   | 6648/9708 [00:31<00:13, 225.49batch/s]

Training batches on cuda:  69%|██████▊   | 6671/9708 [00:31<00:13, 225.37batch/s]

Training batches on cuda:  69%|██████▉   | 6694/9708 [00:31<00:13, 224.86batch/s]

Training batches on cuda:  69%|██████▉   | 6717/9708 [00:31<00:14, 212.15batch/s]

Training batches on cuda:  69%|██████▉   | 6739/9708 [00:32<00:14, 211.62batch/s]

Training batches on cuda:  70%|██████▉   | 6761/9708 [00:32<00:14, 209.60batch/s]

Training batches on cuda:  70%|██████▉   | 6783/9708 [00:32<00:13, 211.27batch/s]

Training batches on cuda:  70%|███████   | 6805/9708 [00:32<00:13, 211.30batch/s]

Training batches on cuda:  70%|███████   | 6828/9708 [00:32<00:13, 214.53batch/s]

Training batches on cuda:  71%|███████   | 6850/9708 [00:32<00:13, 210.47batch/s]

Training batches on cuda:  71%|███████   | 6872/9708 [00:32<00:14, 199.78batch/s]

Training batches on cuda:  71%|███████   | 6894/9708 [00:32<00:13, 202.80batch/s]

Training batches on cuda:  71%|███████   | 6916/9708 [00:32<00:13, 205.93batch/s]

Training batches on cuda:  71%|███████▏  | 6937/9708 [00:33<00:13, 205.67batch/s]

Training batches on cuda:  72%|███████▏  | 6958/9708 [00:33<00:13, 205.12batch/s]

Training batches on cuda:  72%|███████▏  | 6980/9708 [00:33<00:13, 207.25batch/s]

Training batches on cuda:  72%|███████▏  | 7001/9708 [00:33<00:13, 206.38batch/s]

Training batches on cuda:  72%|███████▏  | 7023/9708 [00:33<00:12, 208.16batch/s]

Training batches on cuda:  73%|███████▎  | 7044/9708 [00:33<00:12, 207.68batch/s]

Training batches on cuda:  73%|███████▎  | 7065/9708 [00:33<00:12, 205.58batch/s]

Training batches on cuda:  73%|███████▎  | 7086/9708 [00:33<00:13, 198.43batch/s]

Training batches on cuda:  73%|███████▎  | 7107/9708 [00:33<00:12, 201.36batch/s]

Training batches on cuda:  73%|███████▎  | 7129/9708 [00:33<00:12, 204.69batch/s]

Training batches on cuda:  74%|███████▎  | 7152/9708 [00:34<00:12, 210.49batch/s]

Training batches on cuda:  74%|███████▍  | 7174/9708 [00:34<00:12, 210.50batch/s]

Training batches on cuda:  74%|███████▍  | 7198/9708 [00:34<00:11, 217.80batch/s]

Training batches on cuda:  74%|███████▍  | 7222/9708 [00:34<00:11, 222.09batch/s]

Training batches on cuda:  75%|███████▍  | 7245/9708 [00:34<00:11, 217.30batch/s]

Training batches on cuda:  75%|███████▍  | 7267/9708 [00:34<00:11, 213.46batch/s]

Training batches on cuda:  75%|███████▌  | 7290/9708 [00:34<00:11, 216.02batch/s]

Training batches on cuda:  75%|███████▌  | 7312/9708 [00:34<00:11, 211.01batch/s]

Training batches on cuda:  76%|███████▌  | 7334/9708 [00:34<00:11, 208.26batch/s]

Training batches on cuda:  76%|███████▌  | 7356/9708 [00:35<00:11, 210.16batch/s]

Training batches on cuda:  76%|███████▌  | 7379/9708 [00:35<00:10, 214.99batch/s]

Training batches on cuda:  76%|███████▌  | 7402/9708 [00:35<00:10, 217.52batch/s]

Training batches on cuda:  76%|███████▋  | 7424/9708 [00:35<00:10, 217.17batch/s]

Training batches on cuda:  77%|███████▋  | 7448/9708 [00:35<00:10, 221.55batch/s]

Training batches on cuda:  77%|███████▋  | 7471/9708 [00:35<00:09, 223.93batch/s]

Training batches on cuda:  77%|███████▋  | 7494/9708 [00:35<00:09, 224.76batch/s]

Training batches on cuda:  77%|███████▋  | 7518/9708 [00:35<00:09, 226.36batch/s]

Training batches on cuda:  78%|███████▊  | 7541/9708 [00:35<00:09, 227.20batch/s]

Training batches on cuda:  78%|███████▊  | 7564/9708 [00:35<00:09, 226.24batch/s]

Training batches on cuda:  78%|███████▊  | 7587/9708 [00:36<00:09, 226.15batch/s]

Training batches on cuda:  78%|███████▊  | 7610/9708 [00:36<00:09, 225.47batch/s]

Training batches on cuda:  79%|███████▊  | 7633/9708 [00:36<00:09, 225.33batch/s]

Training batches on cuda:  79%|███████▉  | 7656/9708 [00:36<00:09, 224.57batch/s]

Training batches on cuda:  79%|███████▉  | 7679/9708 [00:36<00:08, 226.10batch/s]

Training batches on cuda:  79%|███████▉  | 7703/9708 [00:36<00:08, 227.34batch/s]

Training batches on cuda:  80%|███████▉  | 7726/9708 [00:36<00:08, 226.75batch/s]

Training batches on cuda:  80%|███████▉  | 7749/9708 [00:36<00:08, 225.54batch/s]

Training batches on cuda:  80%|████████  | 7772/9708 [00:36<00:08, 218.03batch/s]

Training batches on cuda:  80%|████████  | 7794/9708 [00:37<00:09, 211.96batch/s]

Training batches on cuda:  81%|████████  | 7816/9708 [00:38<00:33, 56.48batch/s] 

Training batches on cuda:  81%|████████  | 7836/9708 [00:38<00:26, 71.71batch/s]

Training batches on cuda:  81%|████████  | 7856/9708 [00:38<00:20, 88.34batch/s]

Training batches on cuda:  81%|████████  | 7878/9708 [00:38<00:17, 107.08batch/s]

Training batches on cuda:  81%|████████▏ | 7900/9708 [00:38<00:14, 126.57batch/s]

Training batches on cuda:  82%|████████▏ | 7924/9708 [00:38<00:12, 146.57batch/s]

Training batches on cuda:  82%|████████▏ | 7948/9708 [00:38<00:10, 164.89batch/s]

Training batches on cuda:  82%|████████▏ | 7970/9708 [00:38<00:10, 162.94batch/s]

Training batches on cuda:  82%|████████▏ | 7994/9708 [00:38<00:09, 179.41batch/s]

Training batches on cuda:  83%|████████▎ | 8018/9708 [00:39<00:08, 192.67batch/s]

Training batches on cuda:  83%|████████▎ | 8042/9708 [00:39<00:08, 202.79batch/s]

Training batches on cuda:  83%|████████▎ | 8065/9708 [00:39<00:07, 209.68batch/s]

Training batches on cuda:  83%|████████▎ | 8088/9708 [00:39<00:07, 209.19batch/s]

Training batches on cuda:  84%|████████▎ | 8110/9708 [00:39<00:07, 211.29batch/s]

Training batches on cuda:  84%|████████▍ | 8134/9708 [00:39<00:07, 217.24batch/s]

Training batches on cuda:  84%|████████▍ | 8158/9708 [00:39<00:06, 221.60batch/s]

Training batches on cuda:  84%|████████▍ | 8181/9708 [00:39<00:06, 223.03batch/s]

Training batches on cuda:  85%|████████▍ | 8204/9708 [00:39<00:06, 223.68batch/s]

Training batches on cuda:  85%|████████▍ | 8228/9708 [00:39<00:06, 226.15batch/s]

Training batches on cuda:  85%|████████▌ | 8252/9708 [00:40<00:06, 228.18batch/s]

Training batches on cuda:  85%|████████▌ | 8275/9708 [00:40<00:06, 228.38batch/s]

Training batches on cuda:  85%|████████▌ | 8298/9708 [00:40<00:06, 227.28batch/s]

Training batches on cuda:  86%|████████▌ | 8322/9708 [00:40<00:06, 228.38batch/s]

Training batches on cuda:  86%|████████▌ | 8346/9708 [00:40<00:05, 229.90batch/s]

Training batches on cuda:  86%|████████▌ | 8370/9708 [00:40<00:05, 231.14batch/s]

Training batches on cuda:  86%|████████▋ | 8394/9708 [00:40<00:05, 231.33batch/s]

Training batches on cuda:  87%|████████▋ | 8418/9708 [00:40<00:05, 230.33batch/s]

Training batches on cuda:  87%|████████▋ | 8442/9708 [00:40<00:05, 226.33batch/s]

Training batches on cuda:  87%|████████▋ | 8465/9708 [00:41<00:05, 219.65batch/s]

Training batches on cuda:  87%|████████▋ | 8488/9708 [00:41<00:05, 222.18batch/s]

Training batches on cuda:  88%|████████▊ | 8511/9708 [00:41<00:05, 223.95batch/s]

Training batches on cuda:  88%|████████▊ | 8535/9708 [00:41<00:05, 226.55batch/s]

Training batches on cuda:  88%|████████▊ | 8558/9708 [00:41<00:05, 222.89batch/s]

Training batches on cuda:  88%|████████▊ | 8581/9708 [00:41<00:05, 223.66batch/s]

Training batches on cuda:  89%|████████▊ | 8604/9708 [00:41<00:04, 224.19batch/s]

Training batches on cuda:  89%|████████▉ | 8627/9708 [00:41<00:04, 223.82batch/s]

Training batches on cuda:  89%|████████▉ | 8651/9708 [00:41<00:04, 226.62batch/s]

Training batches on cuda:  89%|████████▉ | 8674/9708 [00:41<00:04, 226.11batch/s]

Training batches on cuda:  90%|████████▉ | 8697/9708 [00:42<00:04, 223.91batch/s]

Training batches on cuda:  90%|████████▉ | 8720/9708 [00:42<00:04, 220.77batch/s]

Training batches on cuda:  90%|█████████ | 8743/9708 [00:42<00:04, 217.96batch/s]

Training batches on cuda:  90%|█████████ | 8765/9708 [00:42<00:04, 203.62batch/s]

Training batches on cuda:  91%|█████████ | 8786/9708 [00:42<00:04, 202.39batch/s]

Training batches on cuda:  91%|█████████ | 8807/9708 [00:42<00:04, 204.48batch/s]

Training batches on cuda:  91%|█████████ | 8829/9708 [00:42<00:04, 206.01batch/s]

Training batches on cuda:  91%|█████████ | 8850/9708 [00:42<00:04, 199.33batch/s]

Training batches on cuda:  91%|█████████▏| 8871/9708 [00:42<00:04, 196.43batch/s]

Training batches on cuda:  92%|█████████▏| 8892/9708 [00:43<00:04, 198.01batch/s]

Training batches on cuda:  92%|█████████▏| 8913/9708 [00:43<00:03, 200.72batch/s]

Training batches on cuda:  92%|█████████▏| 8934/9708 [00:43<00:03, 202.88batch/s]

Training batches on cuda:  92%|█████████▏| 8955/9708 [00:43<00:03, 204.05batch/s]

Training batches on cuda:  92%|█████████▏| 8976/9708 [00:43<00:03, 202.01batch/s]

Training batches on cuda:  93%|█████████▎| 8997/9708 [00:43<00:03, 201.92batch/s]

Training batches on cuda:  93%|█████████▎| 9018/9708 [00:43<00:03, 202.78batch/s]

Training batches on cuda:  93%|█████████▎| 9039/9708 [00:43<00:03, 201.52batch/s]

Training batches on cuda:  93%|█████████▎| 9060/9708 [00:43<00:03, 173.69batch/s]

Training batches on cuda:  94%|█████████▎| 9080/9708 [00:44<00:03, 180.40batch/s]

Training batches on cuda:  94%|█████████▍| 9102/9708 [00:44<00:03, 189.29batch/s]

Training batches on cuda:  94%|█████████▍| 9123/9708 [00:44<00:03, 194.91batch/s]

Training batches on cuda:  94%|█████████▍| 9144/9708 [00:44<00:02, 196.78batch/s]

Training batches on cuda:  94%|█████████▍| 9165/9708 [00:44<00:02, 199.43batch/s]

Training batches on cuda:  95%|█████████▍| 9187/9708 [00:44<00:02, 204.64batch/s]

Training batches on cuda:  95%|█████████▍| 9209/9708 [00:44<00:02, 207.00batch/s]

Training batches on cuda:  95%|█████████▌| 9231/9708 [00:44<00:02, 208.26batch/s]

Training batches on cuda:  95%|█████████▌| 9253/9708 [00:44<00:02, 208.60batch/s]

Training batches on cuda:  96%|█████████▌| 9275/9708 [00:44<00:02, 209.88batch/s]

Training batches on cuda:  96%|█████████▌| 9297/9708 [00:45<00:01, 211.63batch/s]

Training batches on cuda:  96%|█████████▌| 9320/9708 [00:45<00:01, 214.80batch/s]

Training batches on cuda:  96%|█████████▌| 9342/9708 [00:45<00:01, 211.37batch/s]

Training batches on cuda:  96%|█████████▋| 9364/9708 [00:45<00:01, 205.65batch/s]

Training batches on cuda:  97%|█████████▋| 9386/9708 [00:45<00:01, 207.86batch/s]

Training batches on cuda:  97%|█████████▋| 9408/9708 [00:45<00:01, 208.49batch/s]

Training batches on cuda:  97%|█████████▋| 9429/9708 [00:45<00:01, 199.44batch/s]

Training batches on cuda:  97%|█████████▋| 9450/9708 [00:45<00:01, 202.05batch/s]

Training batches on cuda:  98%|█████████▊| 9471/9708 [00:45<00:01, 204.17batch/s]

Training batches on cuda:  98%|█████████▊| 9492/9708 [00:45<00:01, 199.90batch/s]

Training batches on cuda:  98%|█████████▊| 9513/9708 [00:46<00:00, 202.35batch/s]

Training batches on cuda:  98%|█████████▊| 9534/9708 [00:46<00:00, 204.58batch/s]

Training batches on cuda:  98%|█████████▊| 9555/9708 [00:46<00:00, 204.53batch/s]

Training batches on cuda:  99%|█████████▊| 9576/9708 [00:46<00:00, 202.55batch/s]

Training batches on cuda:  99%|█████████▉| 9597/9708 [00:46<00:00, 202.08batch/s]

Training batches on cuda:  99%|█████████▉| 9618/9708 [00:46<00:00, 203.36batch/s]

Training batches on cuda:  99%|█████████▉| 9639/9708 [00:46<00:00, 200.45batch/s]

Training batches on cuda: 100%|█████████▉| 9660/9708 [00:46<00:00, 200.29batch/s]

Training batches on cuda: 100%|█████████▉| 9681/9708 [00:46<00:00, 199.76batch/s]

Training batches on cuda: 100%|█████████▉| 9702/9708 [00:47<00:00, 201.15batch/s]

                                                                                 
Training epochs on cuda:  20%|██        | 2/10 [02:21<06:13, 46.74s/epoch, loss=0.096, prev_loss=0.096]
Training epochs on cuda:  30%|███       | 3/10 [02:21<05:29, 47.05s/epoch, loss=0.096, prev_loss=0.096]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 10/9708 [00:00<01:41, 96.00batch/s]

Training batches on cuda:   0%|          | 32/9708 [00:00<01:24, 114.85batch/s]

Training batches on cuda:   1%|          | 53/9708 [00:00<01:12, 132.33batch/s]

Training batches on cuda:   1%|          | 74/9708 [00:00<01:05, 147.27batch/s]

Training batches on cuda:   1%|          | 96/9708 [00:00<00:58, 163.08batch/s]

Training batches on cuda:   1%|          | 118/9708 [00:00<00:54, 175.12batch/s]

Training batches on cuda:   1%|▏         | 137/9708 [00:00<00:54, 176.86batch/s]

Training batches on cuda:   2%|▏         | 160/9708 [00:00<00:50, 188.47batch/s]

Training batches on cuda:   2%|▏         | 184/9708 [00:00<00:47, 200.18batch/s]

Training batches on cuda:   2%|▏         | 208/9708 [00:01<00:45, 208.40batch/s]

Training batches on cuda:   2%|▏         | 230/9708 [00:01<00:50, 187.51batch/s]

Training batches on cuda:   3%|▎         | 253/9708 [00:01<00:47, 198.13batch/s]

Training batches on cuda:   3%|▎         | 276/9708 [00:01<00:45, 205.89batch/s]

Training batches on cuda:   3%|▎         | 299/9708 [00:01<00:44, 212.21batch/s]

Training batches on cuda:   3%|▎         | 322/9708 [00:01<00:43, 215.78batch/s]

Training batches on cuda:   4%|▎         | 344/9708 [00:01<00:43, 215.56batch/s]

Training batches on cuda:   4%|▍         | 366/9708 [00:01<00:43, 215.05batch/s]

Training batches on cuda:   4%|▍         | 388/9708 [00:01<00:43, 213.46batch/s]

Training batches on cuda:   4%|▍         | 411/9708 [00:01<00:42, 217.16batch/s]

Training batches on cuda:   4%|▍         | 434/9708 [00:02<00:42, 219.90batch/s]

Training batches on cuda:   5%|▍         | 457/9708 [00:02<00:41, 221.46batch/s]

Training batches on cuda:   5%|▍         | 480/9708 [00:02<00:42, 219.69batch/s]

Training batches on cuda:   5%|▌         | 503/9708 [00:02<00:41, 219.51batch/s]

Training batches on cuda:   5%|▌         | 527/9708 [00:02<00:40, 224.31batch/s]

Training batches on cuda:   6%|▌         | 551/9708 [00:02<00:40, 226.97batch/s]

Training batches on cuda:   6%|▌         | 574/9708 [00:02<00:41, 220.79batch/s]

Training batches on cuda:   6%|▌         | 597/9708 [00:02<00:42, 212.51batch/s]

Training batches on cuda:   6%|▋         | 620/9708 [00:02<00:42, 216.22batch/s]

Training batches on cuda:   7%|▋         | 642/9708 [00:03<00:41, 217.13batch/s]

Training batches on cuda:   7%|▋         | 664/9708 [00:03<00:43, 207.01batch/s]

Training batches on cuda:   7%|▋         | 686/9708 [00:03<00:42, 209.89batch/s]

Training batches on cuda:   7%|▋         | 710/9708 [00:03<00:41, 216.60batch/s]

Training batches on cuda:   8%|▊         | 734/9708 [00:03<00:40, 221.03batch/s]

Training batches on cuda:   8%|▊         | 758/9708 [00:03<00:39, 224.57batch/s]

Training batches on cuda:   8%|▊         | 781/9708 [00:03<00:39, 225.50batch/s]

Training batches on cuda:   8%|▊         | 804/9708 [00:03<00:39, 226.11batch/s]

Training batches on cuda:   9%|▊         | 827/9708 [00:03<00:39, 226.60batch/s]

Training batches on cuda:   9%|▉         | 850/9708 [00:03<00:39, 227.06batch/s]

Training batches on cuda:   9%|▉         | 873/9708 [00:04<00:39, 225.03batch/s]

Training batches on cuda:   9%|▉         | 896/9708 [00:04<00:38, 226.42batch/s]

Training batches on cuda:   9%|▉         | 919/9708 [00:04<00:41, 211.50batch/s]

Training batches on cuda:  10%|▉         | 941/9708 [00:04<00:42, 207.56batch/s]

Training batches on cuda:  10%|▉         | 965/9708 [00:04<00:40, 214.32batch/s]

Training batches on cuda:  10%|█         | 989/9708 [00:04<00:39, 218.54batch/s]

Training batches on cuda:  10%|█         | 1011/9708 [00:04<00:39, 218.53batch/s]

Training batches on cuda:  11%|█         | 1033/9708 [00:04<00:40, 213.49batch/s]

Training batches on cuda:  11%|█         | 1055/9708 [00:04<00:41, 209.87batch/s]

Training batches on cuda:  11%|█         | 1077/9708 [00:05<00:40, 211.90batch/s]

Training batches on cuda:  11%|█▏        | 1099/9708 [00:05<00:40, 210.27batch/s]

Training batches on cuda:  12%|█▏        | 1122/9708 [00:05<00:39, 215.45batch/s]

Training batches on cuda:  12%|█▏        | 1145/9708 [00:05<00:39, 219.56batch/s]

Training batches on cuda:  12%|█▏        | 1168/9708 [00:05<00:40, 213.35batch/s]

Training batches on cuda:  12%|█▏        | 1190/9708 [00:05<00:40, 211.07batch/s]

Training batches on cuda:  12%|█▏        | 1212/9708 [00:05<00:40, 207.72batch/s]

Training batches on cuda:  13%|█▎        | 1233/9708 [00:05<00:43, 196.88batch/s]

Training batches on cuda:  13%|█▎        | 1253/9708 [00:05<00:42, 197.26batch/s]

Training batches on cuda:  13%|█▎        | 1273/9708 [00:06<00:43, 193.71batch/s]

Training batches on cuda:  13%|█▎        | 1293/9708 [00:06<00:43, 191.82batch/s]

Training batches on cuda:  14%|█▎        | 1313/9708 [00:06<00:49, 169.17batch/s]

Training batches on cuda:  14%|█▎        | 1334/9708 [00:06<00:46, 178.73batch/s]

Training batches on cuda:  14%|█▍        | 1354/9708 [00:06<00:45, 183.17batch/s]

Training batches on cuda:  14%|█▍        | 1377/9708 [00:06<00:43, 192.81batch/s]

Training batches on cuda:  14%|█▍        | 1397/9708 [00:06<00:43, 193.00batch/s]

Training batches on cuda:  15%|█▍        | 1420/9708 [00:06<00:41, 201.33batch/s]

Training batches on cuda:  15%|█▍        | 1442/9708 [00:06<00:40, 205.94batch/s]

Training batches on cuda:  15%|█▌        | 1463/9708 [00:06<00:40, 204.52batch/s]

Training batches on cuda:  15%|█▌        | 1484/9708 [00:07<00:40, 202.84batch/s]

Training batches on cuda:  16%|█▌        | 1506/9708 [00:07<00:39, 205.22batch/s]

Training batches on cuda:  16%|█▌        | 1527/9708 [00:07<00:39, 206.50batch/s]

Training batches on cuda:  16%|█▌        | 1548/9708 [00:07<00:39, 206.65batch/s]

Training batches on cuda:  16%|█▌        | 1569/9708 [00:07<00:39, 207.59batch/s]

Training batches on cuda:  16%|█▋        | 1590/9708 [00:07<00:39, 203.80batch/s]

Training batches on cuda:  17%|█▋        | 1611/9708 [00:07<00:40, 199.10batch/s]

Training batches on cuda:  17%|█▋        | 1631/9708 [00:07<00:40, 198.07batch/s]

Training batches on cuda:  17%|█▋        | 1651/9708 [00:07<00:40, 198.54batch/s]

Training batches on cuda:  17%|█▋        | 1671/9708 [00:08<00:40, 198.82batch/s]

Training batches on cuda:  17%|█▋        | 1691/9708 [00:08<00:40, 197.78batch/s]

Training batches on cuda:  18%|█▊        | 1712/9708 [00:08<00:40, 199.54batch/s]

Training batches on cuda:  18%|█▊        | 1734/9708 [00:08<00:39, 203.54batch/s]

Training batches on cuda:  18%|█▊        | 1756/9708 [00:08<00:38, 206.19batch/s]

Training batches on cuda:  18%|█▊        | 1778/9708 [00:08<00:37, 208.81batch/s]

Training batches on cuda:  19%|█▊        | 1800/9708 [00:08<00:37, 209.82batch/s]

Training batches on cuda:  19%|█▉        | 1822/9708 [00:08<00:37, 209.59batch/s]

Training batches on cuda:  19%|█▉        | 1844/9708 [00:08<00:37, 210.40batch/s]

Training batches on cuda:  19%|█▉        | 1866/9708 [00:08<00:37, 211.32batch/s]

Training batches on cuda:  19%|█▉        | 1888/9708 [00:09<00:37, 207.76batch/s]

Training batches on cuda:  20%|█▉        | 1910/9708 [00:09<00:37, 209.74batch/s]

Training batches on cuda:  20%|█▉        | 1931/9708 [00:09<00:37, 208.78batch/s]

Training batches on cuda:  20%|██        | 1953/9708 [00:09<00:36, 209.96batch/s]

Training batches on cuda:  20%|██        | 1975/9708 [00:09<00:37, 207.89batch/s]

Training batches on cuda:  21%|██        | 1997/9708 [00:09<00:36, 209.91batch/s]

Training batches on cuda:  21%|██        | 2019/9708 [00:09<00:39, 196.81batch/s]

Training batches on cuda:  21%|██        | 2040/9708 [00:09<00:38, 198.55batch/s]

Training batches on cuda:  21%|██        | 2061/9708 [00:09<00:38, 200.43batch/s]

Training batches on cuda:  21%|██▏       | 2084/9708 [00:10<00:36, 206.26batch/s]

Training batches on cuda:  22%|██▏       | 2107/9708 [00:10<00:35, 211.18batch/s]

Training batches on cuda:  22%|██▏       | 2129/9708 [00:10<00:35, 211.09batch/s]

Training batches on cuda:  22%|██▏       | 2151/9708 [00:10<00:35, 212.46batch/s]

Training batches on cuda:  22%|██▏       | 2174/9708 [00:10<00:34, 216.00batch/s]

Training batches on cuda:  23%|██▎       | 2198/9708 [00:10<00:33, 222.21batch/s]

Training batches on cuda:  23%|██▎       | 2221/9708 [00:10<00:33, 224.38batch/s]

Training batches on cuda:  23%|██▎       | 2244/9708 [00:10<00:33, 224.27batch/s]

Training batches on cuda:  23%|██▎       | 2267/9708 [00:10<00:33, 224.33batch/s]

Training batches on cuda:  24%|██▎       | 2290/9708 [00:10<00:32, 225.18batch/s]

Training batches on cuda:  24%|██▍       | 2313/9708 [00:11<00:33, 223.85batch/s]

Training batches on cuda:  24%|██▍       | 2337/9708 [00:11<00:32, 228.17batch/s]

Training batches on cuda:  24%|██▍       | 2360/9708 [00:11<00:37, 193.46batch/s]

Training batches on cuda:  25%|██▍       | 2382/9708 [00:11<00:36, 200.51batch/s]

Training batches on cuda:  25%|██▍       | 2405/9708 [00:11<00:35, 207.35batch/s]

Training batches on cuda:  25%|██▌       | 2428/9708 [00:11<00:34, 211.53batch/s]

Training batches on cuda:  25%|██▌       | 2451/9708 [00:11<00:33, 216.15batch/s]

Training batches on cuda:  25%|██▌       | 2475/9708 [00:11<00:32, 220.72batch/s]

Training batches on cuda:  26%|██▌       | 2498/9708 [00:11<00:32, 221.52batch/s]

Training batches on cuda:  26%|██▌       | 2521/9708 [00:12<00:32, 222.23batch/s]

Training batches on cuda:  26%|██▌       | 2545/9708 [00:12<00:31, 225.76batch/s]

Training batches on cuda:  26%|██▋       | 2568/9708 [00:12<00:32, 221.61batch/s]

Training batches on cuda:  27%|██▋       | 2591/9708 [00:12<00:33, 213.08batch/s]

Training batches on cuda:  27%|██▋       | 2613/9708 [00:12<00:34, 207.05batch/s]

Training batches on cuda:  27%|██▋       | 2635/9708 [00:12<00:33, 209.28batch/s]

Training batches on cuda:  27%|██▋       | 2657/9708 [00:12<00:33, 209.67batch/s]

Training batches on cuda:  28%|██▊       | 2679/9708 [00:12<00:33, 209.62batch/s]

Training batches on cuda:  28%|██▊       | 2701/9708 [00:12<00:33, 209.05batch/s]

Training batches on cuda:  28%|██▊       | 2723/9708 [00:12<00:33, 211.40batch/s]

Training batches on cuda:  28%|██▊       | 2745/9708 [00:13<00:32, 213.21batch/s]

Training batches on cuda:  29%|██▊       | 2767/9708 [00:13<00:33, 207.87batch/s]

Training batches on cuda:  29%|██▊       | 2789/9708 [00:13<00:32, 209.91batch/s]

Training batches on cuda:  29%|██▉       | 2811/9708 [00:13<00:32, 212.82batch/s]

Training batches on cuda:  29%|██▉       | 2834/9708 [00:13<00:31, 217.64batch/s]

Training batches on cuda:  29%|██▉       | 2858/9708 [00:13<00:30, 221.67batch/s]

Training batches on cuda:  30%|██▉       | 2882/9708 [00:13<00:30, 225.03batch/s]

Training batches on cuda:  30%|██▉       | 2906/9708 [00:13<00:29, 227.88batch/s]

Training batches on cuda:  30%|███       | 2930/9708 [00:13<00:29, 228.89batch/s]

Training batches on cuda:  30%|███       | 2953/9708 [00:14<00:29, 228.55batch/s]

Training batches on cuda:  31%|███       | 2977/9708 [00:14<00:29, 230.46batch/s]

Training batches on cuda:  31%|███       | 3001/9708 [00:14<00:29, 231.25batch/s]

Training batches on cuda:  31%|███       | 3025/9708 [00:14<00:29, 229.37batch/s]

Training batches on cuda:  31%|███▏      | 3048/9708 [00:14<00:29, 229.10batch/s]

Training batches on cuda:  32%|███▏      | 3071/9708 [00:14<00:29, 227.10batch/s]

Training batches on cuda:  32%|███▏      | 3094/9708 [00:14<00:29, 227.51batch/s]

Training batches on cuda:  32%|███▏      | 3117/9708 [00:14<00:29, 226.48batch/s]

Training batches on cuda:  32%|███▏      | 3140/9708 [00:14<00:29, 220.34batch/s]

Training batches on cuda:  33%|███▎      | 3163/9708 [00:14<00:30, 216.03batch/s]

Training batches on cuda:  33%|███▎      | 3185/9708 [00:15<00:30, 212.07batch/s]

Training batches on cuda:  33%|███▎      | 3207/9708 [00:15<00:31, 207.55batch/s]

Training batches on cuda:  33%|███▎      | 3228/9708 [00:15<00:31, 204.49batch/s]

Training batches on cuda:  33%|███▎      | 3250/9708 [00:15<00:30, 208.80batch/s]

Training batches on cuda:  34%|███▎      | 3272/9708 [00:15<00:30, 209.98batch/s]

Training batches on cuda:  34%|███▍      | 3294/9708 [00:15<00:31, 206.39batch/s]

Training batches on cuda:  34%|███▍      | 3316/9708 [00:15<00:30, 208.61batch/s]

Training batches on cuda:  34%|███▍      | 3339/9708 [00:15<00:29, 212.66batch/s]

Training batches on cuda:  35%|███▍      | 3363/9708 [00:15<00:28, 219.54batch/s]

Training batches on cuda:  35%|███▍      | 3387/9708 [00:16<00:28, 223.75batch/s]

Training batches on cuda:  35%|███▌      | 3410/9708 [00:16<00:28, 224.10batch/s]

Training batches on cuda:  35%|███▌      | 3433/9708 [00:16<00:29, 215.21batch/s]

Training batches on cuda:  36%|███▌      | 3455/9708 [00:16<00:45, 138.45batch/s]

Training batches on cuda:  36%|███▌      | 3474/9708 [00:16<00:41, 149.99batch/s]

Training batches on cuda:  36%|███▌      | 3496/9708 [00:16<00:37, 164.74batch/s]

Training batches on cuda:  36%|███▌      | 3518/9708 [00:16<00:34, 177.21batch/s]

Training batches on cuda:  36%|███▋      | 3539/9708 [00:16<00:33, 185.84batch/s]

Training batches on cuda:  37%|███▋      | 3561/9708 [00:17<00:31, 193.56batch/s]

Training batches on cuda:  37%|███▋      | 3583/9708 [00:17<00:30, 198.45batch/s]

Training batches on cuda:  37%|███▋      | 3605/9708 [00:17<00:29, 204.28batch/s]

Training batches on cuda:  37%|███▋      | 3627/9708 [00:17<00:29, 204.14batch/s]

Training batches on cuda:  38%|███▊      | 3648/9708 [00:17<00:30, 202.00batch/s]

Training batches on cuda:  38%|███▊      | 3669/9708 [00:17<00:30, 199.76batch/s]

Training batches on cuda:  38%|███▊      | 3690/9708 [00:17<00:30, 199.55batch/s]

Training batches on cuda:  38%|███▊      | 3711/9708 [00:17<00:30, 198.40batch/s]

Training batches on cuda:  38%|███▊      | 3731/9708 [00:17<00:30, 197.94batch/s]

Training batches on cuda:  39%|███▊      | 3753/9708 [00:17<00:29, 202.20batch/s]

Training batches on cuda:  39%|███▉      | 3774/9708 [00:18<00:29, 202.83batch/s]

Training batches on cuda:  39%|███▉      | 3796/9708 [00:18<00:28, 205.82batch/s]

Training batches on cuda:  39%|███▉      | 3819/9708 [00:18<00:28, 210.28batch/s]

Training batches on cuda:  40%|███▉      | 3843/9708 [00:18<00:27, 216.33batch/s]

Training batches on cuda:  40%|███▉      | 3867/9708 [00:18<00:26, 219.74batch/s]

Training batches on cuda:  40%|████      | 3891/9708 [00:18<00:25, 224.29batch/s]

Training batches on cuda:  40%|████      | 3914/9708 [00:18<00:26, 222.33batch/s]

Training batches on cuda:  41%|████      | 3938/9708 [00:18<00:25, 224.83batch/s]

Training batches on cuda:  41%|████      | 3961/9708 [00:18<00:25, 225.86batch/s]

Training batches on cuda:  41%|████      | 3984/9708 [00:19<00:25, 225.82batch/s]

Training batches on cuda:  41%|████▏     | 4007/9708 [00:19<00:25, 225.04batch/s]

Training batches on cuda:  42%|████▏     | 4030/9708 [00:19<00:25, 226.35batch/s]

Training batches on cuda:  42%|████▏     | 4054/9708 [00:19<00:24, 228.22batch/s]

Training batches on cuda:  42%|████▏     | 4077/9708 [00:19<00:24, 227.66batch/s]

Training batches on cuda:  42%|████▏     | 4102/9708 [00:19<00:24, 231.88batch/s]

Training batches on cuda:  43%|████▎     | 4126/9708 [00:19<00:24, 231.90batch/s]

Training batches on cuda:  43%|████▎     | 4150/9708 [00:19<00:24, 231.43batch/s]

Training batches on cuda:  43%|████▎     | 4174/9708 [00:19<00:23, 231.52batch/s]

Training batches on cuda:  43%|████▎     | 4198/9708 [00:19<00:23, 233.16batch/s]

Training batches on cuda:  43%|████▎     | 4222/9708 [00:20<00:23, 232.97batch/s]

Training batches on cuda:  44%|████▎     | 4246/9708 [00:20<00:23, 232.26batch/s]

Training batches on cuda:  44%|████▍     | 4270/9708 [00:20<00:23, 226.69batch/s]

Training batches on cuda:  44%|████▍     | 4293/9708 [00:20<00:24, 218.27batch/s]

Training batches on cuda:  44%|████▍     | 4317/9708 [00:20<00:24, 221.71batch/s]

Training batches on cuda:  45%|████▍     | 4340/9708 [00:20<00:24, 220.40batch/s]

Training batches on cuda:  45%|████▍     | 4363/9708 [00:20<00:24, 218.01batch/s]

Training batches on cuda:  45%|████▌     | 4385/9708 [00:20<00:24, 216.51batch/s]

Training batches on cuda:  45%|████▌     | 4408/9708 [00:20<00:24, 217.65batch/s]

Training batches on cuda:  46%|████▌     | 4430/9708 [00:20<00:24, 216.72batch/s]

Training batches on cuda:  46%|████▌     | 4452/9708 [00:21<00:24, 211.15batch/s]

Training batches on cuda:  46%|████▌     | 4474/9708 [00:21<00:24, 211.92batch/s]

Training batches on cuda:  46%|████▋     | 4496/9708 [00:21<00:25, 208.02batch/s]

Training batches on cuda:  47%|████▋     | 4517/9708 [00:21<00:24, 208.39batch/s]

Training batches on cuda:  47%|████▋     | 4538/9708 [00:21<00:24, 207.27batch/s]

Training batches on cuda:  47%|████▋     | 4559/9708 [00:21<00:25, 202.92batch/s]

Training batches on cuda:  47%|████▋     | 4580/9708 [00:21<00:25, 201.48batch/s]

Training batches on cuda:  47%|████▋     | 4601/9708 [00:21<00:25, 200.06batch/s]

Training batches on cuda:  48%|████▊     | 4622/9708 [00:21<00:25, 196.46batch/s]

Training batches on cuda:  48%|████▊     | 4645/9708 [00:22<00:24, 205.15batch/s]

Training batches on cuda:  48%|████▊     | 4670/9708 [00:22<00:23, 215.15batch/s]

Training batches on cuda:  48%|████▊     | 4695/9708 [00:22<00:22, 222.72batch/s]

Training batches on cuda:  49%|████▊     | 4720/9708 [00:22<00:21, 228.56batch/s]

Training batches on cuda:  49%|████▉     | 4745/9708 [00:22<00:21, 232.73batch/s]

Training batches on cuda:  49%|████▉     | 4772/9708 [00:22<00:20, 240.95batch/s]

Training batches on cuda:  49%|████▉     | 4799/9708 [00:22<00:19, 248.29batch/s]

Training batches on cuda:  50%|████▉     | 4826/9708 [00:22<00:19, 253.45batch/s]

Training batches on cuda:  50%|█████     | 4854/9708 [00:22<00:18, 258.30batch/s]

Training batches on cuda:  50%|█████     | 4882/9708 [00:22<00:18, 262.17batch/s]

Training batches on cuda:  51%|█████     | 4910/9708 [00:23<00:18, 266.07batch/s]

Training batches on cuda:  51%|█████     | 4938/9708 [00:23<00:17, 268.11batch/s]

Training batches on cuda:  51%|█████     | 4966/9708 [00:23<00:17, 270.34batch/s]

Training batches on cuda:  51%|█████▏    | 4994/9708 [00:23<00:17, 272.27batch/s]

Training batches on cuda:  52%|█████▏    | 5022/9708 [00:23<00:17, 272.40batch/s]

Training batches on cuda:  52%|█████▏    | 5050/9708 [00:23<00:17, 271.67batch/s]

Training batches on cuda:  52%|█████▏    | 5078/9708 [00:23<00:17, 269.94batch/s]

Training batches on cuda:  53%|█████▎    | 5106/9708 [00:23<00:17, 270.04batch/s]

Training batches on cuda:  53%|█████▎    | 5134/9708 [00:23<00:17, 268.23batch/s]

Training batches on cuda:  53%|█████▎    | 5162/9708 [00:24<00:16, 270.11batch/s]

Training batches on cuda:  53%|█████▎    | 5190/9708 [00:24<00:16, 271.98batch/s]

Training batches on cuda:  54%|█████▎    | 5218/9708 [00:24<00:16, 273.13batch/s]

Training batches on cuda:  54%|█████▍    | 5246/9708 [00:24<00:16, 271.99batch/s]

Training batches on cuda:  54%|█████▍    | 5274/9708 [00:24<00:16, 271.88batch/s]

Training batches on cuda:  55%|█████▍    | 5302/9708 [00:24<00:16, 270.51batch/s]

Training batches on cuda:  55%|█████▍    | 5330/9708 [00:24<00:16, 268.03batch/s]

Training batches on cuda:  55%|█████▌    | 5357/9708 [00:24<00:16, 268.30batch/s]

Training batches on cuda:  55%|█████▌    | 5385/9708 [00:24<00:16, 269.18batch/s]

Training batches on cuda:  56%|█████▌    | 5413/9708 [00:24<00:15, 270.18batch/s]

Training batches on cuda:  56%|█████▌    | 5441/9708 [00:25<00:15, 269.24batch/s]

Training batches on cuda:  56%|█████▋    | 5469/9708 [00:25<00:15, 270.34batch/s]

Training batches on cuda:  57%|█████▋    | 5497/9708 [00:25<00:15, 271.34batch/s]

Training batches on cuda:  57%|█████▋    | 5525/9708 [00:25<00:15, 271.46batch/s]

Training batches on cuda:  57%|█████▋    | 5553/9708 [00:25<00:15, 272.01batch/s]

Training batches on cuda:  57%|█████▋    | 5581/9708 [00:25<00:15, 273.69batch/s]

Training batches on cuda:  58%|█████▊    | 5609/9708 [00:25<00:14, 273.54batch/s]

Training batches on cuda:  58%|█████▊    | 5637/9708 [00:25<00:14, 272.94batch/s]

Training batches on cuda:  58%|█████▊    | 5665/9708 [00:25<00:14, 273.17batch/s]

Training batches on cuda:  59%|█████▊    | 5693/9708 [00:25<00:14, 272.86batch/s]

Training batches on cuda:  59%|█████▉    | 5721/9708 [00:26<00:14, 271.68batch/s]

Training batches on cuda:  59%|█████▉    | 5749/9708 [00:26<00:14, 273.17batch/s]

Training batches on cuda:  60%|█████▉    | 5777/9708 [00:26<00:14, 273.71batch/s]

Training batches on cuda:  60%|█████▉    | 5805/9708 [00:26<00:14, 274.18batch/s]

Training batches on cuda:  60%|██████    | 5833/9708 [00:26<00:17, 223.46batch/s]

Training batches on cuda:  60%|██████    | 5860/9708 [00:26<00:16, 235.60batch/s]

Training batches on cuda:  61%|██████    | 5886/9708 [00:26<00:15, 239.63batch/s]

Training batches on cuda:  61%|██████    | 5911/9708 [00:26<00:16, 230.84batch/s]

Training batches on cuda:  61%|██████    | 5935/9708 [00:26<00:16, 225.32batch/s]

Training batches on cuda:  61%|██████▏   | 5959/9708 [00:27<00:16, 222.09batch/s]

Training batches on cuda:  62%|██████▏   | 5982/9708 [00:27<00:17, 219.07batch/s]

Training batches on cuda:  62%|██████▏   | 6005/9708 [00:27<00:16, 220.95batch/s]

Training batches on cuda:  62%|██████▏   | 6028/9708 [00:27<00:16, 218.72batch/s]

Training batches on cuda:  62%|██████▏   | 6051/9708 [00:27<00:17, 214.36batch/s]

Training batches on cuda:  63%|██████▎   | 6073/9708 [00:27<00:18, 201.11batch/s]

Training batches on cuda:  63%|██████▎   | 6094/9708 [00:27<00:18, 198.15batch/s]

Training batches on cuda:  63%|██████▎   | 6116/9708 [00:27<00:17, 202.88batch/s]

Training batches on cuda:  63%|██████▎   | 6137/9708 [00:27<00:17, 203.51batch/s]

Training batches on cuda:  63%|██████▎   | 6159/9708 [00:28<00:17, 205.50batch/s]

Training batches on cuda:  64%|██████▎   | 6180/9708 [00:28<00:17, 206.72batch/s]

Training batches on cuda:  64%|██████▍   | 6201/9708 [00:28<00:16, 207.68batch/s]

Training batches on cuda:  64%|██████▍   | 6222/9708 [00:28<00:17, 200.31batch/s]

Training batches on cuda:  64%|██████▍   | 6243/9708 [00:28<00:17, 202.23batch/s]

Training batches on cuda:  65%|██████▍   | 6265/9708 [00:28<00:16, 204.80batch/s]

Training batches on cuda:  65%|██████▍   | 6287/9708 [00:28<00:16, 206.26batch/s]

Training batches on cuda:  65%|██████▍   | 6308/9708 [00:28<00:17, 199.16batch/s]

Training batches on cuda:  65%|██████▌   | 6328/9708 [00:28<00:17, 191.31batch/s]

Training batches on cuda:  65%|██████▌   | 6348/9708 [00:29<00:17, 190.60batch/s]

Training batches on cuda:  66%|██████▌   | 6368/9708 [00:29<00:17, 189.89batch/s]

Training batches on cuda:  66%|██████▌   | 6388/9708 [00:29<00:17, 189.18batch/s]

Training batches on cuda:  66%|██████▌   | 6407/9708 [00:29<00:17, 188.42batch/s]

Training batches on cuda:  66%|██████▌   | 6426/9708 [00:29<00:17, 188.15batch/s]

Training batches on cuda:  66%|██████▋   | 6446/9708 [00:29<00:17, 189.73batch/s]

Training batches on cuda:  67%|██████▋   | 6468/9708 [00:29<00:16, 197.23batch/s]

Training batches on cuda:  67%|██████▋   | 6490/9708 [00:29<00:15, 202.43batch/s]

Training batches on cuda:  67%|██████▋   | 6512/9708 [00:29<00:15, 206.07batch/s]

Training batches on cuda:  67%|██████▋   | 6533/9708 [00:29<00:15, 205.16batch/s]

Training batches on cuda:  68%|██████▊   | 6554/9708 [00:30<00:15, 206.56batch/s]

Training batches on cuda:  68%|██████▊   | 6576/9708 [00:30<00:15, 208.20batch/s]

Training batches on cuda:  68%|██████▊   | 6597/9708 [00:30<00:14, 208.55batch/s]

Training batches on cuda:  68%|██████▊   | 6619/9708 [00:30<00:14, 209.50batch/s]

Training batches on cuda:  68%|██████▊   | 6640/9708 [00:30<00:15, 199.25batch/s]

Training batches on cuda:  69%|██████▊   | 6661/9708 [00:30<00:15, 200.97batch/s]

Training batches on cuda:  69%|██████▉   | 6682/9708 [00:30<00:15, 200.57batch/s]

Training batches on cuda:  69%|██████▉   | 6703/9708 [00:30<00:15, 193.55batch/s]

Training batches on cuda:  69%|██████▉   | 6723/9708 [00:30<00:15, 191.98batch/s]

Training batches on cuda:  69%|██████▉   | 6743/9708 [00:31<00:15, 189.78batch/s]

Training batches on cuda:  70%|██████▉   | 6763/9708 [00:31<00:16, 176.83batch/s]

Training batches on cuda:  70%|██████▉   | 6781/9708 [00:31<00:16, 177.01batch/s]

Training batches on cuda:  70%|███████   | 6802/9708 [00:31<00:15, 183.18batch/s]

Training batches on cuda:  70%|███████   | 6821/9708 [00:31<00:21, 131.94batch/s]

Training batches on cuda:  70%|███████   | 6842/9708 [00:31<00:19, 148.23batch/s]

Training batches on cuda:  71%|███████   | 6860/9708 [00:31<00:18, 155.51batch/s]

Training batches on cuda:  71%|███████   | 6878/9708 [00:31<00:17, 161.53batch/s]

Training batches on cuda:  71%|███████   | 6900/9708 [00:31<00:16, 174.13batch/s]

Training batches on cuda:  71%|███████▏  | 6920/9708 [00:32<00:15, 180.44batch/s]

Training batches on cuda:  71%|███████▏  | 6939/9708 [00:32<00:15, 177.83batch/s]

Training batches on cuda:  72%|███████▏  | 6959/9708 [00:32<00:14, 183.92batch/s]

Training batches on cuda:  72%|███████▏  | 6981/9708 [00:32<00:14, 191.53batch/s]

Training batches on cuda:  72%|███████▏  | 7001/9708 [00:32<00:14, 187.99batch/s]

Training batches on cuda:  72%|███████▏  | 7021/9708 [00:32<00:14, 188.96batch/s]

Training batches on cuda:  73%|███████▎  | 7042/9708 [00:32<00:13, 192.84batch/s]

Training batches on cuda:  73%|███████▎  | 7066/9708 [00:32<00:12, 203.62batch/s]

Training batches on cuda:  73%|███████▎  | 7087/9708 [00:32<00:12, 204.94batch/s]

Training batches on cuda:  73%|███████▎  | 7108/9708 [00:33<00:13, 195.42batch/s]

Training batches on cuda:  73%|███████▎  | 7128/9708 [00:33<00:13, 190.76batch/s]

Training batches on cuda:  74%|███████▎  | 7148/9708 [00:33<00:13, 192.69batch/s]

Training batches on cuda:  74%|███████▍  | 7168/9708 [00:33<00:13, 190.81batch/s]

Training batches on cuda:  74%|███████▍  | 7188/9708 [00:33<00:13, 189.60batch/s]

Training batches on cuda:  74%|███████▍  | 7209/9708 [00:33<00:12, 195.00batch/s]

Training batches on cuda:  74%|███████▍  | 7229/9708 [00:33<00:12, 192.89batch/s]

Training batches on cuda:  75%|███████▍  | 7250/9708 [00:33<00:12, 195.79batch/s]

Training batches on cuda:  75%|███████▍  | 7271/9708 [00:33<00:12, 197.40batch/s]

Training batches on cuda:  75%|███████▌  | 7291/9708 [00:34<00:12, 190.90batch/s]

Training batches on cuda:  75%|███████▌  | 7313/9708 [00:34<00:12, 198.28batch/s]

Training batches on cuda:  76%|███████▌  | 7335/9708 [00:34<00:11, 203.37batch/s]

Training batches on cuda:  76%|███████▌  | 7356/9708 [00:34<00:11, 196.35batch/s]

Training batches on cuda:  76%|███████▌  | 7378/9708 [00:34<00:11, 201.55batch/s]

Training batches on cuda:  76%|███████▌  | 7399/9708 [00:34<00:11, 203.66batch/s]

Training batches on cuda:  76%|███████▋  | 7421/9708 [00:34<00:11, 207.38batch/s]

Training batches on cuda:  77%|███████▋  | 7442/9708 [00:34<00:11, 203.55batch/s]

Training batches on cuda:  77%|███████▋  | 7465/9708 [00:34<00:10, 209.21batch/s]

Training batches on cuda:  77%|███████▋  | 7488/9708 [00:34<00:10, 214.50batch/s]

Training batches on cuda:  77%|███████▋  | 7512/9708 [00:35<00:10, 219.53batch/s]

Training batches on cuda:  78%|███████▊  | 7535/9708 [00:35<00:09, 220.41batch/s]

Training batches on cuda:  78%|███████▊  | 7558/9708 [00:35<00:10, 214.09batch/s]

Training batches on cuda:  78%|███████▊  | 7580/9708 [00:35<00:10, 210.18batch/s]

Training batches on cuda:  78%|███████▊  | 7602/9708 [00:35<00:10, 204.53batch/s]

Training batches on cuda:  79%|███████▊  | 7623/9708 [00:35<00:10, 203.77batch/s]

Training batches on cuda:  79%|███████▊  | 7644/9708 [00:35<00:10, 204.63batch/s]

Training batches on cuda:  79%|███████▉  | 7666/9708 [00:35<00:09, 206.70batch/s]

Training batches on cuda:  79%|███████▉  | 7688/9708 [00:35<00:09, 209.58batch/s]

Training batches on cuda:  79%|███████▉  | 7710/9708 [00:36<00:09, 210.56batch/s]

Training batches on cuda:  80%|███████▉  | 7732/9708 [00:36<00:09, 211.97batch/s]

Training batches on cuda:  80%|███████▉  | 7755/9708 [00:36<00:09, 215.98batch/s]

Training batches on cuda:  80%|████████  | 7780/9708 [00:36<00:08, 223.07batch/s]

Training batches on cuda:  80%|████████  | 7804/9708 [00:36<00:08, 227.19batch/s]

Training batches on cuda:  81%|████████  | 7827/9708 [00:36<00:10, 171.17batch/s]

Training batches on cuda:  81%|████████  | 7852/9708 [00:36<00:09, 188.92batch/s]

Training batches on cuda:  81%|████████  | 7876/9708 [00:36<00:09, 201.31batch/s]

Training batches on cuda:  81%|████████▏ | 7900/9708 [00:36<00:08, 209.98batch/s]

Training batches on cuda:  82%|████████▏ | 7924/9708 [00:37<00:08, 217.11batch/s]

Training batches on cuda:  82%|████████▏ | 7947/9708 [00:37<00:07, 220.18batch/s]

Training batches on cuda:  82%|████████▏ | 7970/9708 [00:37<00:07, 222.67batch/s]

Training batches on cuda:  82%|████████▏ | 7993/9708 [00:37<00:07, 223.66batch/s]

Training batches on cuda:  83%|████████▎ | 8016/9708 [00:37<00:07, 224.59batch/s]

Training batches on cuda:  83%|████████▎ | 8039/9708 [00:37<00:07, 221.24batch/s]

Training batches on cuda:  83%|████████▎ | 8062/9708 [00:37<00:07, 216.06batch/s]

Training batches on cuda:  83%|████████▎ | 8085/9708 [00:37<00:07, 217.49batch/s]

Training batches on cuda:  84%|████████▎ | 8108/9708 [00:37<00:07, 220.42batch/s]

Training batches on cuda:  84%|████████▍ | 8132/9708 [00:37<00:07, 223.82batch/s]

Training batches on cuda:  84%|████████▍ | 8155/9708 [00:38<00:06, 224.54batch/s]

Training batches on cuda:  84%|████████▍ | 8178/9708 [00:38<00:06, 220.83batch/s]

Training batches on cuda:  84%|████████▍ | 8201/9708 [00:38<00:07, 215.17batch/s]

Training batches on cuda:  85%|████████▍ | 8223/9708 [00:38<00:06, 214.84batch/s]

Training batches on cuda:  85%|████████▍ | 8245/9708 [00:38<00:06, 212.16batch/s]

Training batches on cuda:  85%|████████▌ | 8267/9708 [00:38<00:06, 210.43batch/s]

Training batches on cuda:  85%|████████▌ | 8289/9708 [00:38<00:06, 206.87batch/s]

Training batches on cuda:  86%|████████▌ | 8311/9708 [00:38<00:06, 208.97batch/s]

Training batches on cuda:  86%|████████▌ | 8333/9708 [00:38<00:06, 209.77batch/s]

Training batches on cuda:  86%|████████▌ | 8354/9708 [00:39<00:06, 197.42batch/s]

Training batches on cuda:  86%|████████▋ | 8378/9708 [00:39<00:06, 206.23batch/s]

Training batches on cuda:  87%|████████▋ | 8401/9708 [00:39<00:06, 211.19batch/s]

Training batches on cuda:  87%|████████▋ | 8424/9708 [00:39<00:05, 216.19batch/s]

Training batches on cuda:  87%|████████▋ | 8446/9708 [00:39<00:05, 216.38batch/s]

Training batches on cuda:  87%|████████▋ | 8468/9708 [00:39<00:05, 215.61batch/s]

Training batches on cuda:  87%|████████▋ | 8490/9708 [00:39<00:05, 214.28batch/s]

Training batches on cuda:  88%|████████▊ | 8513/9708 [00:39<00:05, 217.31batch/s]

Training batches on cuda:  88%|████████▊ | 8536/9708 [00:39<00:05, 219.35batch/s]

Training batches on cuda:  88%|████████▊ | 8558/9708 [00:39<00:05, 217.88batch/s]

Training batches on cuda:  88%|████████▊ | 8581/9708 [00:40<00:05, 218.74batch/s]

Training batches on cuda:  89%|████████▊ | 8603/9708 [00:40<00:05, 215.42batch/s]

Training batches on cuda:  89%|████████▉ | 8625/9708 [00:40<00:05, 207.99batch/s]

Training batches on cuda:  89%|████████▉ | 8647/9708 [00:40<00:05, 211.42batch/s]

Training batches on cuda:  89%|████████▉ | 8669/9708 [00:40<00:04, 209.76batch/s]

Training batches on cuda:  90%|████████▉ | 8691/9708 [00:40<00:04, 211.41batch/s]

Training batches on cuda:  90%|████████▉ | 8713/9708 [00:40<00:04, 208.95batch/s]

Training batches on cuda:  90%|████████▉ | 8735/9708 [00:40<00:04, 209.75batch/s]

Training batches on cuda:  90%|█████████ | 8758/9708 [00:40<00:04, 215.37batch/s]

Training batches on cuda:  90%|█████████ | 8780/9708 [00:41<00:04, 215.29batch/s]

Training batches on cuda:  91%|█████████ | 8802/9708 [00:41<00:04, 212.92batch/s]

Training batches on cuda:  91%|█████████ | 8824/9708 [00:41<00:04, 210.53batch/s]

Training batches on cuda:  91%|█████████ | 8846/9708 [00:41<00:04, 211.17batch/s]

Training batches on cuda:  91%|█████████▏| 8868/9708 [00:41<00:03, 212.35batch/s]

Training batches on cuda:  92%|█████████▏| 8890/9708 [00:41<00:04, 185.10batch/s]

Training batches on cuda:  92%|█████████▏| 8912/9708 [00:41<00:04, 193.60batch/s]

Training batches on cuda:  92%|█████████▏| 8934/9708 [00:41<00:03, 199.87batch/s]

Training batches on cuda:  92%|█████████▏| 8956/9708 [00:41<00:03, 204.34batch/s]

Training batches on cuda:  93%|█████████▎| 8980/9708 [00:41<00:03, 212.08batch/s]

Training batches on cuda:  93%|█████████▎| 9005/9708 [00:42<00:03, 220.06batch/s]

Training batches on cuda:  93%|█████████▎| 9029/9708 [00:42<00:03, 223.41batch/s]

Training batches on cuda:  93%|█████████▎| 9052/9708 [00:42<00:03, 216.79batch/s]

Training batches on cuda:  93%|█████████▎| 9074/9708 [00:42<00:03, 203.68batch/s]

Training batches on cuda:  94%|█████████▎| 9096/9708 [00:42<00:02, 206.03batch/s]

Training batches on cuda:  94%|█████████▍| 9120/9708 [00:42<00:02, 213.00batch/s]

Training batches on cuda:  94%|█████████▍| 9144/9708 [00:42<00:02, 218.27batch/s]

Training batches on cuda:  94%|█████████▍| 9168/9708 [00:42<00:02, 221.81batch/s]

Training batches on cuda:  95%|█████████▍| 9191/9708 [00:42<00:02, 216.26batch/s]

Training batches on cuda:  95%|█████████▍| 9215/9708 [00:43<00:02, 220.67batch/s]

Training batches on cuda:  95%|█████████▌| 9239/9708 [00:43<00:02, 224.02batch/s]

Training batches on cuda:  95%|█████████▌| 9262/9708 [00:43<00:01, 225.77batch/s]

Training batches on cuda:  96%|█████████▌| 9285/9708 [00:43<00:01, 214.94batch/s]

Training batches on cuda:  96%|█████████▌| 9307/9708 [00:43<00:01, 215.02batch/s]

Training batches on cuda:  96%|█████████▌| 9329/9708 [00:43<00:01, 208.37batch/s]

Training batches on cuda:  96%|█████████▋| 9351/9708 [00:43<00:01, 209.93batch/s]

Training batches on cuda:  97%|█████████▋| 9373/9708 [00:43<00:01, 209.56batch/s]

Training batches on cuda:  97%|█████████▋| 9395/9708 [00:43<00:01, 210.35batch/s]

Training batches on cuda:  97%|█████████▋| 9417/9708 [00:44<00:01, 211.01batch/s]

Training batches on cuda:  97%|█████████▋| 9440/9708 [00:44<00:01, 216.06batch/s]

Training batches on cuda:  97%|█████████▋| 9464/9708 [00:44<00:01, 220.06batch/s]

Training batches on cuda:  98%|█████████▊| 9487/9708 [00:44<00:00, 221.82batch/s]

Training batches on cuda:  98%|█████████▊| 9511/9708 [00:44<00:00, 225.71batch/s]

Training batches on cuda:  98%|█████████▊| 9535/9708 [00:44<00:00, 227.51batch/s]

Training batches on cuda:  98%|█████████▊| 9558/9708 [00:44<00:00, 227.98batch/s]

Training batches on cuda:  99%|█████████▊| 9581/9708 [00:44<00:00, 218.60batch/s]

Training batches on cuda:  99%|█████████▉| 9603/9708 [00:44<00:00, 213.72batch/s]

Training batches on cuda:  99%|█████████▉| 9625/9708 [00:44<00:00, 214.72batch/s]

Training batches on cuda:  99%|█████████▉| 9649/9708 [00:45<00:00, 219.31batch/s]

Training batches on cuda: 100%|█████████▉| 9672/9708 [00:45<00:00, 219.47batch/s]

Training batches on cuda: 100%|█████████▉| 9694/9708 [00:45<00:00, 216.12batch/s]

                                                                                 
Training epochs on cuda:  30%|███       | 3/10 [03:06<05:29, 47.05s/epoch, loss=0.096, prev_loss=0.096]
Training epochs on cuda:  40%|████      | 4/10 [03:06<04:39, 46.62s/epoch, loss=0.096, prev_loss=0.096]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 16/9708 [00:00<01:02, 153.94batch/s]

Training batches on cuda:   0%|          | 38/9708 [00:00<00:57, 167.54batch/s]

Training batches on cuda:   1%|          | 60/9708 [00:00<00:53, 179.16batch/s]

Training batches on cuda:   1%|          | 80/9708 [00:00<00:52, 184.71batch/s]

Training batches on cuda:   1%|          | 102/9708 [00:00<00:49, 194.02batch/s]

Training batches on cuda:   1%|▏         | 126/9708 [00:00<00:46, 205.29batch/s]

Training batches on cuda:   2%|▏         | 148/9708 [00:00<00:46, 206.66batch/s]

Training batches on cuda:   2%|▏         | 168/9708 [00:00<00:47, 202.10batch/s]

Training batches on cuda:   2%|▏         | 188/9708 [00:00<00:55, 172.27batch/s]

Training batches on cuda:   2%|▏         | 207/9708 [00:01<00:53, 176.47batch/s]

Training batches on cuda:   2%|▏         | 227/9708 [00:01<00:52, 181.32batch/s]

Training batches on cuda:   3%|▎         | 247/9708 [00:01<00:50, 186.00batch/s]

Training batches on cuda:   3%|▎         | 266/9708 [00:01<00:50, 185.77batch/s]

Training batches on cuda:   3%|▎         | 285/9708 [00:01<00:50, 184.78batch/s]

Training batches on cuda:   3%|▎         | 304/9708 [00:01<00:50, 184.75batch/s]

Training batches on cuda:   3%|▎         | 323/9708 [00:01<00:51, 183.86batch/s]

Training batches on cuda:   4%|▎         | 343/9708 [00:01<00:49, 187.36batch/s]

Training batches on cuda:   4%|▎         | 363/9708 [00:01<00:49, 190.40batch/s]

Training batches on cuda:   4%|▍         | 384/9708 [00:02<00:47, 194.38batch/s]

Training batches on cuda:   4%|▍         | 404/9708 [00:02<00:47, 195.71batch/s]

Training batches on cuda:   4%|▍         | 425/9708 [00:02<00:46, 197.68batch/s]

Training batches on cuda:   5%|▍         | 447/9708 [00:02<00:45, 201.56batch/s]

Training batches on cuda:   5%|▍         | 468/9708 [00:02<00:45, 203.42batch/s]

Training batches on cuda:   5%|▌         | 489/9708 [00:02<00:45, 203.02batch/s]

Training batches on cuda:   5%|▌         | 511/9708 [00:02<00:44, 205.55batch/s]

Training batches on cuda:   5%|▌         | 533/9708 [00:02<00:43, 209.11batch/s]

Training batches on cuda:   6%|▌         | 554/9708 [00:02<00:43, 209.03batch/s]

Training batches on cuda:   6%|▌         | 575/9708 [00:02<00:44, 205.89batch/s]

Training batches on cuda:   6%|▌         | 596/9708 [00:03<00:44, 206.29batch/s]

Training batches on cuda:   6%|▋         | 618/9708 [00:03<00:43, 207.34batch/s]

Training batches on cuda:   7%|▋         | 639/9708 [00:03<00:43, 207.12batch/s]

Training batches on cuda:   7%|▋         | 661/9708 [00:03<00:43, 210.09batch/s]

Training batches on cuda:   7%|▋         | 683/9708 [00:03<00:43, 209.81batch/s]

Training batches on cuda:   7%|▋         | 704/9708 [00:03<00:43, 209.19batch/s]

Training batches on cuda:   7%|▋         | 725/9708 [00:03<00:43, 205.14batch/s]

Training batches on cuda:   8%|▊         | 746/9708 [00:03<00:43, 205.78batch/s]

Training batches on cuda:   8%|▊         | 768/9708 [00:03<00:42, 209.70batch/s]

Training batches on cuda:   8%|▊         | 792/9708 [00:03<00:41, 215.49batch/s]

Training batches on cuda:   8%|▊         | 816/9708 [00:04<00:40, 220.25batch/s]

Training batches on cuda:   9%|▊         | 840/9708 [00:04<00:39, 224.33batch/s]

Training batches on cuda:   9%|▉         | 863/9708 [00:04<00:39, 225.19batch/s]

Training batches on cuda:   9%|▉         | 886/9708 [00:04<00:39, 226.10batch/s]

Training batches on cuda:   9%|▉         | 910/9708 [00:04<00:38, 228.58batch/s]

Training batches on cuda:  10%|▉         | 934/9708 [00:04<00:38, 229.82batch/s]

Training batches on cuda:  10%|▉         | 958/9708 [00:04<00:37, 230.45batch/s]

Training batches on cuda:  10%|█         | 982/9708 [00:04<00:37, 230.78batch/s]

Training batches on cuda:  10%|█         | 1006/9708 [00:04<00:38, 228.06batch/s]

Training batches on cuda:  11%|█         | 1030/9708 [00:04<00:37, 228.78batch/s]

Training batches on cuda:  11%|█         | 1054/9708 [00:05<00:37, 231.24batch/s]

Training batches on cuda:  11%|█         | 1078/9708 [00:05<00:37, 232.75batch/s]

Training batches on cuda:  11%|█▏        | 1102/9708 [00:05<00:36, 233.81batch/s]

Training batches on cuda:  12%|█▏        | 1126/9708 [00:05<00:36, 234.41batch/s]

Training batches on cuda:  12%|█▏        | 1150/9708 [00:05<00:36, 235.36batch/s]

Training batches on cuda:  12%|█▏        | 1174/9708 [00:05<00:36, 234.51batch/s]

Training batches on cuda:  12%|█▏        | 1198/9708 [00:05<00:36, 232.60batch/s]

Training batches on cuda:  13%|█▎        | 1222/9708 [00:05<00:36, 234.50batch/s]

Training batches on cuda:  13%|█▎        | 1246/9708 [00:05<00:36, 233.91batch/s]

Training batches on cuda:  13%|█▎        | 1270/9708 [00:06<00:43, 192.71batch/s]

Training batches on cuda:  13%|█▎        | 1291/9708 [00:06<00:43, 194.65batch/s]

Training batches on cuda:  14%|█▎        | 1312/9708 [00:06<00:44, 190.48batch/s]

Training batches on cuda:  14%|█▎        | 1333/9708 [00:06<00:43, 193.98batch/s]

Training batches on cuda:  14%|█▍        | 1356/9708 [00:06<00:41, 201.15batch/s]

Training batches on cuda:  14%|█▍        | 1377/9708 [00:06<00:40, 203.28batch/s]

Training batches on cuda:  14%|█▍        | 1399/9708 [00:06<00:40, 205.74batch/s]

Training batches on cuda:  15%|█▍        | 1420/9708 [00:06<00:42, 193.92batch/s]

Training batches on cuda:  15%|█▍        | 1441/9708 [00:06<00:42, 196.61batch/s]

Training batches on cuda:  15%|█▌        | 1461/9708 [00:07<00:41, 196.46batch/s]

Training batches on cuda:  15%|█▌        | 1481/9708 [00:07<00:42, 194.66batch/s]

Training batches on cuda:  15%|█▌        | 1501/9708 [00:07<00:42, 194.83batch/s]

Training batches on cuda:  16%|█▌        | 1523/9708 [00:07<00:40, 200.25batch/s]

Training batches on cuda:  16%|█▌        | 1544/9708 [00:07<00:40, 199.98batch/s]

Training batches on cuda:  16%|█▌        | 1567/9708 [00:07<00:39, 207.76batch/s]

Training batches on cuda:  16%|█▋        | 1591/9708 [00:07<00:37, 213.93batch/s]

Training batches on cuda:  17%|█▋        | 1613/9708 [00:07<00:37, 213.70batch/s]

Training batches on cuda:  17%|█▋        | 1635/9708 [00:07<00:37, 214.56batch/s]

Training batches on cuda:  17%|█▋        | 1657/9708 [00:07<00:37, 214.62batch/s]

Training batches on cuda:  17%|█▋        | 1680/9708 [00:08<00:36, 218.26batch/s]

Training batches on cuda:  18%|█▊        | 1704/9708 [00:08<00:36, 222.08batch/s]

Training batches on cuda:  18%|█▊        | 1728/9708 [00:08<00:35, 225.86batch/s]

Training batches on cuda:  18%|█▊        | 1752/9708 [00:08<00:34, 228.82batch/s]

Training batches on cuda:  18%|█▊        | 1776/9708 [00:08<00:34, 229.59batch/s]

Training batches on cuda:  19%|█▊        | 1799/9708 [00:08<00:34, 229.49batch/s]

Training batches on cuda:  19%|█▉        | 1822/9708 [00:08<00:34, 226.43batch/s]

Training batches on cuda:  19%|█▉        | 1845/9708 [00:08<00:35, 224.23batch/s]

Training batches on cuda:  19%|█▉        | 1868/9708 [00:08<00:36, 217.15batch/s]

Training batches on cuda:  19%|█▉        | 1890/9708 [00:09<00:36, 216.03batch/s]

Training batches on cuda:  20%|█▉        | 1912/9708 [00:09<00:36, 215.83batch/s]

Training batches on cuda:  20%|█▉        | 1934/9708 [00:09<00:36, 211.31batch/s]

Training batches on cuda:  20%|██        | 1956/9708 [00:09<00:36, 210.46batch/s]

Training batches on cuda:  20%|██        | 1978/9708 [00:09<00:37, 208.21batch/s]

Training batches on cuda:  21%|██        | 2000/9708 [00:09<00:36, 210.29batch/s]

Training batches on cuda:  21%|██        | 2022/9708 [00:09<00:36, 211.82batch/s]

Training batches on cuda:  21%|██        | 2044/9708 [00:09<00:35, 212.93batch/s]

Training batches on cuda:  21%|██▏       | 2067/9708 [00:09<00:35, 215.48batch/s]

Training batches on cuda:  22%|██▏       | 2091/9708 [00:09<00:34, 222.28batch/s]

Training batches on cuda:  22%|██▏       | 2115/9708 [00:10<00:33, 226.13batch/s]

Training batches on cuda:  22%|██▏       | 2139/9708 [00:10<00:33, 228.59batch/s]

Training batches on cuda:  22%|██▏       | 2163/9708 [00:10<00:32, 230.56batch/s]

Training batches on cuda:  23%|██▎       | 2187/9708 [00:10<00:32, 230.59batch/s]

Training batches on cuda:  23%|██▎       | 2211/9708 [00:10<00:32, 231.70batch/s]

Training batches on cuda:  23%|██▎       | 2235/9708 [00:10<00:32, 233.35batch/s]

Training batches on cuda:  23%|██▎       | 2259/9708 [00:10<00:31, 233.46batch/s]

Training batches on cuda:  24%|██▎       | 2283/9708 [00:10<00:31, 232.98batch/s]

Training batches on cuda:  24%|██▍       | 2307/9708 [00:10<00:31, 233.91batch/s]

Training batches on cuda:  24%|██▍       | 2331/9708 [00:11<00:31, 232.87batch/s]

Training batches on cuda:  24%|██▍       | 2355/9708 [00:11<00:42, 173.70batch/s]

Training batches on cuda:  25%|██▍       | 2381/9708 [00:11<00:38, 191.76batch/s]

Training batches on cuda:  25%|██▍       | 2409/9708 [00:11<00:34, 210.05batch/s]

Training batches on cuda:  25%|██▌       | 2437/9708 [00:11<00:32, 225.80batch/s]

Training batches on cuda:  25%|██▌       | 2465/9708 [00:11<00:30, 238.84batch/s]

Training batches on cuda:  26%|██▌       | 2494/9708 [00:11<00:28, 250.31batch/s]

Training batches on cuda:  26%|██▌       | 2522/9708 [00:11<00:27, 256.67batch/s]

Training batches on cuda:  26%|██▋       | 2550/9708 [00:11<00:27, 261.82batch/s]

Training batches on cuda:  27%|██▋       | 2578/9708 [00:12<00:26, 266.84batch/s]

Training batches on cuda:  27%|██▋       | 2606/9708 [00:12<00:26, 269.63batch/s]

Training batches on cuda:  27%|██▋       | 2634/9708 [00:12<00:26, 271.15batch/s]

Training batches on cuda:  27%|██▋       | 2662/9708 [00:12<00:25, 271.52batch/s]

Training batches on cuda:  28%|██▊       | 2690/9708 [00:12<00:25, 271.62batch/s]

Training batches on cuda:  28%|██▊       | 2718/9708 [00:12<00:25, 271.58batch/s]

Training batches on cuda:  28%|██▊       | 2746/9708 [00:12<00:25, 270.44batch/s]

Training batches on cuda:  29%|██▊       | 2774/9708 [00:12<00:25, 269.51batch/s]

Training batches on cuda:  29%|██▉       | 2801/9708 [00:12<00:25, 269.12batch/s]

Training batches on cuda:  29%|██▉       | 2828/9708 [00:12<00:25, 267.78batch/s]

Training batches on cuda:  29%|██▉       | 2856/9708 [00:13<00:25, 269.70batch/s]

Training batches on cuda:  30%|██▉       | 2884/9708 [00:13<00:25, 271.38batch/s]

Training batches on cuda:  30%|██▉       | 2912/9708 [00:13<00:24, 272.63batch/s]

Training batches on cuda:  30%|███       | 2940/9708 [00:13<00:24, 273.90batch/s]

Training batches on cuda:  31%|███       | 2968/9708 [00:13<00:24, 275.05batch/s]

Training batches on cuda:  31%|███       | 2996/9708 [00:13<00:24, 275.71batch/s]

Training batches on cuda:  31%|███       | 3024/9708 [00:13<00:24, 275.45batch/s]

Training batches on cuda:  31%|███▏      | 3052/9708 [00:13<00:24, 275.40batch/s]

Training batches on cuda:  32%|███▏      | 3080/9708 [00:13<00:24, 275.99batch/s]

Training batches on cuda:  32%|███▏      | 3108/9708 [00:13<00:24, 273.05batch/s]

Training batches on cuda:  32%|███▏      | 3136/9708 [00:14<00:24, 271.44batch/s]

Training batches on cuda:  33%|███▎      | 3164/9708 [00:14<00:24, 271.75batch/s]

Training batches on cuda:  33%|███▎      | 3192/9708 [00:14<00:23, 271.76batch/s]

Training batches on cuda:  33%|███▎      | 3220/9708 [00:14<00:23, 271.75batch/s]

Training batches on cuda:  33%|███▎      | 3248/9708 [00:14<00:23, 272.22batch/s]

Training batches on cuda:  34%|███▎      | 3276/9708 [00:14<00:23, 272.50batch/s]

Training batches on cuda:  34%|███▍      | 3304/9708 [00:14<00:23, 273.78batch/s]

Training batches on cuda:  34%|███▍      | 3332/9708 [00:14<00:23, 273.94batch/s]

Training batches on cuda:  35%|███▍      | 3360/9708 [00:14<00:23, 274.55batch/s]

Training batches on cuda:  35%|███▍      | 3388/9708 [00:14<00:23, 273.51batch/s]

Training batches on cuda:  35%|███▌      | 3416/9708 [00:15<00:23, 272.89batch/s]

Training batches on cuda:  35%|███▌      | 3444/9708 [00:15<00:22, 272.63batch/s]

Training batches on cuda:  36%|███▌      | 3472/9708 [00:15<00:22, 273.45batch/s]

Training batches on cuda:  36%|███▌      | 3500/9708 [00:15<00:22, 271.00batch/s]

Training batches on cuda:  36%|███▋      | 3528/9708 [00:15<00:22, 268.73batch/s]

Training batches on cuda:  37%|███▋      | 3555/9708 [00:15<00:22, 269.10batch/s]

Training batches on cuda:  37%|███▋      | 3583/9708 [00:15<00:22, 269.98batch/s]

Training batches on cuda:  37%|███▋      | 3611/9708 [00:15<00:22, 270.59batch/s]

Training batches on cuda:  37%|███▋      | 3639/9708 [00:15<00:22, 271.03batch/s]

Training batches on cuda:  38%|███▊      | 3667/9708 [00:16<00:22, 272.45batch/s]

Training batches on cuda:  38%|███▊      | 3695/9708 [00:16<00:27, 215.90batch/s]

Training batches on cuda:  38%|███▊      | 3719/9708 [00:16<00:27, 215.59batch/s]

Training batches on cuda:  39%|███▊      | 3742/9708 [00:16<00:27, 214.86batch/s]

Training batches on cuda:  39%|███▉      | 3765/9708 [00:16<00:27, 214.03batch/s]

Training batches on cuda:  39%|███▉      | 3788/9708 [00:16<00:27, 216.42batch/s]

Training batches on cuda:  39%|███▉      | 3812/9708 [00:16<00:26, 220.54batch/s]

Training batches on cuda:  40%|███▉      | 3836/9708 [00:16<00:26, 224.84batch/s]

Training batches on cuda:  40%|███▉      | 3860/9708 [00:16<00:25, 227.80batch/s]

Training batches on cuda:  40%|████      | 3884/9708 [00:17<00:25, 229.46batch/s]

Training batches on cuda:  40%|████      | 3908/9708 [00:17<00:25, 230.02batch/s]

Training batches on cuda:  41%|████      | 3932/9708 [00:17<00:25, 230.63batch/s]

Training batches on cuda:  41%|████      | 3956/9708 [00:17<00:24, 231.37batch/s]

Training batches on cuda:  41%|████      | 3980/9708 [00:17<00:24, 232.68batch/s]

Training batches on cuda:  41%|████      | 4004/9708 [00:17<00:24, 231.80batch/s]

Training batches on cuda:  41%|████▏     | 4028/9708 [00:17<00:25, 225.82batch/s]

Training batches on cuda:  42%|████▏     | 4051/9708 [00:17<00:26, 217.14batch/s]

Training batches on cuda:  42%|████▏     | 4073/9708 [00:17<00:27, 206.57batch/s]

Training batches on cuda:  42%|████▏     | 4094/9708 [00:18<00:27, 203.24batch/s]

Training batches on cuda:  42%|████▏     | 4118/9708 [00:18<00:26, 210.79batch/s]

Training batches on cuda:  43%|████▎     | 4142/9708 [00:18<00:25, 216.48batch/s]

Training batches on cuda:  43%|████▎     | 4166/9708 [00:18<00:25, 221.31batch/s]

Training batches on cuda:  43%|████▎     | 4191/9708 [00:18<00:24, 226.89batch/s]

Training batches on cuda:  43%|████▎     | 4215/9708 [00:18<00:24, 228.43batch/s]

Training batches on cuda:  44%|████▎     | 4239/9708 [00:18<00:23, 229.17batch/s]

Training batches on cuda:  44%|████▍     | 4263/9708 [00:18<00:23, 231.71batch/s]

Training batches on cuda:  44%|████▍     | 4287/9708 [00:18<00:23, 229.55batch/s]

Training batches on cuda:  44%|████▍     | 4310/9708 [00:18<00:23, 228.51batch/s]

Training batches on cuda:  45%|████▍     | 4334/9708 [00:19<00:23, 231.49batch/s]

Training batches on cuda:  45%|████▍     | 4358/9708 [00:19<00:23, 231.30batch/s]

Training batches on cuda:  45%|████▌     | 4382/9708 [00:19<00:23, 224.41batch/s]

Training batches on cuda:  45%|████▌     | 4405/9708 [00:19<00:24, 220.90batch/s]

Training batches on cuda:  46%|████▌     | 4428/9708 [00:19<00:24, 213.48batch/s]

Training batches on cuda:  46%|████▌     | 4450/9708 [00:19<00:25, 204.87batch/s]

Training batches on cuda:  46%|████▌     | 4471/9708 [00:19<00:26, 195.61batch/s]

Training batches on cuda:  46%|████▋     | 4492/9708 [00:19<00:26, 197.31batch/s]

Training batches on cuda:  46%|████▋     | 4514/9708 [00:19<00:25, 202.39batch/s]

Training batches on cuda:  47%|████▋     | 4536/9708 [00:20<00:25, 205.07batch/s]

Training batches on cuda:  47%|████▋     | 4558/9708 [00:20<00:24, 206.54batch/s]

Training batches on cuda:  47%|████▋     | 4580/9708 [00:20<00:24, 209.92batch/s]

Training batches on cuda:  47%|████▋     | 4602/9708 [00:20<00:24, 211.37batch/s]

Training batches on cuda:  48%|████▊     | 4624/9708 [00:20<00:23, 212.43batch/s]

Training batches on cuda:  48%|████▊     | 4647/9708 [00:20<00:23, 215.82batch/s]

Training batches on cuda:  48%|████▊     | 4671/9708 [00:20<00:22, 220.09batch/s]

Training batches on cuda:  48%|████▊     | 4695/9708 [00:20<00:22, 224.83batch/s]

Training batches on cuda:  49%|████▊     | 4718/9708 [00:20<00:22, 224.06batch/s]

Training batches on cuda:  49%|████▉     | 4741/9708 [00:20<00:22, 222.66batch/s]

Training batches on cuda:  49%|████▉     | 4764/9708 [00:21<00:22, 219.78batch/s]

Training batches on cuda:  49%|████▉     | 4787/9708 [00:21<00:27, 178.11batch/s]

Training batches on cuda:  50%|████▉     | 4811/9708 [00:21<00:25, 191.84batch/s]

Training batches on cuda:  50%|████▉     | 4832/9708 [00:21<00:25, 192.98batch/s]

Training batches on cuda:  50%|█████     | 4854/9708 [00:21<00:24, 198.30batch/s]

Training batches on cuda:  50%|█████     | 4875/9708 [00:21<00:24, 201.29batch/s]

Training batches on cuda:  50%|█████     | 4896/9708 [00:21<00:23, 201.62batch/s]

Training batches on cuda:  51%|█████     | 4917/9708 [00:21<00:23, 203.27batch/s]

Training batches on cuda:  51%|█████     | 4938/9708 [00:21<00:23, 202.92batch/s]

Training batches on cuda:  51%|█████     | 4961/9708 [00:22<00:22, 209.62batch/s]

Training batches on cuda:  51%|█████▏    | 4983/9708 [00:22<00:22, 211.27batch/s]

Training batches on cuda:  52%|█████▏    | 5005/9708 [00:22<00:22, 207.38batch/s]

Training batches on cuda:  52%|█████▏    | 5026/9708 [00:22<00:22, 203.86batch/s]

Training batches on cuda:  52%|█████▏    | 5048/9708 [00:22<00:22, 206.71batch/s]

Training batches on cuda:  52%|█████▏    | 5070/9708 [00:22<00:22, 208.86batch/s]

Training batches on cuda:  52%|█████▏    | 5094/9708 [00:22<00:21, 215.29batch/s]

Training batches on cuda:  53%|█████▎    | 5116/9708 [00:22<00:21, 215.19batch/s]

Training batches on cuda:  53%|█████▎    | 5138/9708 [00:22<00:21, 216.38batch/s]

Training batches on cuda:  53%|█████▎    | 5162/9708 [00:23<00:20, 221.08batch/s]

Training batches on cuda:  53%|█████▎    | 5185/9708 [00:23<00:20, 223.62batch/s]

Training batches on cuda:  54%|█████▎    | 5208/9708 [00:23<00:20, 223.35batch/s]

Training batches on cuda:  54%|█████▍    | 5231/9708 [00:23<00:19, 224.25batch/s]

Training batches on cuda:  54%|█████▍    | 5255/9708 [00:23<00:19, 225.84batch/s]

Training batches on cuda:  54%|█████▍    | 5278/9708 [00:23<00:19, 224.56batch/s]

Training batches on cuda:  55%|█████▍    | 5302/9708 [00:23<00:19, 226.27batch/s]

Training batches on cuda:  55%|█████▍    | 5325/9708 [00:23<00:19, 227.10batch/s]

Training batches on cuda:  55%|█████▌    | 5348/9708 [00:23<00:19, 220.06batch/s]

Training batches on cuda:  55%|█████▌    | 5371/9708 [00:23<00:20, 216.72batch/s]

Training batches on cuda:  56%|█████▌    | 5393/9708 [00:24<00:20, 214.56batch/s]

Training batches on cuda:  56%|█████▌    | 5415/9708 [00:24<00:20, 212.11batch/s]

Training batches on cuda:  56%|█████▌    | 5437/9708 [00:24<00:20, 212.27batch/s]

Training batches on cuda:  56%|█████▋    | 5461/9708 [00:24<00:19, 218.25batch/s]

Training batches on cuda:  57%|█████▋    | 5486/9708 [00:24<00:18, 225.48batch/s]

Training batches on cuda:  57%|█████▋    | 5510/9708 [00:24<00:18, 227.54batch/s]

Training batches on cuda:  57%|█████▋    | 5535/9708 [00:24<00:18, 231.37batch/s]

Training batches on cuda:  57%|█████▋    | 5559/9708 [00:24<00:17, 231.15batch/s]

Training batches on cuda:  58%|█████▊    | 5583/9708 [00:24<00:17, 232.24batch/s]

Training batches on cuda:  58%|█████▊    | 5607/9708 [00:24<00:17, 230.61batch/s]

Training batches on cuda:  58%|█████▊    | 5631/9708 [00:25<00:17, 230.22batch/s]

Training batches on cuda:  58%|█████▊    | 5655/9708 [00:25<00:17, 229.41batch/s]

Training batches on cuda:  58%|█████▊    | 5679/9708 [00:25<00:17, 229.99batch/s]

Training batches on cuda:  59%|█████▊    | 5703/9708 [00:25<00:17, 224.05batch/s]

Training batches on cuda:  59%|█████▉    | 5726/9708 [00:25<00:18, 215.26batch/s]

Training batches on cuda:  59%|█████▉    | 5748/9708 [00:25<00:18, 212.08batch/s]

Training batches on cuda:  59%|█████▉    | 5770/9708 [00:25<00:18, 212.46batch/s]

Training batches on cuda:  60%|█████▉    | 5792/9708 [00:25<00:18, 211.67batch/s]

Training batches on cuda:  60%|█████▉    | 5814/9708 [00:25<00:18, 207.05batch/s]

Training batches on cuda:  60%|██████    | 5838/9708 [00:26<00:18, 214.66batch/s]

Training batches on cuda:  60%|██████    | 5860/9708 [00:26<00:24, 154.18batch/s]

Training batches on cuda:  61%|██████    | 5884/9708 [00:26<00:22, 171.80batch/s]

Training batches on cuda:  61%|██████    | 5908/9708 [00:26<00:20, 186.18batch/s]

Training batches on cuda:  61%|██████    | 5931/9708 [00:26<00:19, 196.61batch/s]

Training batches on cuda:  61%|██████▏   | 5955/9708 [00:26<00:18, 205.62batch/s]

Training batches on cuda:  62%|██████▏   | 5978/9708 [00:26<00:17, 211.84batch/s]

Training batches on cuda:  62%|██████▏   | 6001/9708 [00:26<00:17, 208.79batch/s]

Training batches on cuda:  62%|██████▏   | 6025/9708 [00:27<00:17, 215.60batch/s]

Training batches on cuda:  62%|██████▏   | 6048/9708 [00:27<00:16, 219.35batch/s]

Training batches on cuda:  63%|██████▎   | 6072/9708 [00:27<00:16, 224.12batch/s]

Training batches on cuda:  63%|██████▎   | 6096/9708 [00:27<00:15, 227.80batch/s]

Training batches on cuda:  63%|██████▎   | 6119/9708 [00:27<00:15, 225.71batch/s]

Training batches on cuda:  63%|██████▎   | 6142/9708 [00:27<00:15, 226.47batch/s]

Training batches on cuda:  64%|██████▎   | 6165/9708 [00:27<00:15, 222.41batch/s]

Training batches on cuda:  64%|██████▎   | 6188/9708 [00:27<00:16, 218.32batch/s]

Training batches on cuda:  64%|██████▍   | 6210/9708 [00:27<00:16, 213.86batch/s]

Training batches on cuda:  64%|██████▍   | 6233/9708 [00:27<00:16, 216.19batch/s]

Training batches on cuda:  64%|██████▍   | 6255/9708 [00:28<00:15, 216.20batch/s]

Training batches on cuda:  65%|██████▍   | 6277/9708 [00:28<00:15, 215.30batch/s]

Training batches on cuda:  65%|██████▍   | 6299/9708 [00:28<00:15, 215.01batch/s]

Training batches on cuda:  65%|██████▌   | 6322/9708 [00:28<00:15, 217.19batch/s]

Training batches on cuda:  65%|██████▌   | 6344/9708 [00:28<00:15, 217.33batch/s]

Training batches on cuda:  66%|██████▌   | 6366/9708 [00:28<00:15, 214.76batch/s]

Training batches on cuda:  66%|██████▌   | 6388/9708 [00:28<00:15, 214.51batch/s]

Training batches on cuda:  66%|██████▌   | 6410/9708 [00:28<00:15, 211.91batch/s]

Training batches on cuda:  66%|██████▋   | 6432/9708 [00:28<00:15, 209.73batch/s]

Training batches on cuda:  66%|██████▋   | 6455/9708 [00:29<00:15, 213.33batch/s]

Training batches on cuda:  67%|██████▋   | 6479/9708 [00:29<00:14, 219.59batch/s]

Training batches on cuda:  67%|██████▋   | 6502/9708 [00:29<00:14, 217.33batch/s]

Training batches on cuda:  67%|██████▋   | 6526/9708 [00:29<00:14, 222.36batch/s]

Training batches on cuda:  67%|██████▋   | 6549/9708 [00:29<00:14, 218.73batch/s]

Training batches on cuda:  68%|██████▊   | 6571/9708 [00:29<00:14, 215.56batch/s]

Training batches on cuda:  68%|██████▊   | 6593/9708 [00:29<00:14, 214.57batch/s]

Training batches on cuda:  68%|██████▊   | 6616/9708 [00:29<00:14, 218.28batch/s]

Training batches on cuda:  68%|██████▊   | 6639/9708 [00:29<00:14, 218.97batch/s]

Training batches on cuda:  69%|██████▊   | 6662/9708 [00:29<00:13, 222.04batch/s]

Training batches on cuda:  69%|██████▉   | 6685/9708 [00:30<00:13, 222.02batch/s]

Training batches on cuda:  69%|██████▉   | 6708/9708 [00:30<00:13, 218.62batch/s]

Training batches on cuda:  69%|██████▉   | 6730/9708 [00:30<00:13, 214.30batch/s]

Training batches on cuda:  70%|██████▉   | 6752/9708 [00:30<00:14, 208.79batch/s]

Training batches on cuda:  70%|██████▉   | 6773/9708 [00:30<00:14, 201.46batch/s]

Training batches on cuda:  70%|███████   | 6796/9708 [00:30<00:14, 206.90batch/s]

Training batches on cuda:  70%|███████   | 6820/9708 [00:30<00:13, 212.76batch/s]

Training batches on cuda:  70%|███████   | 6842/9708 [00:30<00:13, 212.50batch/s]

Training batches on cuda:  71%|███████   | 6864/9708 [00:30<00:13, 212.26batch/s]

Training batches on cuda:  71%|███████   | 6886/9708 [00:31<00:13, 212.40batch/s]

Training batches on cuda:  71%|███████   | 6908/9708 [00:31<00:13, 213.27batch/s]

Training batches on cuda:  71%|███████▏  | 6930/9708 [00:31<00:15, 178.30batch/s]

Training batches on cuda:  72%|███████▏  | 6949/9708 [00:31<00:15, 173.32batch/s]

Training batches on cuda:  72%|███████▏  | 6968/9708 [00:31<00:16, 166.74batch/s]

Training batches on cuda:  72%|███████▏  | 6989/9708 [00:31<00:15, 177.54batch/s]

Training batches on cuda:  72%|███████▏  | 7010/9708 [00:31<00:14, 185.07batch/s]

Training batches on cuda:  72%|███████▏  | 7032/9708 [00:31<00:13, 193.51batch/s]

Training batches on cuda:  73%|███████▎  | 7053/9708 [00:31<00:13, 195.21batch/s]

Training batches on cuda:  73%|███████▎  | 7073/9708 [00:32<00:13, 196.28batch/s]

Training batches on cuda:  73%|███████▎  | 7093/9708 [00:32<00:13, 195.88batch/s]

Training batches on cuda:  73%|███████▎  | 7114/9708 [00:32<00:13, 197.49batch/s]

Training batches on cuda:  74%|███████▎  | 7136/9708 [00:32<00:12, 202.90batch/s]

Training batches on cuda:  74%|███████▎  | 7157/9708 [00:32<00:12, 203.87batch/s]

Training batches on cuda:  74%|███████▍  | 7178/9708 [00:32<00:12, 204.42batch/s]

Training batches on cuda:  74%|███████▍  | 7200/9708 [00:32<00:12, 206.50batch/s]

Training batches on cuda:  74%|███████▍  | 7222/9708 [00:32<00:11, 207.74batch/s]

Training batches on cuda:  75%|███████▍  | 7243/9708 [00:32<00:11, 205.83batch/s]

Training batches on cuda:  75%|███████▍  | 7264/9708 [00:32<00:11, 206.57batch/s]

Training batches on cuda:  75%|███████▌  | 7287/9708 [00:33<00:11, 210.64batch/s]

Training batches on cuda:  75%|███████▌  | 7309/9708 [00:33<00:11, 210.99batch/s]

Training batches on cuda:  76%|███████▌  | 7331/9708 [00:33<00:11, 211.09batch/s]

Training batches on cuda:  76%|███████▌  | 7353/9708 [00:33<00:11, 211.69batch/s]

Training batches on cuda:  76%|███████▌  | 7375/9708 [00:33<00:10, 213.25batch/s]

Training batches on cuda:  76%|███████▌  | 7397/9708 [00:33<00:10, 212.93batch/s]

Training batches on cuda:  76%|███████▋  | 7419/9708 [00:33<00:10, 213.63batch/s]

Training batches on cuda:  77%|███████▋  | 7441/9708 [00:33<00:10, 212.63batch/s]

Training batches on cuda:  77%|███████▋  | 7463/9708 [00:33<00:10, 213.42batch/s]

Training batches on cuda:  77%|███████▋  | 7485/9708 [00:33<00:10, 212.81batch/s]

Training batches on cuda:  77%|███████▋  | 7507/9708 [00:34<00:10, 211.42batch/s]

Training batches on cuda:  78%|███████▊  | 7531/9708 [00:34<00:09, 218.63batch/s]

Training batches on cuda:  78%|███████▊  | 7555/9708 [00:34<00:09, 222.34batch/s]

Training batches on cuda:  78%|███████▊  | 7578/9708 [00:34<00:10, 210.17batch/s]

Training batches on cuda:  78%|███████▊  | 7600/9708 [00:34<00:10, 205.84batch/s]

Training batches on cuda:  79%|███████▊  | 7621/9708 [00:34<00:10, 202.30batch/s]

Training batches on cuda:  79%|███████▊  | 7643/9708 [00:34<00:10, 206.45batch/s]

Training batches on cuda:  79%|███████▉  | 7666/9708 [00:34<00:09, 212.78batch/s]

Training batches on cuda:  79%|███████▉  | 7690/9708 [00:34<00:09, 218.02batch/s]

Training batches on cuda:  79%|███████▉  | 7712/9708 [00:35<00:09, 218.31batch/s]

Training batches on cuda:  80%|███████▉  | 7736/9708 [00:35<00:08, 222.27batch/s]

Training batches on cuda:  80%|███████▉  | 7759/9708 [00:35<00:08, 224.42batch/s]

Training batches on cuda:  80%|████████  | 7783/9708 [00:35<00:08, 226.32batch/s]

Training batches on cuda:  80%|████████  | 7808/9708 [00:35<00:08, 232.65batch/s]

Training batches on cuda:  81%|████████  | 7832/9708 [00:35<00:08, 234.00batch/s]

Training batches on cuda:  81%|████████  | 7856/9708 [00:35<00:07, 233.12batch/s]

Training batches on cuda:  81%|████████  | 7880/9708 [00:35<00:08, 222.19batch/s]

Training batches on cuda:  81%|████████▏ | 7903/9708 [00:35<00:08, 219.10batch/s]

Training batches on cuda:  82%|████████▏ | 7926/9708 [00:36<00:08, 216.59batch/s]

Training batches on cuda:  82%|████████▏ | 7948/9708 [00:36<00:08, 214.50batch/s]

Training batches on cuda:  82%|████████▏ | 7970/9708 [00:36<00:08, 214.17batch/s]

Training batches on cuda:  82%|████████▏ | 7992/9708 [00:36<00:10, 161.24batch/s]

Training batches on cuda:  83%|████████▎ | 8013/9708 [00:36<00:09, 173.30batch/s]

Training batches on cuda:  83%|████████▎ | 8035/9708 [00:36<00:09, 184.77batch/s]

Training batches on cuda:  83%|████████▎ | 8057/9708 [00:36<00:08, 192.80batch/s]

Training batches on cuda:  83%|████████▎ | 8078/9708 [00:36<00:08, 197.65batch/s]

Training batches on cuda:  83%|████████▎ | 8101/9708 [00:36<00:07, 204.35batch/s]

Training batches on cuda:  84%|████████▎ | 8123/9708 [00:37<00:07, 206.88batch/s]

Training batches on cuda:  84%|████████▍ | 8145/9708 [00:37<00:07, 208.37batch/s]

Training batches on cuda:  84%|████████▍ | 8167/9708 [00:37<00:07, 210.17batch/s]

Training batches on cuda:  84%|████████▍ | 8189/9708 [00:37<00:07, 212.42batch/s]

Training batches on cuda:  85%|████████▍ | 8211/9708 [00:37<00:06, 214.00batch/s]

Training batches on cuda:  85%|████████▍ | 8234/9708 [00:37<00:06, 217.80batch/s]

Training batches on cuda:  85%|████████▌ | 8258/9708 [00:37<00:06, 223.18batch/s]

Training batches on cuda:  85%|████████▌ | 8282/9708 [00:37<00:06, 226.36batch/s]

Training batches on cuda:  86%|████████▌ | 8308/9708 [00:37<00:05, 233.94batch/s]

Training batches on cuda:  86%|████████▌ | 8333/9708 [00:37<00:05, 237.23batch/s]

Training batches on cuda:  86%|████████▌ | 8357/9708 [00:38<00:05, 234.06batch/s]

Training batches on cuda:  86%|████████▋ | 8381/9708 [00:38<00:05, 235.79batch/s]

Training batches on cuda:  87%|████████▋ | 8406/9708 [00:38<00:05, 237.23batch/s]

Training batches on cuda:  87%|████████▋ | 8430/9708 [00:38<00:05, 231.31batch/s]

Training batches on cuda:  87%|████████▋ | 8454/9708 [00:38<00:05, 225.29batch/s]

Training batches on cuda:  87%|████████▋ | 8477/9708 [00:38<00:05, 217.93batch/s]

Training batches on cuda:  88%|████████▊ | 8499/9708 [00:38<00:05, 218.08batch/s]

Training batches on cuda:  88%|████████▊ | 8521/9708 [00:38<00:05, 211.79batch/s]

Training batches on cuda:  88%|████████▊ | 8546/9708 [00:38<00:05, 221.01batch/s]

Training batches on cuda:  88%|████████▊ | 8570/9708 [00:39<00:05, 224.46batch/s]

Training batches on cuda:  89%|████████▊ | 8593/9708 [00:39<00:04, 225.97batch/s]

Training batches on cuda:  89%|████████▉ | 8617/9708 [00:39<00:04, 229.61batch/s]

Training batches on cuda:  89%|████████▉ | 8642/9708 [00:39<00:04, 232.82batch/s]

Training batches on cuda:  89%|████████▉ | 8666/9708 [00:39<00:04, 231.23batch/s]

Training batches on cuda:  90%|████████▉ | 8690/9708 [00:39<00:04, 230.81batch/s]

Training batches on cuda:  90%|████████▉ | 8715/9708 [00:39<00:04, 234.65batch/s]

Training batches on cuda:  90%|█████████ | 8739/9708 [00:39<00:04, 231.16batch/s]

Training batches on cuda:  90%|█████████ | 8763/9708 [00:39<00:04, 224.37batch/s]

Training batches on cuda:  91%|█████████ | 8786/9708 [00:39<00:04, 225.72batch/s]

Training batches on cuda:  91%|█████████ | 8809/9708 [00:40<00:04, 217.56batch/s]

Training batches on cuda:  91%|█████████ | 8831/9708 [00:40<00:04, 197.68batch/s]

Training batches on cuda:  91%|█████████ | 8852/9708 [00:40<00:04, 189.40batch/s]

Training batches on cuda:  91%|█████████▏| 8874/9708 [00:40<00:04, 196.53batch/s]

Training batches on cuda:  92%|█████████▏| 8898/9708 [00:40<00:03, 206.27batch/s]

Training batches on cuda:  92%|█████████▏| 8920/9708 [00:40<00:03, 209.56batch/s]

Training batches on cuda:  92%|█████████▏| 8942/9708 [00:40<00:03, 203.04batch/s]

Training batches on cuda:  92%|█████████▏| 8964/9708 [00:40<00:03, 207.62batch/s]

Training batches on cuda:  93%|█████████▎| 8986/9708 [00:40<00:03, 210.30batch/s]

Training batches on cuda:  93%|█████████▎| 9008/9708 [00:41<00:03, 208.40batch/s]

Training batches on cuda:  93%|█████████▎| 9030/9708 [00:41<00:03, 209.21batch/s]

Training batches on cuda:  93%|█████████▎| 9051/9708 [00:41<00:03, 206.14batch/s]

Training batches on cuda:  93%|█████████▎| 9072/9708 [00:41<00:03, 168.85batch/s]

Training batches on cuda:  94%|█████████▎| 9094/9708 [00:41<00:03, 180.31batch/s]

Training batches on cuda:  94%|█████████▍| 9115/9708 [00:41<00:03, 187.74batch/s]

Training batches on cuda:  94%|█████████▍| 9136/9708 [00:41<00:02, 193.81batch/s]

Training batches on cuda:  94%|█████████▍| 9158/9708 [00:41<00:02, 198.67batch/s]

Training batches on cuda:  95%|█████████▍| 9179/9708 [00:41<00:02, 200.44batch/s]

Training batches on cuda:  95%|█████████▍| 9200/9708 [00:42<00:02, 200.82batch/s]

Training batches on cuda:  95%|█████████▍| 9221/9708 [00:42<00:02, 193.32batch/s]

Training batches on cuda:  95%|█████████▌| 9245/9708 [00:42<00:02, 203.71batch/s]

Training batches on cuda:  95%|█████████▌| 9266/9708 [00:42<00:02, 203.61batch/s]

Training batches on cuda:  96%|█████████▌| 9287/9708 [00:42<00:02, 195.46batch/s]

Training batches on cuda:  96%|█████████▌| 9309/9708 [00:42<00:01, 200.56batch/s]

Training batches on cuda:  96%|█████████▌| 9330/9708 [00:42<00:01, 203.12batch/s]

Training batches on cuda:  96%|█████████▋| 9352/9708 [00:42<00:01, 205.26batch/s]

Training batches on cuda:  97%|█████████▋| 9373/9708 [00:42<00:01, 206.32batch/s]

Training batches on cuda:  97%|█████████▋| 9394/9708 [00:43<00:01, 206.33batch/s]

Training batches on cuda:  97%|█████████▋| 9415/9708 [00:43<00:01, 203.45batch/s]

Training batches on cuda:  97%|█████████▋| 9436/9708 [00:43<00:01, 200.02batch/s]

Training batches on cuda:  97%|█████████▋| 9457/9708 [00:43<00:01, 199.78batch/s]

Training batches on cuda:  98%|█████████▊| 9479/9708 [00:43<00:01, 202.88batch/s]

Training batches on cuda:  98%|█████████▊| 9501/9708 [00:43<00:01, 205.51batch/s]

Training batches on cuda:  98%|█████████▊| 9523/9708 [00:43<00:00, 208.18batch/s]

Training batches on cuda:  98%|█████████▊| 9545/9708 [00:43<00:00, 209.08batch/s]

Training batches on cuda:  99%|█████████▊| 9568/9708 [00:43<00:00, 213.10batch/s]

Training batches on cuda:  99%|█████████▉| 9590/9708 [00:43<00:00, 212.72batch/s]

Training batches on cuda:  99%|█████████▉| 9612/9708 [00:44<00:00, 212.31batch/s]

Training batches on cuda:  99%|█████████▉| 9634/9708 [00:44<00:00, 213.64batch/s]

Training batches on cuda:  99%|█████████▉| 9656/9708 [00:44<00:00, 211.44batch/s]

Training batches on cuda: 100%|█████████▉| 9678/9708 [00:44<00:00, 211.42batch/s]

Training batches on cuda: 100%|█████████▉| 9700/9708 [00:44<00:00, 211.06batch/s]

                                                                                 
Training epochs on cuda:  40%|████      | 4/10 [03:51<04:39, 46.62s/epoch, loss=0.096, prev_loss=0.096]
Training epochs on cuda:  50%|█████     | 5/10 [03:51<03:50, 46.04s/epoch, loss=0.096, prev_loss=0.096]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 18/9708 [00:00<00:55, 175.14batch/s]

Training batches on cuda:   0%|          | 42/9708 [00:00<00:51, 188.31batch/s]

Training batches on cuda:   1%|          | 61/9708 [00:00<00:51, 188.60batch/s]

Training batches on cuda:   1%|          | 81/9708 [00:00<00:50, 189.75batch/s]

Training batches on cuda:   1%|          | 100/9708 [00:00<00:50, 189.11batch/s]

Training batches on cuda:   1%|          | 120/9708 [00:00<00:50, 191.36batch/s]

Training batches on cuda:   1%|▏         | 141/9708 [00:00<00:49, 194.15batch/s]

Training batches on cuda:   2%|▏         | 164/9708 [00:00<00:47, 202.60batch/s]

Training batches on cuda:   2%|▏         | 188/9708 [00:00<00:45, 210.57batch/s]

Training batches on cuda:   2%|▏         | 212/9708 [00:01<00:43, 218.10batch/s]

Training batches on cuda:   2%|▏         | 235/9708 [00:01<00:43, 219.85batch/s]

Training batches on cuda:   3%|▎         | 258/9708 [00:01<00:42, 221.80batch/s]

Training batches on cuda:   3%|▎         | 281/9708 [00:01<00:42, 223.70batch/s]

Training batches on cuda:   3%|▎         | 304/9708 [00:01<00:41, 225.26batch/s]

Training batches on cuda:   3%|▎         | 328/9708 [00:01<00:41, 226.73batch/s]

Training batches on cuda:   4%|▎         | 351/9708 [00:01<00:41, 226.94batch/s]

Training batches on cuda:   4%|▍         | 374/9708 [00:01<00:44, 208.50batch/s]

Training batches on cuda:   4%|▍         | 396/9708 [00:01<00:44, 211.39batch/s]

Training batches on cuda:   4%|▍         | 419/9708 [00:01<00:43, 214.56batch/s]

Training batches on cuda:   5%|▍         | 442/9708 [00:02<00:42, 216.76batch/s]

Training batches on cuda:   5%|▍         | 465/9708 [00:02<00:42, 219.63batch/s]

Training batches on cuda:   5%|▌         | 488/9708 [00:02<00:41, 222.13batch/s]

Training batches on cuda:   5%|▌         | 511/9708 [00:02<00:41, 220.96batch/s]

Training batches on cuda:   6%|▌         | 534/9708 [00:02<00:43, 210.33batch/s]

Training batches on cuda:   6%|▌         | 556/9708 [00:02<00:45, 203.35batch/s]

Training batches on cuda:   6%|▌         | 577/9708 [00:02<00:45, 201.24batch/s]

Training batches on cuda:   6%|▌         | 598/9708 [00:02<00:45, 200.52batch/s]

Training batches on cuda:   6%|▋         | 619/9708 [00:02<00:44, 203.08batch/s]

Training batches on cuda:   7%|▋         | 641/9708 [00:03<00:43, 206.68batch/s]

Training batches on cuda:   7%|▋         | 666/9708 [00:03<00:41, 216.92batch/s]

Training batches on cuda:   7%|▋         | 691/9708 [00:03<00:40, 223.33batch/s]

Training batches on cuda:   7%|▋         | 715/9708 [00:03<00:39, 226.50batch/s]

Training batches on cuda:   8%|▊         | 738/9708 [00:03<00:39, 227.44batch/s]

Training batches on cuda:   8%|▊         | 763/9708 [00:03<00:38, 232.53batch/s]

Training batches on cuda:   8%|▊         | 787/9708 [00:03<00:38, 232.32batch/s]

Training batches on cuda:   8%|▊         | 811/9708 [00:03<00:38, 229.22batch/s]

Training batches on cuda:   9%|▊         | 835/9708 [00:03<00:38, 231.57batch/s]

Training batches on cuda:   9%|▉         | 859/9708 [00:03<00:38, 231.75batch/s]

Training batches on cuda:   9%|▉         | 883/9708 [00:04<00:38, 230.10batch/s]

Training batches on cuda:   9%|▉         | 907/9708 [00:04<00:38, 229.16batch/s]

Training batches on cuda:  10%|▉         | 931/9708 [00:04<00:38, 229.77batch/s]

Training batches on cuda:  10%|▉         | 955/9708 [00:04<00:38, 230.11batch/s]

Training batches on cuda:  10%|█         | 979/9708 [00:04<00:38, 226.06batch/s]

Training batches on cuda:  10%|█         | 1002/9708 [00:04<00:38, 225.44batch/s]

Training batches on cuda:  11%|█         | 1025/9708 [00:04<00:39, 222.14batch/s]

Training batches on cuda:  11%|█         | 1048/9708 [00:04<00:39, 217.16batch/s]

Training batches on cuda:  11%|█         | 1071/9708 [00:04<00:39, 219.17batch/s]

Training batches on cuda:  11%|█▏        | 1095/9708 [00:05<00:38, 224.44batch/s]

Training batches on cuda:  12%|█▏        | 1118/9708 [00:05<00:38, 223.14batch/s]

Training batches on cuda:  12%|█▏        | 1141/9708 [00:05<00:38, 223.14batch/s]

Training batches on cuda:  12%|█▏        | 1165/9708 [00:05<00:37, 227.84batch/s]

Training batches on cuda:  12%|█▏        | 1189/9708 [00:05<00:37, 228.91batch/s]

Training batches on cuda:  12%|█▏        | 1213/9708 [00:05<00:37, 229.49batch/s]

Training batches on cuda:  13%|█▎        | 1236/9708 [00:05<00:37, 228.65batch/s]

Training batches on cuda:  13%|█▎        | 1261/9708 [00:05<00:36, 232.03batch/s]

Training batches on cuda:  13%|█▎        | 1285/9708 [00:05<00:36, 230.88batch/s]

Training batches on cuda:  13%|█▎        | 1309/9708 [00:05<00:37, 225.42batch/s]

Training batches on cuda:  14%|█▎        | 1332/9708 [00:06<00:37, 222.32batch/s]

Training batches on cuda:  14%|█▍        | 1355/9708 [00:06<00:38, 217.47batch/s]

Training batches on cuda:  14%|█▍        | 1378/9708 [00:06<00:37, 219.28batch/s]

Training batches on cuda:  14%|█▍        | 1401/9708 [00:06<00:37, 221.98batch/s]

Training batches on cuda:  15%|█▍        | 1424/9708 [00:06<00:37, 220.62batch/s]

Training batches on cuda:  15%|█▍        | 1448/9708 [00:06<00:36, 223.67batch/s]

Training batches on cuda:  15%|█▌        | 1472/9708 [00:06<00:36, 227.20batch/s]

Training batches on cuda:  15%|█▌        | 1495/9708 [00:06<00:42, 193.00batch/s]

Training batches on cuda:  16%|█▌        | 1516/9708 [00:06<00:42, 192.94batch/s]

Training batches on cuda:  16%|█▌        | 1537/9708 [00:07<00:41, 195.55batch/s]

Training batches on cuda:  16%|█▌        | 1560/9708 [00:07<00:40, 202.53batch/s]

Training batches on cuda:  16%|█▋        | 1584/9708 [00:07<00:38, 209.85batch/s]

Training batches on cuda:  17%|█▋        | 1606/9708 [00:07<00:39, 206.94batch/s]

Training batches on cuda:  17%|█▋        | 1628/9708 [00:07<00:38, 208.44batch/s]

Training batches on cuda:  17%|█▋        | 1650/9708 [00:07<00:38, 206.72batch/s]

Training batches on cuda:  17%|█▋        | 1672/9708 [00:07<00:38, 207.79batch/s]

Training batches on cuda:  17%|█▋        | 1693/9708 [00:07<00:38, 207.26batch/s]

Training batches on cuda:  18%|█▊        | 1715/9708 [00:07<00:38, 207.29batch/s]

Training batches on cuda:  18%|█▊        | 1736/9708 [00:08<00:40, 197.67batch/s]

Training batches on cuda:  18%|█▊        | 1760/9708 [00:08<00:38, 206.72batch/s]

Training batches on cuda:  18%|█▊        | 1784/9708 [00:08<00:36, 214.75batch/s]

Training batches on cuda:  19%|█▊        | 1806/9708 [00:08<00:36, 215.15batch/s]

Training batches on cuda:  19%|█▉        | 1829/9708 [00:08<00:36, 217.16batch/s]

Training batches on cuda:  19%|█▉        | 1852/9708 [00:08<00:35, 220.84batch/s]

Training batches on cuda:  19%|█▉        | 1875/9708 [00:08<00:35, 218.80batch/s]

Training batches on cuda:  20%|█▉        | 1897/9708 [00:08<00:37, 207.16batch/s]

Training batches on cuda:  20%|█▉        | 1919/9708 [00:08<00:37, 208.39batch/s]

Training batches on cuda:  20%|█▉        | 1941/9708 [00:08<00:37, 209.21batch/s]

Training batches on cuda:  20%|██        | 1963/9708 [00:09<00:36, 211.81batch/s]

Training batches on cuda:  20%|██        | 1985/9708 [00:09<00:36, 213.98batch/s]

Training batches on cuda:  21%|██        | 2007/9708 [00:09<00:36, 213.70batch/s]

Training batches on cuda:  21%|██        | 2029/9708 [00:09<00:36, 212.53batch/s]

Training batches on cuda:  21%|██        | 2051/9708 [00:09<00:35, 213.53batch/s]

Training batches on cuda:  21%|██▏       | 2073/9708 [00:09<00:35, 212.82batch/s]

Training batches on cuda:  22%|██▏       | 2095/9708 [00:09<00:35, 212.67batch/s]

Training batches on cuda:  22%|██▏       | 2117/9708 [00:09<00:36, 209.93batch/s]

Training batches on cuda:  22%|██▏       | 2139/9708 [00:09<00:35, 211.55batch/s]

Training batches on cuda:  22%|██▏       | 2161/9708 [00:10<00:35, 213.18batch/s]

Training batches on cuda:  22%|██▏       | 2183/9708 [00:10<00:36, 206.74batch/s]

Training batches on cuda:  23%|██▎       | 2205/9708 [00:10<00:35, 209.74batch/s]

Training batches on cuda:  23%|██▎       | 2227/9708 [00:10<00:35, 212.58batch/s]

Training batches on cuda:  23%|██▎       | 2249/9708 [00:10<00:35, 210.20batch/s]

Training batches on cuda:  23%|██▎       | 2271/9708 [00:10<00:36, 203.99batch/s]

Training batches on cuda:  24%|██▎       | 2292/9708 [00:10<00:36, 204.06batch/s]

Training batches on cuda:  24%|██▍       | 2315/9708 [00:10<00:35, 210.54batch/s]

Training batches on cuda:  24%|██▍       | 2337/9708 [00:10<00:34, 210.88batch/s]

Training batches on cuda:  24%|██▍       | 2360/9708 [00:10<00:34, 213.92batch/s]

Training batches on cuda:  25%|██▍       | 2382/9708 [00:11<00:34, 212.36batch/s]

Training batches on cuda:  25%|██▍       | 2404/9708 [00:11<00:34, 212.88batch/s]

Training batches on cuda:  25%|██▍       | 2426/9708 [00:11<00:35, 202.47batch/s]

Training batches on cuda:  25%|██▌       | 2447/9708 [00:11<00:35, 204.37batch/s]

Training batches on cuda:  25%|██▌       | 2468/9708 [00:11<00:35, 204.73batch/s]

Training batches on cuda:  26%|██▌       | 2489/9708 [00:11<00:35, 205.98batch/s]

Training batches on cuda:  26%|██▌       | 2510/9708 [00:11<00:36, 197.91batch/s]

Training batches on cuda:  26%|██▌       | 2530/9708 [00:11<00:39, 181.54batch/s]

Training batches on cuda:  26%|██▋       | 2552/9708 [00:11<00:37, 190.62batch/s]

Training batches on cuda:  27%|██▋       | 2574/9708 [00:12<00:35, 198.18batch/s]

Training batches on cuda:  27%|██▋       | 2596/9708 [00:12<00:34, 203.22batch/s]

Training batches on cuda:  27%|██▋       | 2617/9708 [00:12<00:34, 204.63batch/s]

Training batches on cuda:  27%|██▋       | 2638/9708 [00:12<00:34, 204.38batch/s]

Training batches on cuda:  27%|██▋       | 2660/9708 [00:12<00:33, 207.83batch/s]

Training batches on cuda:  28%|██▊       | 2681/9708 [00:12<00:35, 200.74batch/s]

Training batches on cuda:  28%|██▊       | 2702/9708 [00:12<00:35, 196.24batch/s]

Training batches on cuda:  28%|██▊       | 2726/9708 [00:12<00:33, 205.49batch/s]

Training batches on cuda:  28%|██▊       | 2748/9708 [00:12<00:33, 209.16batch/s]

Training batches on cuda:  29%|██▊       | 2770/9708 [00:12<00:34, 203.70batch/s]

Training batches on cuda:  29%|██▉       | 2792/9708 [00:13<00:33, 206.78batch/s]

Training batches on cuda:  29%|██▉       | 2814/9708 [00:13<00:33, 208.76batch/s]

Training batches on cuda:  29%|██▉       | 2835/9708 [00:13<00:33, 205.72batch/s]

Training batches on cuda:  29%|██▉       | 2856/9708 [00:13<00:33, 205.31batch/s]

Training batches on cuda:  30%|██▉       | 2877/9708 [00:13<00:34, 195.57batch/s]

Training batches on cuda:  30%|██▉       | 2904/9708 [00:13<00:32, 212.14batch/s]

Training batches on cuda:  30%|███       | 2926/9708 [00:13<00:32, 210.28batch/s]

Training batches on cuda:  30%|███       | 2948/9708 [00:13<00:33, 200.41batch/s]

Training batches on cuda:  31%|███       | 2969/9708 [00:13<00:34, 193.05batch/s]

Training batches on cuda:  31%|███       | 2989/9708 [00:14<00:35, 188.90batch/s]

Training batches on cuda:  31%|███       | 3009/9708 [00:14<00:34, 192.02batch/s]

Training batches on cuda:  31%|███       | 3032/9708 [00:14<00:33, 200.74batch/s]

Training batches on cuda:  31%|███▏      | 3055/9708 [00:14<00:32, 207.46batch/s]

Training batches on cuda:  32%|███▏      | 3077/9708 [00:14<00:31, 208.44batch/s]

Training batches on cuda:  32%|███▏      | 3098/9708 [00:14<00:31, 206.86batch/s]

Training batches on cuda:  32%|███▏      | 3119/9708 [00:14<00:31, 206.26batch/s]

Training batches on cuda:  32%|███▏      | 3140/9708 [00:14<00:31, 205.52batch/s]

Training batches on cuda:  33%|███▎      | 3161/9708 [00:14<00:33, 198.38batch/s]

Training batches on cuda:  33%|███▎      | 3181/9708 [00:15<00:33, 197.01batch/s]

Training batches on cuda:  33%|███▎      | 3203/9708 [00:15<00:32, 201.48batch/s]

Training batches on cuda:  33%|███▎      | 3225/9708 [00:15<00:31, 204.47batch/s]

Training batches on cuda:  33%|███▎      | 3248/9708 [00:15<00:30, 208.75batch/s]

Training batches on cuda:  34%|███▎      | 3269/9708 [00:15<00:30, 208.44batch/s]

Training batches on cuda:  34%|███▍      | 3291/9708 [00:15<00:30, 209.39batch/s]

Training batches on cuda:  34%|███▍      | 3312/9708 [00:15<00:30, 207.75batch/s]

Training batches on cuda:  34%|███▍      | 3333/9708 [00:15<00:31, 204.21batch/s]

Training batches on cuda:  35%|███▍      | 3354/9708 [00:15<00:31, 202.90batch/s]

Training batches on cuda:  35%|███▍      | 3375/9708 [00:15<00:31, 198.88batch/s]

Training batches on cuda:  35%|███▍      | 3395/9708 [00:16<00:31, 198.79batch/s]

Training batches on cuda:  35%|███▌      | 3415/9708 [00:16<00:33, 189.81batch/s]

Training batches on cuda:  35%|███▌      | 3436/9708 [00:16<00:32, 195.25batch/s]

Training batches on cuda:  36%|███▌      | 3461/9708 [00:16<00:30, 206.95batch/s]

Training batches on cuda:  36%|███▌      | 3485/9708 [00:16<00:29, 213.69batch/s]

Training batches on cuda:  36%|███▌      | 3507/9708 [00:16<00:28, 214.29batch/s]

Training batches on cuda:  36%|███▋      | 3529/9708 [00:16<00:28, 213.21batch/s]

Training batches on cuda:  37%|███▋      | 3551/9708 [00:16<00:29, 206.14batch/s]

Training batches on cuda:  37%|███▋      | 3572/9708 [00:16<00:33, 181.28batch/s]

Training batches on cuda:  37%|███▋      | 3595/9708 [00:17<00:31, 191.98batch/s]

Training batches on cuda:  37%|███▋      | 3617/9708 [00:17<00:30, 198.86batch/s]

Training batches on cuda:  37%|███▋      | 3639/9708 [00:17<00:29, 203.76batch/s]

Training batches on cuda:  38%|███▊      | 3662/9708 [00:17<00:28, 210.86batch/s]

Training batches on cuda:  38%|███▊      | 3684/9708 [00:17<00:28, 210.88batch/s]

Training batches on cuda:  38%|███▊      | 3707/9708 [00:17<00:28, 213.62batch/s]

Training batches on cuda:  38%|███▊      | 3729/9708 [00:17<00:27, 214.50batch/s]

Training batches on cuda:  39%|███▊      | 3752/9708 [00:17<00:27, 216.93batch/s]

Training batches on cuda:  39%|███▉      | 3774/9708 [00:17<00:27, 217.79batch/s]

Training batches on cuda:  39%|███▉      | 3796/9708 [00:17<00:27, 217.89batch/s]

Training batches on cuda:  39%|███▉      | 3819/9708 [00:18<00:26, 219.85batch/s]

Training batches on cuda:  40%|███▉      | 3842/9708 [00:18<00:26, 219.57batch/s]

Training batches on cuda:  40%|███▉      | 3864/9708 [00:18<00:27, 213.81batch/s]

Training batches on cuda:  40%|████      | 3886/9708 [00:18<00:27, 215.58batch/s]

Training batches on cuda:  40%|████      | 3908/9708 [00:18<00:27, 213.17batch/s]

Training batches on cuda:  40%|████      | 3931/9708 [00:18<00:26, 217.72batch/s]

Training batches on cuda:  41%|████      | 3955/9708 [00:18<00:25, 222.02batch/s]

Training batches on cuda:  41%|████      | 3978/9708 [00:18<00:26, 216.69batch/s]

Training batches on cuda:  41%|████      | 4000/9708 [00:18<00:27, 205.73batch/s]

Training batches on cuda:  41%|████▏     | 4021/9708 [00:19<00:27, 206.33batch/s]

Training batches on cuda:  42%|████▏     | 4044/9708 [00:19<00:26, 210.89batch/s]

Training batches on cuda:  42%|████▏     | 4066/9708 [00:19<00:26, 212.29batch/s]

Training batches on cuda:  42%|████▏     | 4088/9708 [00:19<00:26, 212.70batch/s]

Training batches on cuda:  42%|████▏     | 4110/9708 [00:19<00:27, 206.59batch/s]

Training batches on cuda:  43%|████▎     | 4131/9708 [00:19<00:28, 197.71batch/s]

Training batches on cuda:  43%|████▎     | 4151/9708 [00:19<00:28, 197.51batch/s]

Training batches on cuda:  43%|████▎     | 4172/9708 [00:19<00:27, 198.44batch/s]

Training batches on cuda:  43%|████▎     | 4194/9708 [00:19<00:27, 202.21batch/s]

Training batches on cuda:  43%|████▎     | 4215/9708 [00:19<00:27, 203.29batch/s]

Training batches on cuda:  44%|████▎     | 4236/9708 [00:20<00:26, 205.18batch/s]

Training batches on cuda:  44%|████▍     | 4257/9708 [00:20<00:27, 201.77batch/s]

Training batches on cuda:  44%|████▍     | 4279/9708 [00:20<00:26, 204.87batch/s]

Training batches on cuda:  44%|████▍     | 4301/9708 [00:20<00:25, 208.28batch/s]

Training batches on cuda:  45%|████▍     | 4324/9708 [00:20<00:25, 211.33batch/s]

Training batches on cuda:  45%|████▍     | 4346/9708 [00:20<00:26, 203.73batch/s]

Training batches on cuda:  45%|████▍     | 4367/9708 [00:20<00:26, 199.39batch/s]

Training batches on cuda:  45%|████▌     | 4388/9708 [00:20<00:27, 190.73batch/s]

Training batches on cuda:  45%|████▌     | 4409/9708 [00:20<00:27, 195.93batch/s]

Training batches on cuda:  46%|████▌     | 4431/9708 [00:21<00:26, 200.28batch/s]

Training batches on cuda:  46%|████▌     | 4453/9708 [00:21<00:25, 204.74batch/s]

Training batches on cuda:  46%|████▌     | 4475/9708 [00:21<00:25, 208.97batch/s]

Training batches on cuda:  46%|████▋     | 4497/9708 [00:21<00:24, 211.07batch/s]

Training batches on cuda:  47%|████▋     | 4519/9708 [00:21<00:24, 213.37batch/s]

Training batches on cuda:  47%|████▋     | 4542/9708 [00:21<00:23, 216.03batch/s]

Training batches on cuda:  47%|████▋     | 4564/9708 [00:21<00:23, 214.90batch/s]

Training batches on cuda:  47%|████▋     | 4586/9708 [00:21<00:23, 214.11batch/s]

Training batches on cuda:  47%|████▋     | 4610/9708 [00:21<00:23, 219.78batch/s]

Training batches on cuda:  48%|████▊     | 4633/9708 [00:22<00:25, 202.80batch/s]

Training batches on cuda:  48%|████▊     | 4654/9708 [00:22<00:24, 203.02batch/s]

Training batches on cuda:  48%|████▊     | 4675/9708 [00:22<00:25, 200.68batch/s]

Training batches on cuda:  48%|████▊     | 4696/9708 [00:22<00:25, 197.63batch/s]

Training batches on cuda:  49%|████▊     | 4718/9708 [00:22<00:24, 202.25batch/s]

Training batches on cuda:  49%|████▉     | 4741/9708 [00:22<00:23, 207.32batch/s]

Training batches on cuda:  49%|████▉     | 4763/9708 [00:22<00:23, 209.98batch/s]

Training batches on cuda:  49%|████▉     | 4785/9708 [00:22<00:23, 206.06batch/s]

Training batches on cuda:  50%|████▉     | 4806/9708 [00:22<00:23, 206.84batch/s]

Training batches on cuda:  50%|████▉     | 4827/9708 [00:22<00:23, 206.84batch/s]

Training batches on cuda:  50%|████▉     | 4848/9708 [00:23<00:23, 203.97batch/s]

Training batches on cuda:  50%|█████     | 4869/9708 [00:23<00:24, 195.86batch/s]

Training batches on cuda:  50%|█████     | 4890/9708 [00:23<00:24, 197.68batch/s]

Training batches on cuda:  51%|█████     | 4911/9708 [00:23<00:23, 200.04batch/s]

Training batches on cuda:  51%|█████     | 4933/9708 [00:23<00:23, 205.10batch/s]

Training batches on cuda:  51%|█████     | 4955/9708 [00:23<00:22, 207.93batch/s]

Training batches on cuda:  51%|█████▏    | 4977/9708 [00:23<00:22, 209.47batch/s]

Training batches on cuda:  51%|█████▏    | 4998/9708 [00:23<00:22, 207.70batch/s]

Training batches on cuda:  52%|█████▏    | 5022/9708 [00:23<00:21, 215.77batch/s]

Training batches on cuda:  52%|█████▏    | 5047/9708 [00:23<00:20, 222.96batch/s]

Training batches on cuda:  52%|█████▏    | 5072/9708 [00:24<00:20, 229.48batch/s]

Training batches on cuda:  53%|█████▎    | 5097/9708 [00:24<00:19, 232.88batch/s]

Training batches on cuda:  53%|█████▎    | 5121/9708 [00:24<00:20, 226.50batch/s]

Training batches on cuda:  53%|█████▎    | 5144/9708 [00:24<00:21, 212.55batch/s]

Training batches on cuda:  53%|█████▎    | 5166/9708 [00:24<00:21, 211.59batch/s]

Training batches on cuda:  53%|█████▎    | 5188/9708 [00:24<00:21, 207.70batch/s]

Training batches on cuda:  54%|█████▎    | 5210/9708 [00:24<00:21, 209.05batch/s]

Training batches on cuda:  54%|█████▍    | 5232/9708 [00:24<00:21, 209.80batch/s]

Training batches on cuda:  54%|█████▍    | 5254/9708 [00:24<00:21, 208.37batch/s]

Training batches on cuda:  54%|█████▍    | 5275/9708 [00:25<00:21, 207.64batch/s]

Training batches on cuda:  55%|█████▍    | 5296/9708 [00:25<00:21, 206.76batch/s]

Training batches on cuda:  55%|█████▍    | 5317/9708 [00:25<00:22, 198.30batch/s]

Training batches on cuda:  55%|█████▍    | 5339/9708 [00:25<00:21, 203.81batch/s]

Training batches on cuda:  55%|█████▌    | 5360/9708 [00:25<00:21, 204.26batch/s]

Training batches on cuda:  55%|█████▌    | 5382/9708 [00:25<00:20, 207.01batch/s]

Training batches on cuda:  56%|█████▌    | 5403/9708 [00:25<00:20, 207.89batch/s]

Training batches on cuda:  56%|█████▌    | 5424/9708 [00:25<00:20, 208.39batch/s]

Training batches on cuda:  56%|█████▌    | 5445/9708 [00:25<00:20, 205.25batch/s]

Training batches on cuda:  56%|█████▋    | 5467/9708 [00:26<00:20, 207.15batch/s]

Training batches on cuda:  57%|█████▋    | 5489/9708 [00:26<00:20, 210.63batch/s]

Training batches on cuda:  57%|█████▋    | 5511/9708 [00:26<00:19, 211.19batch/s]

Training batches on cuda:  57%|█████▋    | 5533/9708 [00:26<00:19, 212.29batch/s]

Training batches on cuda:  57%|█████▋    | 5555/9708 [00:26<00:19, 211.75batch/s]

Training batches on cuda:  57%|█████▋    | 5579/9708 [00:26<00:19, 217.20batch/s]

Training batches on cuda:  58%|█████▊    | 5601/9708 [00:26<00:18, 217.52batch/s]

Training batches on cuda:  58%|█████▊    | 5623/9708 [00:26<00:19, 212.24batch/s]

Training batches on cuda:  58%|█████▊    | 5647/9708 [00:26<00:18, 219.37batch/s]

Training batches on cuda:  58%|█████▊    | 5671/9708 [00:26<00:18, 222.51batch/s]

Training batches on cuda:  59%|█████▊    | 5694/9708 [00:27<00:20, 196.34batch/s]

Training batches on cuda:  59%|█████▉    | 5715/9708 [00:27<00:20, 199.61batch/s]

Training batches on cuda:  59%|█████▉    | 5736/9708 [00:27<00:19, 201.51batch/s]

Training batches on cuda:  59%|█████▉    | 5758/9708 [00:27<00:19, 204.90batch/s]

Training batches on cuda:  60%|█████▉    | 5779/9708 [00:27<00:19, 204.32batch/s]

Training batches on cuda:  60%|█████▉    | 5800/9708 [00:27<00:20, 191.24batch/s]

Training batches on cuda:  60%|█████▉    | 5821/9708 [00:27<00:19, 195.81batch/s]

Training batches on cuda:  60%|██████    | 5841/9708 [00:27<00:20, 190.68batch/s]

Training batches on cuda:  60%|██████    | 5861/9708 [00:27<00:20, 188.24batch/s]

Training batches on cuda:  61%|██████    | 5880/9708 [00:28<00:20, 186.17batch/s]

Training batches on cuda:  61%|██████    | 5902/9708 [00:28<00:19, 193.95batch/s]

Training batches on cuda:  61%|██████    | 5922/9708 [00:28<00:19, 192.10batch/s]

Training batches on cuda:  61%|██████    | 5944/9708 [00:28<00:19, 197.16batch/s]

Training batches on cuda:  61%|██████▏   | 5965/9708 [00:28<00:18, 200.78batch/s]

Training batches on cuda:  62%|██████▏   | 5987/9708 [00:28<00:18, 204.11batch/s]

Training batches on cuda:  62%|██████▏   | 6011/9708 [00:28<00:17, 211.06batch/s]

Training batches on cuda:  62%|██████▏   | 6034/9708 [00:28<00:17, 215.27batch/s]

Training batches on cuda:  62%|██████▏   | 6056/9708 [00:28<00:17, 214.66batch/s]

Training batches on cuda:  63%|██████▎   | 6078/9708 [00:28<00:17, 212.14batch/s]

Training batches on cuda:  63%|██████▎   | 6100/9708 [00:29<00:17, 211.27batch/s]

Training batches on cuda:  63%|██████▎   | 6122/9708 [00:29<00:17, 210.40batch/s]

Training batches on cuda:  63%|██████▎   | 6144/9708 [00:29<00:17, 206.22batch/s]

Training batches on cuda:  64%|██████▎   | 6166/9708 [00:29<00:17, 207.19batch/s]

Training batches on cuda:  64%|██████▎   | 6187/9708 [00:29<00:17, 198.45batch/s]

Training batches on cuda:  64%|██████▍   | 6207/9708 [00:29<00:17, 195.71batch/s]

Training batches on cuda:  64%|██████▍   | 6228/9708 [00:29<00:17, 197.71batch/s]

Training batches on cuda:  64%|██████▍   | 6250/9708 [00:29<00:17, 201.54batch/s]

Training batches on cuda:  65%|██████▍   | 6272/9708 [00:29<00:16, 205.00batch/s]

Training batches on cuda:  65%|██████▍   | 6294/9708 [00:30<00:16, 207.57batch/s]

Training batches on cuda:  65%|██████▌   | 6316/9708 [00:30<00:16, 209.56batch/s]

Training batches on cuda:  65%|██████▌   | 6338/9708 [00:30<00:15, 210.64batch/s]

Training batches on cuda:  66%|██████▌   | 6360/9708 [00:30<00:15, 211.25batch/s]

Training batches on cuda:  66%|██████▌   | 6383/9708 [00:30<00:15, 216.07batch/s]

Training batches on cuda:  66%|██████▌   | 6409/9708 [00:30<00:14, 225.36batch/s]

Training batches on cuda:  66%|██████▋   | 6433/9708 [00:30<00:14, 226.99batch/s]

Training batches on cuda:  67%|██████▋   | 6456/9708 [00:30<00:15, 216.72batch/s]

Training batches on cuda:  67%|██████▋   | 6478/9708 [00:30<00:15, 206.57batch/s]

Training batches on cuda:  67%|██████▋   | 6499/9708 [00:30<00:15, 204.06batch/s]

Training batches on cuda:  67%|██████▋   | 6521/9708 [00:31<00:15, 208.57batch/s]

Training batches on cuda:  67%|██████▋   | 6545/9708 [00:31<00:14, 216.02batch/s]

Training batches on cuda:  68%|██████▊   | 6569/9708 [00:31<00:14, 220.90batch/s]

Training batches on cuda:  68%|██████▊   | 6593/9708 [00:31<00:13, 224.55batch/s]

Training batches on cuda:  68%|██████▊   | 6616/9708 [00:31<00:13, 226.08batch/s]

Training batches on cuda:  68%|██████▊   | 6639/9708 [00:31<00:13, 222.66batch/s]

Training batches on cuda:  69%|██████▊   | 6662/9708 [00:31<00:13, 221.68batch/s]

Training batches on cuda:  69%|██████▉   | 6685/9708 [00:31<00:13, 220.08batch/s]

Training batches on cuda:  69%|██████▉   | 6708/9708 [00:31<00:13, 218.93batch/s]

Training batches on cuda:  69%|██████▉   | 6730/9708 [00:32<00:15, 197.67batch/s]

Training batches on cuda:  70%|██████▉   | 6752/9708 [00:32<00:14, 203.27batch/s]

Training batches on cuda:  70%|██████▉   | 6775/9708 [00:32<00:13, 209.78batch/s]

Training batches on cuda:  70%|███████   | 6799/9708 [00:32<00:13, 215.97batch/s]

Training batches on cuda:  70%|███████   | 6821/9708 [00:32<00:13, 216.98batch/s]

Training batches on cuda:  71%|███████   | 6845/9708 [00:32<00:12, 221.56batch/s]

Training batches on cuda:  71%|███████   | 6868/9708 [00:32<00:12, 219.24batch/s]

Training batches on cuda:  71%|███████   | 6891/9708 [00:32<00:12, 220.36batch/s]

Training batches on cuda:  71%|███████   | 6916/9708 [00:32<00:12, 226.43batch/s]

Training batches on cuda:  71%|███████▏  | 6939/9708 [00:32<00:12, 224.04batch/s]

Training batches on cuda:  72%|███████▏  | 6963/9708 [00:33<00:12, 228.13batch/s]

Training batches on cuda:  72%|███████▏  | 6986/9708 [00:33<00:12, 220.16batch/s]

Training batches on cuda:  72%|███████▏  | 7009/9708 [00:33<00:12, 212.71batch/s]

Training batches on cuda:  72%|███████▏  | 7031/9708 [00:33<00:12, 212.00batch/s]

Training batches on cuda:  73%|███████▎  | 7053/9708 [00:33<00:12, 213.75batch/s]

Training batches on cuda:  73%|███████▎  | 7075/9708 [00:33<00:12, 215.25batch/s]

Training batches on cuda:  73%|███████▎  | 7097/9708 [00:33<00:12, 206.34batch/s]

Training batches on cuda:  73%|███████▎  | 7120/9708 [00:33<00:12, 210.84batch/s]

Training batches on cuda:  74%|███████▎  | 7142/9708 [00:33<00:12, 213.24batch/s]

Training batches on cuda:  74%|███████▍  | 7165/9708 [00:34<00:11, 217.50batch/s]

Training batches on cuda:  74%|███████▍  | 7187/9708 [00:34<00:11, 217.55batch/s]

Training batches on cuda:  74%|███████▍  | 7209/9708 [00:34<00:11, 215.85batch/s]

Training batches on cuda:  74%|███████▍  | 7231/9708 [00:34<00:11, 214.58batch/s]

Training batches on cuda:  75%|███████▍  | 7253/9708 [00:34<00:11, 213.54batch/s]

Training batches on cuda:  75%|███████▍  | 7275/9708 [00:34<00:11, 213.31batch/s]

Training batches on cuda:  75%|███████▌  | 7297/9708 [00:34<00:11, 207.52batch/s]

Training batches on cuda:  75%|███████▌  | 7318/9708 [00:34<00:11, 204.53batch/s]

Training batches on cuda:  76%|███████▌  | 7340/9708 [00:34<00:11, 207.50batch/s]

Training batches on cuda:  76%|███████▌  | 7363/9708 [00:34<00:11, 212.40batch/s]

Training batches on cuda:  76%|███████▌  | 7385/9708 [00:35<00:10, 213.34batch/s]

Training batches on cuda:  76%|███████▋  | 7407/9708 [00:35<00:10, 213.71batch/s]

Training batches on cuda:  77%|███████▋  | 7429/9708 [00:35<00:11, 204.47batch/s]

Training batches on cuda:  77%|███████▋  | 7450/9708 [00:35<00:11, 197.02batch/s]

Training batches on cuda:  77%|███████▋  | 7472/9708 [00:35<00:11, 201.17batch/s]

Training batches on cuda:  77%|███████▋  | 7495/9708 [00:35<00:10, 206.79batch/s]

Training batches on cuda:  77%|███████▋  | 7516/9708 [00:35<00:10, 202.22batch/s]

Training batches on cuda:  78%|███████▊  | 7537/9708 [00:35<00:10, 201.23batch/s]

Training batches on cuda:  78%|███████▊  | 7558/9708 [00:35<00:10, 196.06batch/s]

Training batches on cuda:  78%|███████▊  | 7580/9708 [00:36<00:10, 202.57batch/s]

Training batches on cuda:  78%|███████▊  | 7601/9708 [00:36<00:10, 203.15batch/s]

Training batches on cuda:  79%|███████▊  | 7622/9708 [00:36<00:10, 204.66batch/s]

Training batches on cuda:  79%|███████▊  | 7643/9708 [00:36<00:10, 205.66batch/s]

Training batches on cuda:  79%|███████▉  | 7664/9708 [00:36<00:10, 203.34batch/s]

Training batches on cuda:  79%|███████▉  | 7685/9708 [00:36<00:10, 196.19batch/s]

Training batches on cuda:  79%|███████▉  | 7705/9708 [00:36<00:10, 190.12batch/s]

Training batches on cuda:  80%|███████▉  | 7725/9708 [00:36<00:10, 187.28batch/s]

Training batches on cuda:  80%|███████▉  | 7748/9708 [00:36<00:09, 197.14batch/s]

Training batches on cuda:  80%|████████  | 7770/9708 [00:37<00:09, 202.38batch/s]

Training batches on cuda:  80%|████████  | 7791/9708 [00:37<00:10, 180.58batch/s]

Training batches on cuda:  80%|████████  | 7811/9708 [00:37<00:10, 183.75batch/s]

Training batches on cuda:  81%|████████  | 7832/9708 [00:37<00:09, 189.72batch/s]

Training batches on cuda:  81%|████████  | 7854/9708 [00:37<00:09, 196.49batch/s]

Training batches on cuda:  81%|████████  | 7877/9708 [00:37<00:08, 203.60batch/s]

Training batches on cuda:  81%|████████▏ | 7901/9708 [00:37<00:08, 211.47batch/s]

Training batches on cuda:  82%|████████▏ | 7924/9708 [00:37<00:08, 214.89batch/s]

Training batches on cuda:  82%|████████▏ | 7946/9708 [00:37<00:08, 214.88batch/s]

Training batches on cuda:  82%|████████▏ | 7968/9708 [00:37<00:08, 215.09batch/s]

Training batches on cuda:  82%|████████▏ | 7991/9708 [00:38<00:07, 216.55batch/s]

Training batches on cuda:  83%|████████▎ | 8013/9708 [00:38<00:07, 214.23batch/s]

Training batches on cuda:  83%|████████▎ | 8035/9708 [00:38<00:07, 210.62batch/s]

Training batches on cuda:  83%|████████▎ | 8057/9708 [00:38<00:07, 208.71batch/s]

Training batches on cuda:  83%|████████▎ | 8078/9708 [00:38<00:07, 209.05batch/s]

Training batches on cuda:  83%|████████▎ | 8099/9708 [00:38<00:07, 208.42batch/s]

Training batches on cuda:  84%|████████▎ | 8120/9708 [00:38<00:07, 207.03batch/s]

Training batches on cuda:  84%|████████▍ | 8142/9708 [00:38<00:07, 210.42batch/s]

Training batches on cuda:  84%|████████▍ | 8164/9708 [00:38<00:07, 203.71batch/s]

Training batches on cuda:  84%|████████▍ | 8186/9708 [00:39<00:07, 205.51batch/s]

Training batches on cuda:  85%|████████▍ | 8208/9708 [00:39<00:07, 207.97batch/s]

Training batches on cuda:  85%|████████▍ | 8230/9708 [00:39<00:07, 208.47batch/s]

Training batches on cuda:  85%|████████▌ | 8252/9708 [00:39<00:06, 211.17batch/s]

Training batches on cuda:  85%|████████▌ | 8275/9708 [00:39<00:06, 214.84batch/s]

Training batches on cuda:  85%|████████▌ | 8299/9708 [00:39<00:06, 221.48batch/s]

Training batches on cuda:  86%|████████▌ | 8326/9708 [00:39<00:05, 232.32batch/s]

Training batches on cuda:  86%|████████▌ | 8352/9708 [00:39<00:05, 237.85batch/s]

Training batches on cuda:  86%|████████▋ | 8376/9708 [00:39<00:05, 237.61batch/s]

Training batches on cuda:  87%|████████▋ | 8400/9708 [00:39<00:05, 228.28batch/s]

Training batches on cuda:  87%|████████▋ | 8423/9708 [00:40<00:06, 212.33batch/s]

Training batches on cuda:  87%|████████▋ | 8445/9708 [00:40<00:05, 210.60batch/s]

Training batches on cuda:  87%|████████▋ | 8470/9708 [00:40<00:05, 220.00batch/s]

Training batches on cuda:  88%|████████▊ | 8495/9708 [00:40<00:05, 227.48batch/s]

Training batches on cuda:  88%|████████▊ | 8519/9708 [00:40<00:05, 227.24batch/s]

Training batches on cuda:  88%|████████▊ | 8542/9708 [00:40<00:05, 223.97batch/s]

Training batches on cuda:  88%|████████▊ | 8565/9708 [00:40<00:05, 215.18batch/s]

Training batches on cuda:  88%|████████▊ | 8587/9708 [00:40<00:05, 209.63batch/s]

Training batches on cuda:  89%|████████▊ | 8609/9708 [00:40<00:05, 200.56batch/s]

Training batches on cuda:  89%|████████▉ | 8632/9708 [00:41<00:05, 206.34batch/s]

Training batches on cuda:  89%|████████▉ | 8655/9708 [00:41<00:04, 210.88batch/s]

Training batches on cuda:  89%|████████▉ | 8678/9708 [00:41<00:04, 216.18batch/s]

Training batches on cuda:  90%|████████▉ | 8702/9708 [00:41<00:04, 221.53batch/s]

Training batches on cuda:  90%|████████▉ | 8726/9708 [00:41<00:04, 225.03batch/s]

Training batches on cuda:  90%|█████████ | 8750/9708 [00:41<00:04, 227.55batch/s]

Training batches on cuda:  90%|█████████ | 8773/9708 [00:41<00:04, 220.44batch/s]

Training batches on cuda:  91%|█████████ | 8796/9708 [00:41<00:04, 212.13batch/s]

Training batches on cuda:  91%|█████████ | 8818/9708 [00:41<00:04, 208.52batch/s]

Training batches on cuda:  91%|█████████ | 8839/9708 [00:42<00:04, 195.95batch/s]

Training batches on cuda:  91%|█████████▏| 8859/9708 [00:42<00:04, 177.52batch/s]

Training batches on cuda:  91%|█████████▏| 8879/9708 [00:42<00:04, 183.13batch/s]

Training batches on cuda:  92%|█████████▏| 8900/9708 [00:42<00:04, 188.38batch/s]

Training batches on cuda:  92%|█████████▏| 8920/9708 [00:42<00:04, 191.53batch/s]

Training batches on cuda:  92%|█████████▏| 8940/9708 [00:42<00:03, 193.23batch/s]

Training batches on cuda:  92%|█████████▏| 8960/9708 [00:42<00:03, 189.89batch/s]

Training batches on cuda:  93%|█████████▎| 8980/9708 [00:42<00:03, 191.65batch/s]

Training batches on cuda:  93%|█████████▎| 9001/9708 [00:42<00:03, 195.43batch/s]

Training batches on cuda:  93%|█████████▎| 9021/9708 [00:43<00:03, 193.88batch/s]

Training batches on cuda:  93%|█████████▎| 9041/9708 [00:43<00:03, 189.60batch/s]

Training batches on cuda:  93%|█████████▎| 9062/9708 [00:43<00:03, 194.30batch/s]

Training batches on cuda:  94%|█████████▎| 9082/9708 [00:43<00:03, 195.32batch/s]

Training batches on cuda:  94%|█████████▍| 9103/9708 [00:43<00:03, 197.56batch/s]

Training batches on cuda:  94%|█████████▍| 9124/9708 [00:43<00:02, 200.18batch/s]

Training batches on cuda:  94%|█████████▍| 9146/9708 [00:43<00:02, 202.86batch/s]

Training batches on cuda:  94%|█████████▍| 9167/9708 [00:43<00:02, 203.91batch/s]

Training batches on cuda:  95%|█████████▍| 9188/9708 [00:43<00:02, 200.46batch/s]

Training batches on cuda:  95%|█████████▍| 9209/9708 [00:43<00:02, 197.28batch/s]

Training batches on cuda:  95%|█████████▌| 9229/9708 [00:44<00:02, 196.42batch/s]

Training batches on cuda:  95%|█████████▌| 9249/9708 [00:44<00:02, 197.38batch/s]

Training batches on cuda:  95%|█████████▌| 9270/9708 [00:44<00:02, 199.16batch/s]

Training batches on cuda:  96%|█████████▌| 9293/9708 [00:44<00:02, 205.13batch/s]

Training batches on cuda:  96%|█████████▌| 9314/9708 [00:44<00:01, 206.52batch/s]

Training batches on cuda:  96%|█████████▌| 9335/9708 [00:44<00:01, 203.88batch/s]

Training batches on cuda:  96%|█████████▋| 9356/9708 [00:44<00:01, 201.39batch/s]

Training batches on cuda:  97%|█████████▋| 9378/9708 [00:44<00:01, 205.18batch/s]

Training batches on cuda:  97%|█████████▋| 9399/9708 [00:44<00:01, 203.91batch/s]

Training batches on cuda:  97%|█████████▋| 9420/9708 [00:44<00:01, 200.79batch/s]

Training batches on cuda:  97%|█████████▋| 9441/9708 [00:45<00:01, 202.09batch/s]

Training batches on cuda:  97%|█████████▋| 9462/9708 [00:45<00:01, 203.24batch/s]

Training batches on cuda:  98%|█████████▊| 9484/9708 [00:45<00:01, 205.63batch/s]

Training batches on cuda:  98%|█████████▊| 9506/9708 [00:45<00:00, 207.82batch/s]

Training batches on cuda:  98%|█████████▊| 9527/9708 [00:45<00:00, 208.02batch/s]

Training batches on cuda:  98%|█████████▊| 9550/9708 [00:45<00:00, 210.58batch/s]

Training batches on cuda:  99%|█████████▊| 9572/9708 [00:45<00:00, 207.43batch/s]

Training batches on cuda:  99%|█████████▉| 9596/9708 [00:45<00:00, 215.80batch/s]

Training batches on cuda:  99%|█████████▉| 9620/9708 [00:45<00:00, 220.39batch/s]

Training batches on cuda:  99%|█████████▉| 9645/9708 [00:46<00:00, 227.25batch/s]

Training batches on cuda: 100%|█████████▉| 9669/9708 [00:46<00:00, 229.43batch/s]

Training batches on cuda: 100%|█████████▉| 9694/9708 [00:46<00:00, 234.02batch/s]

                                                                                 
Training epochs on cuda:  50%|█████     | 5/10 [04:37<03:50, 46.04s/epoch, loss=0.096, prev_loss=0.096]
Training epochs on cuda:  60%|██████    | 6/10 [04:37<03:04, 46.18s/epoch, loss=0.096, prev_loss=0.096]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 20/9708 [00:00<00:50, 193.07batch/s]

Training batches on cuda:   0%|          | 44/9708 [00:00<00:47, 204.38batch/s]

Training batches on cuda:   1%|          | 68/9708 [00:00<00:45, 211.88batch/s]

Training batches on cuda:   1%|          | 91/9708 [00:00<00:44, 216.22batch/s]

Training batches on cuda:   1%|          | 114/9708 [00:00<00:44, 217.62batch/s]

Training batches on cuda:   1%|▏         | 138/9708 [00:00<00:43, 221.32batch/s]

Training batches on cuda:   2%|▏         | 158/9708 [00:00<00:48, 195.77batch/s]

Training batches on cuda:   2%|▏         | 181/9708 [00:00<00:46, 204.02batch/s]

Training batches on cuda:   2%|▏         | 205/9708 [00:00<00:44, 211.55batch/s]

Training batches on cuda:   2%|▏         | 227/9708 [00:01<00:44, 212.23batch/s]

Training batches on cuda:   3%|▎         | 248/9708 [00:01<00:46, 203.33batch/s]

Training batches on cuda:   3%|▎         | 269/9708 [00:01<00:47, 197.92batch/s]

Training batches on cuda:   3%|▎         | 289/9708 [00:01<00:47, 197.66batch/s]

Training batches on cuda:   3%|▎         | 309/9708 [00:01<00:48, 193.32batch/s]

Training batches on cuda:   3%|▎         | 329/9708 [00:01<00:49, 190.18batch/s]

Training batches on cuda:   4%|▎         | 349/9708 [00:01<00:49, 189.90batch/s]

Training batches on cuda:   4%|▍         | 369/9708 [00:01<00:48, 192.15batch/s]

Training batches on cuda:   4%|▍         | 390/9708 [00:01<00:47, 195.28batch/s]

Training batches on cuda:   4%|▍         | 411/9708 [00:02<00:46, 198.11batch/s]

Training batches on cuda:   4%|▍         | 432/9708 [00:02<00:46, 200.26batch/s]

Training batches on cuda:   5%|▍         | 453/9708 [00:02<00:45, 201.31batch/s]

Training batches on cuda:   5%|▍         | 474/9708 [00:02<00:45, 202.30batch/s]

Training batches on cuda:   5%|▌         | 496/9708 [00:02<00:44, 206.95batch/s]

Training batches on cuda:   5%|▌         | 517/9708 [00:02<00:46, 197.76batch/s]

Training batches on cuda:   6%|▌         | 537/9708 [00:02<00:46, 196.98batch/s]

Training batches on cuda:   6%|▌         | 560/9708 [00:02<00:44, 205.01batch/s]

Training batches on cuda:   6%|▌         | 582/9708 [00:02<00:43, 208.93batch/s]

Training batches on cuda:   6%|▌         | 604/9708 [00:02<00:43, 209.36batch/s]

Training batches on cuda:   6%|▋         | 626/9708 [00:03<00:43, 207.74batch/s]

Training batches on cuda:   7%|▋         | 647/9708 [00:03<00:45, 199.23batch/s]

Training batches on cuda:   7%|▋         | 668/9708 [00:03<00:45, 198.86batch/s]

Training batches on cuda:   7%|▋         | 689/9708 [00:03<00:44, 201.21batch/s]

Training batches on cuda:   7%|▋         | 710/9708 [00:03<00:44, 203.37batch/s]

Training batches on cuda:   8%|▊         | 731/9708 [00:03<00:43, 204.48batch/s]

Training batches on cuda:   8%|▊         | 753/9708 [00:03<00:43, 207.93batch/s]

Training batches on cuda:   8%|▊         | 775/9708 [00:03<00:42, 209.35batch/s]

Training batches on cuda:   8%|▊         | 796/9708 [00:03<00:45, 198.00batch/s]

Training batches on cuda:   8%|▊         | 817/9708 [00:04<00:44, 199.34batch/s]

Training batches on cuda:   9%|▊         | 838/9708 [00:04<00:44, 199.87batch/s]

Training batches on cuda:   9%|▉         | 859/9708 [00:04<00:43, 202.65batch/s]

Training batches on cuda:   9%|▉         | 880/9708 [00:04<00:43, 203.42batch/s]

Training batches on cuda:   9%|▉         | 901/9708 [00:04<00:43, 204.69batch/s]

Training batches on cuda:   9%|▉         | 922/9708 [00:04<00:43, 203.00batch/s]

Training batches on cuda:  10%|▉         | 943/9708 [00:04<00:43, 199.59batch/s]

Training batches on cuda:  10%|▉         | 964/9708 [00:04<00:43, 200.86batch/s]

Training batches on cuda:  10%|█         | 985/9708 [00:04<00:43, 202.59batch/s]

Training batches on cuda:  10%|█         | 1006/9708 [00:04<00:43, 198.33batch/s]

Training batches on cuda:  11%|█         | 1027/9708 [00:05<00:43, 200.21batch/s]

Training batches on cuda:  11%|█         | 1048/9708 [00:05<00:42, 202.97batch/s]

Training batches on cuda:  11%|█         | 1071/9708 [00:05<00:41, 209.40batch/s]

Training batches on cuda:  11%|█▏        | 1094/9708 [00:05<00:40, 213.49batch/s]

Training batches on cuda:  12%|█▏        | 1117/9708 [00:05<00:39, 216.42batch/s]

Training batches on cuda:  12%|█▏        | 1139/9708 [00:05<00:39, 216.43batch/s]

Training batches on cuda:  12%|█▏        | 1161/9708 [00:05<00:39, 215.78batch/s]

Training batches on cuda:  12%|█▏        | 1183/9708 [00:05<00:43, 194.50batch/s]

Training batches on cuda:  12%|█▏        | 1205/9708 [00:05<00:42, 199.89batch/s]

Training batches on cuda:  13%|█▎        | 1226/9708 [00:06<00:41, 202.56batch/s]

Training batches on cuda:  13%|█▎        | 1247/9708 [00:06<00:41, 201.87batch/s]

Training batches on cuda:  13%|█▎        | 1269/9708 [00:06<00:41, 205.52batch/s]

Training batches on cuda:  13%|█▎        | 1292/9708 [00:06<00:40, 210.03batch/s]

Training batches on cuda:  14%|█▎        | 1314/9708 [00:06<00:39, 210.30batch/s]

Training batches on cuda:  14%|█▍        | 1336/9708 [00:06<00:39, 211.17batch/s]

Training batches on cuda:  14%|█▍        | 1358/9708 [00:06<00:39, 213.28batch/s]

Training batches on cuda:  14%|█▍        | 1385/9708 [00:06<00:36, 227.47batch/s]

Training batches on cuda:  15%|█▍        | 1412/9708 [00:06<00:34, 238.00batch/s]

Training batches on cuda:  15%|█▍        | 1439/9708 [00:06<00:33, 246.36batch/s]

Training batches on cuda:  15%|█▌        | 1467/9708 [00:07<00:32, 253.04batch/s]

Training batches on cuda:  15%|█▌        | 1495/9708 [00:07<00:31, 258.10batch/s]

Training batches on cuda:  16%|█▌        | 1522/9708 [00:07<00:31, 260.98batch/s]

Training batches on cuda:  16%|█▌        | 1549/9708 [00:07<00:31, 262.64batch/s]

Training batches on cuda:  16%|█▌        | 1576/9708 [00:07<00:30, 263.44batch/s]

Training batches on cuda:  17%|█▋        | 1603/9708 [00:07<00:30, 264.02batch/s]

Training batches on cuda:  17%|█▋        | 1630/9708 [00:07<00:30, 264.57batch/s]

Training batches on cuda:  17%|█▋        | 1657/9708 [00:07<00:30, 265.14batch/s]

Training batches on cuda:  17%|█▋        | 1684/9708 [00:07<00:30, 265.66batch/s]

Training batches on cuda:  18%|█▊        | 1711/9708 [00:07<00:31, 254.62batch/s]

Training batches on cuda:  18%|█▊        | 1737/9708 [00:08<00:32, 245.72batch/s]

Training batches on cuda:  18%|█▊        | 1762/9708 [00:08<00:33, 233.89batch/s]

Training batches on cuda:  18%|█▊        | 1786/9708 [00:08<00:34, 228.11batch/s]

Training batches on cuda:  19%|█▊        | 1809/9708 [00:08<00:35, 224.65batch/s]

Training batches on cuda:  19%|█▉        | 1832/9708 [00:08<00:37, 212.47batch/s]

Training batches on cuda:  19%|█▉        | 1854/9708 [00:08<00:37, 210.26batch/s]

Training batches on cuda:  19%|█▉        | 1876/9708 [00:08<00:37, 208.17batch/s]

Training batches on cuda:  20%|█▉        | 1897/9708 [00:08<00:37, 208.67batch/s]

Training batches on cuda:  20%|█▉        | 1918/9708 [00:08<00:37, 207.17batch/s]

Training batches on cuda:  20%|█▉        | 1940/9708 [00:09<00:37, 207.92batch/s]

Training batches on cuda:  20%|██        | 1961/9708 [00:09<00:38, 200.63batch/s]

Training batches on cuda:  20%|██        | 1982/9708 [00:09<00:39, 197.62batch/s]

Training batches on cuda:  21%|██        | 2002/9708 [00:09<00:39, 192.88batch/s]

Training batches on cuda:  21%|██        | 2022/9708 [00:09<00:40, 190.62batch/s]

Training batches on cuda:  21%|██        | 2042/9708 [00:09<00:40, 189.34batch/s]

Training batches on cuda:  21%|██▏       | 2066/9708 [00:09<00:38, 199.97batch/s]

Training batches on cuda:  21%|██▏       | 2087/9708 [00:09<00:37, 201.86batch/s]

Training batches on cuda:  22%|██▏       | 2110/9708 [00:09<00:36, 207.07batch/s]

Training batches on cuda:  22%|██▏       | 2133/9708 [00:10<00:35, 211.87batch/s]

Training batches on cuda:  22%|██▏       | 2155/9708 [00:10<00:35, 211.08batch/s]

Training batches on cuda:  22%|██▏       | 2178/9708 [00:10<00:35, 214.69batch/s]

Training batches on cuda:  23%|██▎       | 2201/9708 [00:10<00:34, 218.37batch/s]

Training batches on cuda:  23%|██▎       | 2223/9708 [00:10<00:34, 217.91batch/s]

Training batches on cuda:  23%|██▎       | 2245/9708 [00:10<00:34, 218.39batch/s]

Training batches on cuda:  23%|██▎       | 2268/9708 [00:10<00:33, 220.05batch/s]

Training batches on cuda:  24%|██▎       | 2291/9708 [00:10<00:33, 219.82batch/s]

Training batches on cuda:  24%|██▍       | 2314/9708 [00:10<00:37, 198.70batch/s]

Training batches on cuda:  24%|██▍       | 2337/9708 [00:10<00:35, 205.76batch/s]

Training batches on cuda:  24%|██▍       | 2359/9708 [00:11<00:35, 207.23batch/s]

Training batches on cuda:  25%|██▍       | 2381/9708 [00:11<00:34, 209.68batch/s]

Training batches on cuda:  25%|██▍       | 2403/9708 [00:11<00:34, 210.61batch/s]

Training batches on cuda:  25%|██▍       | 2425/9708 [00:11<00:34, 212.10batch/s]

Training batches on cuda:  25%|██▌       | 2447/9708 [00:11<00:34, 213.35batch/s]

Training batches on cuda:  25%|██▌       | 2470/9708 [00:11<00:33, 216.98batch/s]

Training batches on cuda:  26%|██▌       | 2494/9708 [00:11<00:32, 220.75batch/s]

Training batches on cuda:  26%|██▌       | 2517/9708 [00:11<00:32, 223.27batch/s]

Training batches on cuda:  26%|██▌       | 2540/9708 [00:11<00:31, 224.82batch/s]

Training batches on cuda:  26%|██▋       | 2563/9708 [00:12<00:31, 224.25batch/s]

Training batches on cuda:  27%|██▋       | 2586/9708 [00:12<00:32, 218.08batch/s]

Training batches on cuda:  27%|██▋       | 2608/9708 [00:12<00:32, 216.03batch/s]

Training batches on cuda:  27%|██▋       | 2630/9708 [00:12<00:32, 217.12batch/s]

Training batches on cuda:  27%|██▋       | 2652/9708 [00:12<00:32, 217.11batch/s]

Training batches on cuda:  28%|██▊       | 2674/9708 [00:12<00:32, 215.70batch/s]

Training batches on cuda:  28%|██▊       | 2696/9708 [00:12<00:32, 216.36batch/s]

Training batches on cuda:  28%|██▊       | 2718/9708 [00:12<00:32, 215.05batch/s]

Training batches on cuda:  28%|██▊       | 2742/9708 [00:12<00:31, 220.16batch/s]

Training batches on cuda:  28%|██▊       | 2765/9708 [00:12<00:31, 222.01batch/s]

Training batches on cuda:  29%|██▊       | 2788/9708 [00:13<00:31, 222.92batch/s]

Training batches on cuda:  29%|██▉       | 2811/9708 [00:13<00:30, 223.74batch/s]

Training batches on cuda:  29%|██▉       | 2834/9708 [00:13<00:30, 222.40batch/s]

Training batches on cuda:  29%|██▉       | 2857/9708 [00:13<00:32, 209.12batch/s]

Training batches on cuda:  30%|██▉       | 2879/9708 [00:13<00:32, 211.16batch/s]

Training batches on cuda:  30%|██▉       | 2901/9708 [00:13<00:34, 197.25batch/s]

Training batches on cuda:  30%|███       | 2922/9708 [00:13<00:34, 198.80batch/s]

Training batches on cuda:  30%|███       | 2944/9708 [00:13<00:33, 202.45batch/s]

Training batches on cuda:  31%|███       | 2967/9708 [00:13<00:32, 208.39batch/s]

Training batches on cuda:  31%|███       | 2988/9708 [00:14<00:32, 208.31batch/s]

Training batches on cuda:  31%|███       | 3009/9708 [00:14<00:32, 208.31batch/s]

Training batches on cuda:  31%|███       | 3030/9708 [00:14<00:33, 199.04batch/s]

Training batches on cuda:  31%|███▏      | 3051/9708 [00:14<00:34, 194.86batch/s]

Training batches on cuda:  32%|███▏      | 3075/9708 [00:14<00:32, 204.36batch/s]

Training batches on cuda:  32%|███▏      | 3096/9708 [00:14<00:33, 197.64batch/s]

Training batches on cuda:  32%|███▏      | 3120/9708 [00:14<00:31, 206.26batch/s]

Training batches on cuda:  32%|███▏      | 3142/9708 [00:14<00:31, 208.18batch/s]

Training batches on cuda:  33%|███▎      | 3164/9708 [00:14<00:31, 210.56batch/s]

Training batches on cuda:  33%|███▎      | 3186/9708 [00:14<00:30, 212.64batch/s]

Training batches on cuda:  33%|███▎      | 3208/9708 [00:15<00:31, 204.64batch/s]

Training batches on cuda:  33%|███▎      | 3229/9708 [00:15<00:32, 202.04batch/s]

Training batches on cuda:  34%|███▎      | 3253/9708 [00:15<00:30, 210.94batch/s]

Training batches on cuda:  34%|███▍      | 3279/9708 [00:15<00:28, 221.89batch/s]

Training batches on cuda:  34%|███▍      | 3305/9708 [00:15<00:27, 230.75batch/s]

Training batches on cuda:  34%|███▍      | 3329/9708 [00:15<00:27, 231.63batch/s]

Training batches on cuda:  35%|███▍      | 3354/9708 [00:15<00:26, 236.82batch/s]

Training batches on cuda:  35%|███▍      | 3378/9708 [00:15<00:27, 228.58batch/s]

Training batches on cuda:  35%|███▌      | 3402/9708 [00:15<00:30, 203.98batch/s]

Training batches on cuda:  35%|███▌      | 3425/9708 [00:16<00:30, 208.96batch/s]

Training batches on cuda:  36%|███▌      | 3447/9708 [00:16<00:30, 208.22batch/s]

Training batches on cuda:  36%|███▌      | 3469/9708 [00:16<00:30, 205.82batch/s]

Training batches on cuda:  36%|███▌      | 3490/9708 [00:16<00:31, 199.58batch/s]

Training batches on cuda:  36%|███▌      | 3511/9708 [00:16<00:30, 201.85batch/s]

Training batches on cuda:  36%|███▋      | 3532/9708 [00:16<00:30, 202.03batch/s]

Training batches on cuda:  37%|███▋      | 3554/9708 [00:16<00:29, 205.22batch/s]

Training batches on cuda:  37%|███▋      | 3576/9708 [00:16<00:29, 207.14batch/s]

Training batches on cuda:  37%|███▋      | 3597/9708 [00:16<00:29, 206.11batch/s]

Training batches on cuda:  37%|███▋      | 3618/9708 [00:17<00:29, 205.88batch/s]

Training batches on cuda:  38%|███▊      | 3642/9708 [00:17<00:28, 212.60batch/s]

Training batches on cuda:  38%|███▊      | 3666/9708 [00:17<00:27, 218.10batch/s]

Training batches on cuda:  38%|███▊      | 3689/9708 [00:17<00:27, 221.13batch/s]

Training batches on cuda:  38%|███▊      | 3712/9708 [00:17<00:28, 212.29batch/s]

Training batches on cuda:  38%|███▊      | 3734/9708 [00:17<00:28, 210.38batch/s]

Training batches on cuda:  39%|███▊      | 3756/9708 [00:17<00:28, 208.61batch/s]

Training batches on cuda:  39%|███▉      | 3779/9708 [00:17<00:27, 214.25batch/s]

Training batches on cuda:  39%|███▉      | 3803/9708 [00:17<00:26, 219.14batch/s]

Training batches on cuda:  39%|███▉      | 3827/9708 [00:17<00:26, 222.30batch/s]

Training batches on cuda:  40%|███▉      | 3850/9708 [00:18<00:26, 222.22batch/s]

Training batches on cuda:  40%|███▉      | 3873/9708 [00:18<00:26, 221.09batch/s]

Training batches on cuda:  40%|████      | 3896/9708 [00:18<00:26, 221.61batch/s]

Training batches on cuda:  40%|████      | 3919/9708 [00:18<00:26, 216.96batch/s]

Training batches on cuda:  41%|████      | 3942/9708 [00:18<00:26, 219.20batch/s]

Training batches on cuda:  41%|████      | 3965/9708 [00:18<00:26, 220.75batch/s]

Training batches on cuda:  41%|████      | 3988/9708 [00:18<00:25, 221.58batch/s]

Training batches on cuda:  41%|████▏     | 4012/9708 [00:18<00:25, 226.02batch/s]

Training batches on cuda:  42%|████▏     | 4036/9708 [00:18<00:24, 229.30batch/s]

Training batches on cuda:  42%|████▏     | 4059/9708 [00:18<00:24, 228.34batch/s]

Training batches on cuda:  42%|████▏     | 4082/9708 [00:19<00:24, 226.04batch/s]

Training batches on cuda:  42%|████▏     | 4105/9708 [00:19<00:25, 219.79batch/s]

Training batches on cuda:  43%|████▎     | 4128/9708 [00:19<00:26, 208.76batch/s]

Training batches on cuda:  43%|████▎     | 4152/9708 [00:19<00:25, 216.47batch/s]

Training batches on cuda:  43%|████▎     | 4175/9708 [00:19<00:25, 220.31batch/s]

Training batches on cuda:  43%|████▎     | 4198/9708 [00:19<00:24, 222.18batch/s]

Training batches on cuda:  43%|████▎     | 4221/9708 [00:19<00:24, 223.97batch/s]

Training batches on cuda:  44%|████▎     | 4244/9708 [00:19<00:24, 220.48batch/s]

Training batches on cuda:  44%|████▍     | 4267/9708 [00:19<00:25, 216.62batch/s]

Training batches on cuda:  44%|████▍     | 4289/9708 [00:20<00:26, 208.12batch/s]

Training batches on cuda:  44%|████▍     | 4310/9708 [00:20<00:26, 207.25batch/s]

Training batches on cuda:  45%|████▍     | 4331/9708 [00:20<00:26, 204.09batch/s]

Training batches on cuda:  45%|████▍     | 4354/9708 [00:20<00:25, 210.63batch/s]

Training batches on cuda:  45%|████▌     | 4376/9708 [00:20<00:25, 211.96batch/s]

Training batches on cuda:  45%|████▌     | 4399/9708 [00:20<00:24, 214.88batch/s]

Training batches on cuda:  46%|████▌     | 4421/9708 [00:20<00:24, 213.18batch/s]

Training batches on cuda:  46%|████▌     | 4444/9708 [00:20<00:24, 216.89batch/s]

Training batches on cuda:  46%|████▌     | 4467/9708 [00:20<00:23, 220.18batch/s]

Training batches on cuda:  46%|████▋     | 4490/9708 [00:21<00:24, 211.70batch/s]

Training batches on cuda:  46%|████▋     | 4512/9708 [00:21<00:25, 207.08batch/s]

Training batches on cuda:  47%|████▋     | 4534/9708 [00:21<00:24, 208.51batch/s]

Training batches on cuda:  47%|████▋     | 4557/9708 [00:21<00:24, 212.21batch/s]

Training batches on cuda:  47%|████▋     | 4579/9708 [00:21<00:24, 212.61batch/s]

Training batches on cuda:  47%|████▋     | 4602/9708 [00:21<00:23, 217.40batch/s]

Training batches on cuda:  48%|████▊     | 4625/9708 [00:21<00:23, 218.26batch/s]

Training batches on cuda:  48%|████▊     | 4648/9708 [00:21<00:23, 219.49batch/s]

Training batches on cuda:  48%|████▊     | 4671/9708 [00:21<00:22, 221.88batch/s]

Training batches on cuda:  48%|████▊     | 4694/9708 [00:21<00:22, 222.84batch/s]

Training batches on cuda:  49%|████▊     | 4717/9708 [00:22<00:24, 206.05batch/s]

Training batches on cuda:  49%|████▉     | 4738/9708 [00:22<00:24, 205.56batch/s]

Training batches on cuda:  49%|████▉     | 4760/9708 [00:22<00:23, 207.53batch/s]

Training batches on cuda:  49%|████▉     | 4782/9708 [00:22<00:23, 209.74batch/s]

Training batches on cuda:  49%|████▉     | 4804/9708 [00:22<00:23, 210.81batch/s]

Training batches on cuda:  50%|████▉     | 4826/9708 [00:22<00:23, 210.93batch/s]

Training batches on cuda:  50%|████▉     | 4848/9708 [00:22<00:23, 209.31batch/s]

Training batches on cuda:  50%|█████     | 4869/9708 [00:22<00:23, 209.50batch/s]

Training batches on cuda:  50%|█████     | 4891/9708 [00:22<00:22, 210.09batch/s]

Training batches on cuda:  51%|█████     | 4913/9708 [00:23<00:22, 209.81batch/s]

Training batches on cuda:  51%|█████     | 4935/9708 [00:23<00:22, 210.10batch/s]

Training batches on cuda:  51%|█████     | 4957/9708 [00:23<00:22, 210.26batch/s]

Training batches on cuda:  51%|█████▏    | 4979/9708 [00:23<00:22, 210.68batch/s]

Training batches on cuda:  52%|█████▏    | 5001/9708 [00:23<00:22, 205.33batch/s]

Training batches on cuda:  52%|█████▏    | 5022/9708 [00:23<00:22, 205.25batch/s]

Training batches on cuda:  52%|█████▏    | 5043/9708 [00:23<00:23, 202.79batch/s]

Training batches on cuda:  52%|█████▏    | 5064/9708 [00:23<00:22, 203.91batch/s]

Training batches on cuda:  52%|█████▏    | 5085/9708 [00:23<00:22, 205.45batch/s]

Training batches on cuda:  53%|█████▎    | 5110/9708 [00:23<00:21, 214.78batch/s]

Training batches on cuda:  53%|█████▎    | 5135/9708 [00:24<00:20, 222.04batch/s]

Training batches on cuda:  53%|█████▎    | 5158/9708 [00:24<00:20, 221.73batch/s]

Training batches on cuda:  53%|█████▎    | 5182/9708 [00:24<00:20, 225.11batch/s]

Training batches on cuda:  54%|█████▎    | 5206/9708 [00:24<00:19, 229.03batch/s]

Training batches on cuda:  54%|█████▍    | 5231/9708 [00:24<00:19, 232.35batch/s]

Training batches on cuda:  54%|█████▍    | 5255/9708 [00:24<00:19, 231.60batch/s]

Training batches on cuda:  54%|█████▍    | 5279/9708 [00:24<00:19, 230.91batch/s]

Training batches on cuda:  55%|█████▍    | 5305/9708 [00:24<00:18, 236.22batch/s]

Training batches on cuda:  55%|█████▍    | 5330/9708 [00:24<00:18, 238.67batch/s]

Training batches on cuda:  55%|█████▌    | 5354/9708 [00:24<00:18, 235.98batch/s]

Training batches on cuda:  55%|█████▌    | 5378/9708 [00:25<00:18, 229.47batch/s]

Training batches on cuda:  56%|█████▌    | 5402/9708 [00:25<00:19, 217.85batch/s]

Training batches on cuda:  56%|█████▌    | 5424/9708 [00:25<00:19, 214.58batch/s]

Training batches on cuda:  56%|█████▌    | 5446/9708 [00:25<00:20, 204.95batch/s]

Training batches on cuda:  56%|█████▋    | 5467/9708 [00:25<00:21, 198.06batch/s]

Training batches on cuda:  57%|█████▋    | 5487/9708 [00:25<00:21, 194.55batch/s]

Training batches on cuda:  57%|█████▋    | 5508/9708 [00:25<00:21, 198.35batch/s]

Training batches on cuda:  57%|█████▋    | 5529/9708 [00:25<00:20, 200.10batch/s]

Training batches on cuda:  57%|█████▋    | 5551/9708 [00:25<00:20, 204.63batch/s]

Training batches on cuda:  57%|█████▋    | 5572/9708 [00:26<00:22, 182.38batch/s]

Training batches on cuda:  58%|█████▊    | 5591/9708 [00:26<00:22, 183.47batch/s]

Training batches on cuda:  58%|█████▊    | 5613/9708 [00:26<00:21, 192.59batch/s]

Training batches on cuda:  58%|█████▊    | 5634/9708 [00:26<00:20, 196.79batch/s]

Training batches on cuda:  58%|█████▊    | 5654/9708 [00:26<00:20, 196.23batch/s]

Training batches on cuda:  58%|█████▊    | 5676/9708 [00:26<00:20, 200.25batch/s]

Training batches on cuda:  59%|█████▊    | 5697/9708 [00:26<00:20, 198.88batch/s]

Training batches on cuda:  59%|█████▉    | 5718/9708 [00:26<00:20, 196.23batch/s]

Training batches on cuda:  59%|█████▉    | 5739/9708 [00:26<00:20, 197.19batch/s]

Training batches on cuda:  59%|█████▉    | 5760/9708 [00:27<00:19, 198.82batch/s]

Training batches on cuda:  60%|█████▉    | 5780/9708 [00:27<00:19, 196.87batch/s]

Training batches on cuda:  60%|█████▉    | 5801/9708 [00:27<00:19, 198.63batch/s]

Training batches on cuda:  60%|█████▉    | 5823/9708 [00:27<00:19, 203.17batch/s]

Training batches on cuda:  60%|██████    | 5844/9708 [00:27<00:18, 205.15batch/s]

Training batches on cuda:  60%|██████    | 5866/9708 [00:27<00:18, 207.35batch/s]

Training batches on cuda:  61%|██████    | 5887/9708 [00:27<00:18, 207.53batch/s]

Training batches on cuda:  61%|██████    | 5908/9708 [00:27<00:18, 204.95batch/s]

Training batches on cuda:  61%|██████    | 5929/9708 [00:27<00:18, 205.37batch/s]

Training batches on cuda:  61%|██████▏   | 5951/9708 [00:27<00:18, 207.56batch/s]

Training batches on cuda:  62%|██████▏   | 5972/9708 [00:28<00:18, 206.72batch/s]

Training batches on cuda:  62%|██████▏   | 5993/9708 [00:28<00:17, 207.29batch/s]

Training batches on cuda:  62%|██████▏   | 6016/9708 [00:28<00:17, 211.20batch/s]

Training batches on cuda:  62%|██████▏   | 6039/9708 [00:28<00:16, 216.45batch/s]

Training batches on cuda:  62%|██████▏   | 6062/9708 [00:28<00:16, 219.72batch/s]

Training batches on cuda:  63%|██████▎   | 6085/9708 [00:28<00:16, 219.87batch/s]

Training batches on cuda:  63%|██████▎   | 6109/9708 [00:28<00:16, 223.02batch/s]

Training batches on cuda:  63%|██████▎   | 6132/9708 [00:28<00:16, 222.56batch/s]

Training batches on cuda:  63%|██████▎   | 6155/9708 [00:28<00:16, 218.46batch/s]

Training batches on cuda:  64%|██████▎   | 6178/9708 [00:29<00:15, 221.29batch/s]

Training batches on cuda:  64%|██████▍   | 6201/9708 [00:29<00:15, 222.25batch/s]

Training batches on cuda:  64%|██████▍   | 6224/9708 [00:29<00:15, 219.74batch/s]

Training batches on cuda:  64%|██████▍   | 6247/9708 [00:29<00:15, 221.72batch/s]

Training batches on cuda:  65%|██████▍   | 6271/9708 [00:29<00:15, 225.25batch/s]

Training batches on cuda:  65%|██████▍   | 6296/9708 [00:29<00:14, 229.61batch/s]

Training batches on cuda:  65%|██████▌   | 6320/9708 [00:29<00:14, 226.10batch/s]

Training batches on cuda:  65%|██████▌   | 6343/9708 [00:29<00:15, 222.17batch/s]

Training batches on cuda:  66%|██████▌   | 6366/9708 [00:29<00:15, 218.11batch/s]

Training batches on cuda:  66%|██████▌   | 6388/9708 [00:29<00:15, 215.27batch/s]

Training batches on cuda:  66%|██████▌   | 6410/9708 [00:30<00:15, 209.97batch/s]

Training batches on cuda:  66%|██████▋   | 6432/9708 [00:30<00:15, 207.73batch/s]

Training batches on cuda:  66%|██████▋   | 6453/9708 [00:30<00:15, 205.84batch/s]

Training batches on cuda:  67%|██████▋   | 6474/9708 [00:30<00:16, 201.25batch/s]

Training batches on cuda:  67%|██████▋   | 6495/9708 [00:30<00:15, 203.71batch/s]

Training batches on cuda:  67%|██████▋   | 6516/9708 [00:30<00:15, 203.29batch/s]

Training batches on cuda:  67%|██████▋   | 6537/9708 [00:30<00:15, 202.62batch/s]

Training batches on cuda:  68%|██████▊   | 6558/9708 [00:30<00:15, 204.44batch/s]

Training batches on cuda:  68%|██████▊   | 6579/9708 [00:30<00:15, 203.25batch/s]

Training batches on cuda:  68%|██████▊   | 6600/9708 [00:31<00:15, 203.33batch/s]

Training batches on cuda:  68%|██████▊   | 6621/9708 [00:31<00:16, 184.79batch/s]

Training batches on cuda:  68%|██████▊   | 6641/9708 [00:31<00:16, 187.25batch/s]

Training batches on cuda:  69%|██████▊   | 6660/9708 [00:31<00:16, 184.40batch/s]

Training batches on cuda:  69%|██████▉   | 6682/9708 [00:31<00:15, 192.06batch/s]

Training batches on cuda:  69%|██████▉   | 6702/9708 [00:31<00:15, 191.21batch/s]

Training batches on cuda:  69%|██████▉   | 6724/9708 [00:31<00:15, 198.32batch/s]

Training batches on cuda:  69%|██████▉   | 6745/9708 [00:31<00:15, 195.56batch/s]

Training batches on cuda:  70%|██████▉   | 6768/9708 [00:31<00:14, 203.58batch/s]

Training batches on cuda:  70%|██████▉   | 6789/9708 [00:31<00:14, 198.85batch/s]

Training batches on cuda:  70%|███████   | 6811/9708 [00:32<00:14, 203.98batch/s]

Training batches on cuda:  70%|███████   | 6832/9708 [00:32<00:14, 194.32batch/s]

Training batches on cuda:  71%|███████   | 6852/9708 [00:32<00:14, 195.65batch/s]

Training batches on cuda:  71%|███████   | 6872/9708 [00:32<00:14, 191.00batch/s]

Training batches on cuda:  71%|███████   | 6893/9708 [00:32<00:14, 195.00batch/s]

Training batches on cuda:  71%|███████   | 6914/9708 [00:32<00:14, 197.56batch/s]

Training batches on cuda:  71%|███████▏  | 6935/9708 [00:32<00:13, 200.15batch/s]

Training batches on cuda:  72%|███████▏  | 6959/9708 [00:32<00:13, 209.70batch/s]

Training batches on cuda:  72%|███████▏  | 6984/9708 [00:32<00:12, 220.22batch/s]

Training batches on cuda:  72%|███████▏  | 7009/9708 [00:33<00:11, 227.01batch/s]

Training batches on cuda:  72%|███████▏  | 7034/9708 [00:33<00:11, 233.04batch/s]

Training batches on cuda:  73%|███████▎  | 7059/9708 [00:33<00:11, 236.88batch/s]

Training batches on cuda:  73%|███████▎  | 7083/9708 [00:33<00:11, 237.18batch/s]

Training batches on cuda:  73%|███████▎  | 7107/9708 [00:33<00:11, 234.00batch/s]

Training batches on cuda:  73%|███████▎  | 7131/9708 [00:33<00:11, 222.19batch/s]

Training batches on cuda:  74%|███████▎  | 7154/9708 [00:33<00:11, 218.39batch/s]

Training batches on cuda:  74%|███████▍  | 7176/9708 [00:33<00:12, 205.44batch/s]

Training batches on cuda:  74%|███████▍  | 7197/9708 [00:33<00:12, 198.45batch/s]

Training batches on cuda:  74%|███████▍  | 7218/9708 [00:34<00:12, 195.45batch/s]

Training batches on cuda:  75%|███████▍  | 7239/9708 [00:34<00:12, 197.60batch/s]

Training batches on cuda:  75%|███████▍  | 7262/9708 [00:34<00:11, 204.21batch/s]

Training batches on cuda:  75%|███████▌  | 7283/9708 [00:34<00:11, 205.39batch/s]

Training batches on cuda:  75%|███████▌  | 7305/9708 [00:34<00:11, 208.73batch/s]

Training batches on cuda:  75%|███████▌  | 7327/9708 [00:34<00:11, 211.22batch/s]

Training batches on cuda:  76%|███████▌  | 7350/9708 [00:34<00:11, 213.80batch/s]

Training batches on cuda:  76%|███████▌  | 7372/9708 [00:34<00:10, 214.73batch/s]

Training batches on cuda:  76%|███████▌  | 7394/9708 [00:34<00:10, 214.15batch/s]

Training batches on cuda:  76%|███████▋  | 7418/9708 [00:34<00:10, 220.09batch/s]

Training batches on cuda:  77%|███████▋  | 7442/9708 [00:35<00:10, 223.20batch/s]

Training batches on cuda:  77%|███████▋  | 7466/9708 [00:35<00:09, 226.60batch/s]

Training batches on cuda:  77%|███████▋  | 7489/9708 [00:35<00:09, 226.91batch/s]

Training batches on cuda:  77%|███████▋  | 7512/9708 [00:35<00:09, 221.46batch/s]

Training batches on cuda:  78%|███████▊  | 7535/9708 [00:35<00:09, 220.18batch/s]

Training batches on cuda:  78%|███████▊  | 7558/9708 [00:35<00:09, 222.35batch/s]

Training batches on cuda:  78%|███████▊  | 7582/9708 [00:35<00:09, 227.17batch/s]

Training batches on cuda:  78%|███████▊  | 7605/9708 [00:35<00:09, 222.90batch/s]

Training batches on cuda:  79%|███████▊  | 7628/9708 [00:35<00:09, 219.87batch/s]

Training batches on cuda:  79%|███████▉  | 7651/9708 [00:36<00:10, 204.33batch/s]

Training batches on cuda:  79%|███████▉  | 7672/9708 [00:36<00:11, 170.94batch/s]

Training batches on cuda:  79%|███████▉  | 7694/9708 [00:36<00:11, 182.28batch/s]

Training batches on cuda:  79%|███████▉  | 7714/9708 [00:36<00:10, 183.47batch/s]

Training batches on cuda:  80%|███████▉  | 7736/9708 [00:36<00:10, 191.66batch/s]

Training batches on cuda:  80%|███████▉  | 7756/9708 [00:36<00:10, 190.43batch/s]

Training batches on cuda:  80%|████████  | 7777/9708 [00:36<00:09, 194.60batch/s]

Training batches on cuda:  80%|████████  | 7797/9708 [00:36<00:10, 191.09batch/s]

Training batches on cuda:  81%|████████  | 7819/9708 [00:36<00:09, 196.84batch/s]

Training batches on cuda:  81%|████████  | 7839/9708 [00:37<00:09, 191.19batch/s]

Training batches on cuda:  81%|████████  | 7861/9708 [00:37<00:09, 196.99batch/s]

Training batches on cuda:  81%|████████  | 7883/9708 [00:37<00:09, 201.07batch/s]

Training batches on cuda:  81%|████████▏ | 7905/9708 [00:37<00:08, 205.16batch/s]

Training batches on cuda:  82%|████████▏ | 7927/9708 [00:37<00:08, 206.72batch/s]

Training batches on cuda:  82%|████████▏ | 7949/9708 [00:37<00:08, 208.82batch/s]

Training batches on cuda:  82%|████████▏ | 7970/9708 [00:37<00:08, 206.24batch/s]

Training batches on cuda:  82%|████████▏ | 7992/9708 [00:37<00:08, 209.58batch/s]

Training batches on cuda:  83%|████████▎ | 8016/9708 [00:37<00:07, 217.10batch/s]

Training batches on cuda:  83%|████████▎ | 8040/9708 [00:37<00:07, 220.99batch/s]

Training batches on cuda:  83%|████████▎ | 8064/9708 [00:38<00:07, 223.74batch/s]

Training batches on cuda:  83%|████████▎ | 8087/9708 [00:38<00:07, 214.84batch/s]

Training batches on cuda:  84%|████████▎ | 8109/9708 [00:38<00:07, 209.00batch/s]

Training batches on cuda:  84%|████████▍ | 8131/9708 [00:38<00:07, 210.65batch/s]

Training batches on cuda:  84%|████████▍ | 8153/9708 [00:38<00:07, 212.07batch/s]

Training batches on cuda:  84%|████████▍ | 8175/9708 [00:38<00:07, 212.42batch/s]

Training batches on cuda:  84%|████████▍ | 8197/9708 [00:38<00:07, 209.49batch/s]

Training batches on cuda:  85%|████████▍ | 8219/9708 [00:38<00:07, 212.26batch/s]

Training batches on cuda:  85%|████████▍ | 8241/9708 [00:38<00:06, 213.23batch/s]

Training batches on cuda:  85%|████████▌ | 8264/9708 [00:39<00:06, 216.33batch/s]

Training batches on cuda:  85%|████████▌ | 8286/9708 [00:39<00:06, 215.11batch/s]

Training batches on cuda:  86%|████████▌ | 8308/9708 [00:39<00:06, 210.92batch/s]

Training batches on cuda:  86%|████████▌ | 8330/9708 [00:39<00:06, 209.88batch/s]

Training batches on cuda:  86%|████████▌ | 8353/9708 [00:39<00:06, 215.20batch/s]

Training batches on cuda:  86%|████████▋ | 8377/9708 [00:39<00:06, 221.23batch/s]

Training batches on cuda:  87%|████████▋ | 8401/9708 [00:39<00:05, 226.05batch/s]

Training batches on cuda:  87%|████████▋ | 8424/9708 [00:39<00:05, 224.96batch/s]

Training batches on cuda:  87%|████████▋ | 8449/9708 [00:39<00:05, 229.34batch/s]

Training batches on cuda:  87%|████████▋ | 8473/9708 [00:39<00:05, 231.09batch/s]

Training batches on cuda:  88%|████████▊ | 8500/9708 [00:40<00:05, 239.60batch/s]

Training batches on cuda:  88%|████████▊ | 8525/9708 [00:40<00:04, 241.38batch/s]

Training batches on cuda:  88%|████████▊ | 8550/9708 [00:40<00:04, 238.51batch/s]

Training batches on cuda:  88%|████████▊ | 8574/9708 [00:40<00:04, 236.49batch/s]

Training batches on cuda:  89%|████████▊ | 8598/9708 [00:40<00:04, 231.67batch/s]

Training batches on cuda:  89%|████████▉ | 8622/9708 [00:40<00:05, 217.02batch/s]

Training batches on cuda:  89%|████████▉ | 8645/9708 [00:40<00:04, 218.26batch/s]

Training batches on cuda:  89%|████████▉ | 8667/9708 [00:40<00:04, 215.75batch/s]

Training batches on cuda:  90%|████████▉ | 8689/9708 [00:40<00:04, 215.52batch/s]

Training batches on cuda:  90%|████████▉ | 8712/9708 [00:41<00:04, 217.22batch/s]

Training batches on cuda:  90%|████████▉ | 8734/9708 [00:41<00:04, 213.21batch/s]

Training batches on cuda:  90%|█████████ | 8756/9708 [00:41<00:05, 188.83batch/s]

Training batches on cuda:  90%|█████████ | 8778/9708 [00:41<00:04, 195.32batch/s]

Training batches on cuda:  91%|█████████ | 8800/9708 [00:41<00:04, 200.94batch/s]

Training batches on cuda:  91%|█████████ | 8822/9708 [00:41<00:04, 205.63batch/s]

Training batches on cuda:  91%|█████████ | 8846/9708 [00:41<00:04, 213.55batch/s]

Training batches on cuda:  91%|█████████▏| 8870/9708 [00:41<00:03, 219.83batch/s]

Training batches on cuda:  92%|█████████▏| 8893/9708 [00:41<00:03, 222.58batch/s]

Training batches on cuda:  92%|█████████▏| 8918/9708 [00:41<00:03, 227.64batch/s]

Training batches on cuda:  92%|█████████▏| 8941/9708 [00:42<00:03, 225.47batch/s]

Training batches on cuda:  92%|█████████▏| 8964/9708 [00:42<00:03, 222.63batch/s]

Training batches on cuda:  93%|█████████▎| 8987/9708 [00:42<00:03, 219.43batch/s]

Training batches on cuda:  93%|█████████▎| 9010/9708 [00:42<00:03, 219.52batch/s]

Training batches on cuda:  93%|█████████▎| 9033/9708 [00:42<00:03, 217.89batch/s]

Training batches on cuda:  93%|█████████▎| 9055/9708 [00:42<00:03, 215.94batch/s]

Training batches on cuda:  94%|█████████▎| 9077/9708 [00:42<00:02, 214.63batch/s]

Training batches on cuda:  94%|█████████▎| 9099/9708 [00:42<00:02, 215.05batch/s]

Training batches on cuda:  94%|█████████▍| 9122/9708 [00:42<00:02, 217.94batch/s]

Training batches on cuda:  94%|█████████▍| 9146/9708 [00:43<00:02, 223.35batch/s]

Training batches on cuda:  94%|█████████▍| 9170/9708 [00:43<00:02, 227.52batch/s]

Training batches on cuda:  95%|█████████▍| 9194/9708 [00:43<00:02, 231.09batch/s]

Training batches on cuda:  95%|█████████▍| 9218/9708 [00:43<00:02, 232.70batch/s]

Training batches on cuda:  95%|█████████▌| 9242/9708 [00:43<00:01, 233.41batch/s]

Training batches on cuda:  95%|█████████▌| 9266/9708 [00:43<00:01, 232.90batch/s]

Training batches on cuda:  96%|█████████▌| 9290/9708 [00:43<00:01, 233.27batch/s]

Training batches on cuda:  96%|█████████▌| 9314/9708 [00:43<00:01, 233.86batch/s]

Training batches on cuda:  96%|█████████▌| 9338/9708 [00:43<00:01, 234.61batch/s]

Training batches on cuda:  96%|█████████▋| 9362/9708 [00:43<00:01, 234.81batch/s]

Training batches on cuda:  97%|█████████▋| 9386/9708 [00:44<00:01, 231.44batch/s]

Training batches on cuda:  97%|█████████▋| 9410/9708 [00:44<00:01, 229.72batch/s]

Training batches on cuda:  97%|█████████▋| 9433/9708 [00:44<00:01, 225.88batch/s]

Training batches on cuda:  97%|█████████▋| 9459/9708 [00:44<00:01, 232.80batch/s]

Training batches on cuda:  98%|█████████▊| 9483/9708 [00:44<00:00, 231.76batch/s]

Training batches on cuda:  98%|█████████▊| 9507/9708 [00:44<00:00, 218.96batch/s]

Training batches on cuda:  98%|█████████▊| 9531/9708 [00:44<00:00, 224.43batch/s]

Training batches on cuda:  98%|█████████▊| 9554/9708 [00:44<00:00, 219.41batch/s]

Training batches on cuda:  99%|█████████▊| 9577/9708 [00:44<00:00, 218.97batch/s]

Training batches on cuda:  99%|█████████▉| 9600/9708 [00:45<00:00, 221.16batch/s]

Training batches on cuda:  99%|█████████▉| 9627/9708 [00:45<00:00, 232.21batch/s]

Training batches on cuda:  99%|█████████▉| 9652/9708 [00:45<00:00, 235.39batch/s]

Training batches on cuda: 100%|█████████▉| 9677/9708 [00:45<00:00, 237.59batch/s]

Training batches on cuda: 100%|█████████▉| 9702/9708 [00:45<00:00, 239.46batch/s]

                                                                                 
Training epochs on cuda:  60%|██████    | 6/10 [05:23<03:04, 46.18s/epoch, loss=0.096, prev_loss=0.096]
Training epochs on cuda:  70%|███████   | 7/10 [05:23<02:18, 46.03s/epoch, loss=0.096, prev_loss=0.096]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 22/9708 [00:00<00:44, 218.55batch/s]

Training batches on cuda:   1%|          | 50/9708 [00:00<00:41, 232.39batch/s]

Training batches on cuda:   1%|          | 74/9708 [00:00<00:41, 233.21batch/s]

Training batches on cuda:   1%|          | 98/9708 [00:00<00:41, 233.21batch/s]

Training batches on cuda:   1%|▏         | 122/9708 [00:00<00:41, 232.41batch/s]

Training batches on cuda:   2%|▏         | 146/9708 [00:00<00:40, 233.81batch/s]

Training batches on cuda:   2%|▏         | 170/9708 [00:00<00:40, 233.96batch/s]

Training batches on cuda:   2%|▏         | 194/9708 [00:00<00:40, 233.33batch/s]

Training batches on cuda:   2%|▏         | 218/9708 [00:00<00:40, 234.14batch/s]

Training batches on cuda:   2%|▏         | 242/9708 [00:01<00:40, 234.10batch/s]

Training batches on cuda:   3%|▎         | 265/9708 [00:01<00:40, 231.06batch/s]

Training batches on cuda:   3%|▎         | 289/9708 [00:01<00:40, 230.84batch/s]

Training batches on cuda:   3%|▎         | 312/9708 [00:01<00:40, 230.04batch/s]

Training batches on cuda:   3%|▎         | 335/9708 [00:01<00:41, 227.82batch/s]

Training batches on cuda:   4%|▎         | 358/9708 [00:01<00:41, 226.82batch/s]

Training batches on cuda:   4%|▍         | 381/9708 [00:01<00:41, 226.76batch/s]

Training batches on cuda:   4%|▍         | 404/9708 [00:01<00:40, 227.58batch/s]

Training batches on cuda:   4%|▍         | 427/9708 [00:01<00:40, 226.81batch/s]

Training batches on cuda:   5%|▍         | 450/9708 [00:01<00:40, 227.61batch/s]

Training batches on cuda:   5%|▍         | 475/9708 [00:02<00:39, 231.67batch/s]

Training batches on cuda:   5%|▌         | 499/9708 [00:02<00:39, 233.82batch/s]

Training batches on cuda:   5%|▌         | 523/9708 [00:02<00:39, 233.20batch/s]

Training batches on cuda:   6%|▌         | 547/9708 [00:02<00:39, 233.14batch/s]

Training batches on cuda:   6%|▌         | 571/9708 [00:02<00:39, 232.59batch/s]

Training batches on cuda:   6%|▌         | 595/9708 [00:02<00:39, 232.06batch/s]

Training batches on cuda:   6%|▋         | 619/9708 [00:02<00:39, 231.75batch/s]

Training batches on cuda:   7%|▋         | 643/9708 [00:02<00:39, 231.43batch/s]

Training batches on cuda:   7%|▋         | 667/9708 [00:02<00:38, 232.22batch/s]

Training batches on cuda:   7%|▋         | 691/9708 [00:02<00:39, 230.55batch/s]

Training batches on cuda:   7%|▋         | 715/9708 [00:03<00:39, 229.27batch/s]

Training batches on cuda:   8%|▊         | 738/9708 [00:03<00:39, 228.96batch/s]

Training batches on cuda:   8%|▊         | 761/9708 [00:03<00:39, 228.97batch/s]

Training batches on cuda:   8%|▊         | 785/9708 [00:03<00:38, 229.84batch/s]

Training batches on cuda:   8%|▊         | 808/9708 [00:03<00:38, 229.31batch/s]

Training batches on cuda:   9%|▊         | 831/9708 [00:03<00:40, 218.94batch/s]

Training batches on cuda:   9%|▉         | 853/9708 [00:03<00:41, 211.84batch/s]

Training batches on cuda:   9%|▉         | 875/9708 [00:03<00:42, 209.26batch/s]

Training batches on cuda:   9%|▉         | 897/9708 [00:03<00:41, 210.37batch/s]

Training batches on cuda:   9%|▉         | 919/9708 [00:04<00:41, 209.47batch/s]

Training batches on cuda:  10%|▉         | 942/9708 [00:04<00:41, 212.91batch/s]

Training batches on cuda:  10%|▉         | 966/9708 [00:04<00:40, 218.13batch/s]

Training batches on cuda:  10%|█         | 989/9708 [00:04<00:39, 220.16batch/s]

Training batches on cuda:  10%|█         | 1012/9708 [00:04<00:40, 212.98batch/s]

Training batches on cuda:  11%|█         | 1034/9708 [00:04<00:40, 214.04batch/s]

Training batches on cuda:  11%|█         | 1057/9708 [00:04<00:40, 216.23batch/s]

Training batches on cuda:  11%|█         | 1081/9708 [00:04<00:38, 221.63batch/s]

Training batches on cuda:  11%|█▏        | 1104/9708 [00:04<00:38, 222.91batch/s]

Training batches on cuda:  12%|█▏        | 1127/9708 [00:04<00:38, 221.77batch/s]

Training batches on cuda:  12%|█▏        | 1150/9708 [00:05<00:38, 220.82batch/s]

Training batches on cuda:  12%|█▏        | 1173/9708 [00:05<00:39, 217.09batch/s]

Training batches on cuda:  12%|█▏        | 1195/9708 [00:05<00:39, 216.59batch/s]

Training batches on cuda:  13%|█▎        | 1217/9708 [00:05<00:39, 217.23batch/s]

Training batches on cuda:  13%|█▎        | 1240/9708 [00:05<00:38, 219.24batch/s]

Training batches on cuda:  13%|█▎        | 1262/9708 [00:05<00:42, 197.60batch/s]

Training batches on cuda:  13%|█▎        | 1286/9708 [00:05<00:40, 207.13batch/s]

Training batches on cuda:  13%|█▎        | 1309/9708 [00:05<00:39, 213.26batch/s]

Training batches on cuda:  14%|█▎        | 1331/9708 [00:05<00:40, 207.56batch/s]

Training batches on cuda:  14%|█▍        | 1353/9708 [00:06<00:39, 210.26batch/s]

Training batches on cuda:  14%|█▍        | 1375/9708 [00:06<00:42, 196.40batch/s]

Training batches on cuda:  14%|█▍        | 1396/9708 [00:06<00:41, 198.03batch/s]

Training batches on cuda:  15%|█▍        | 1417/9708 [00:06<00:41, 201.43batch/s]

Training batches on cuda:  15%|█▍        | 1440/9708 [00:06<00:39, 208.54batch/s]

Training batches on cuda:  15%|█▌        | 1462/9708 [00:06<00:39, 207.10batch/s]

Training batches on cuda:  15%|█▌        | 1484/9708 [00:06<00:39, 209.99batch/s]

Training batches on cuda:  16%|█▌        | 1507/9708 [00:06<00:38, 213.50batch/s]

Training batches on cuda:  16%|█▌        | 1529/9708 [00:06<00:38, 213.08batch/s]

Training batches on cuda:  16%|█▌        | 1551/9708 [00:07<00:39, 208.45batch/s]

Training batches on cuda:  16%|█▌        | 1573/9708 [00:07<00:38, 209.50batch/s]

Training batches on cuda:  16%|█▋        | 1596/9708 [00:07<00:38, 213.36batch/s]

Training batches on cuda:  17%|█▋        | 1619/9708 [00:07<00:37, 217.31batch/s]

Training batches on cuda:  17%|█▋        | 1642/9708 [00:07<00:36, 220.19batch/s]

Training batches on cuda:  17%|█▋        | 1665/9708 [00:07<00:36, 219.28batch/s]

Training batches on cuda:  17%|█▋        | 1688/9708 [00:07<00:36, 221.49batch/s]

Training batches on cuda:  18%|█▊        | 1711/9708 [00:07<00:36, 217.66batch/s]

Training batches on cuda:  18%|█▊        | 1733/9708 [00:07<00:36, 217.32batch/s]

Training batches on cuda:  18%|█▊        | 1755/9708 [00:07<00:37, 214.59batch/s]

Training batches on cuda:  18%|█▊        | 1777/9708 [00:08<00:37, 213.74batch/s]

Training batches on cuda:  19%|█▊        | 1800/9708 [00:08<00:36, 217.81batch/s]

Training batches on cuda:  19%|█▉        | 1824/9708 [00:08<00:35, 221.33batch/s]

Training batches on cuda:  19%|█▉        | 1848/9708 [00:08<00:35, 224.22batch/s]

Training batches on cuda:  19%|█▉        | 1871/9708 [00:08<00:36, 217.17batch/s]

Training batches on cuda:  19%|█▉        | 1893/9708 [00:08<00:36, 216.16batch/s]

Training batches on cuda:  20%|█▉        | 1915/9708 [00:08<00:36, 214.18batch/s]

Training batches on cuda:  20%|█▉        | 1937/9708 [00:08<00:36, 214.91batch/s]

Training batches on cuda:  20%|██        | 1959/9708 [00:08<00:38, 203.82batch/s]

Training batches on cuda:  20%|██        | 1980/9708 [00:09<00:38, 201.97batch/s]

Training batches on cuda:  21%|██        | 2001/9708 [00:09<00:39, 196.91batch/s]

Training batches on cuda:  21%|██        | 2022/9708 [00:09<00:38, 200.21batch/s]

Training batches on cuda:  21%|██        | 2043/9708 [00:09<00:38, 198.54batch/s]

Training batches on cuda:  21%|██▏       | 2064/9708 [00:09<00:38, 199.69batch/s]

Training batches on cuda:  21%|██▏       | 2086/9708 [00:09<00:37, 203.59batch/s]

Training batches on cuda:  22%|██▏       | 2107/9708 [00:09<00:38, 199.53batch/s]

Training batches on cuda:  22%|██▏       | 2129/9708 [00:09<00:36, 204.92batch/s]

Training batches on cuda:  22%|██▏       | 2151/9708 [00:09<00:36, 208.73batch/s]

Training batches on cuda:  22%|██▏       | 2173/9708 [00:09<00:35, 211.04batch/s]

Training batches on cuda:  23%|██▎       | 2195/9708 [00:10<00:35, 213.44batch/s]

Training batches on cuda:  23%|██▎       | 2217/9708 [00:10<00:34, 215.31batch/s]

Training batches on cuda:  23%|██▎       | 2239/9708 [00:10<00:34, 216.39batch/s]

Training batches on cuda:  23%|██▎       | 2261/9708 [00:10<00:34, 214.13batch/s]

Training batches on cuda:  24%|██▎       | 2283/9708 [00:10<00:35, 208.68batch/s]

Training batches on cuda:  24%|██▎       | 2304/9708 [00:10<00:35, 207.12batch/s]

Training batches on cuda:  24%|██▍       | 2325/9708 [00:10<00:36, 199.85batch/s]

Training batches on cuda:  24%|██▍       | 2346/9708 [00:10<00:36, 200.90batch/s]

Training batches on cuda:  24%|██▍       | 2367/9708 [00:10<00:36, 198.51batch/s]

Training batches on cuda:  25%|██▍       | 2389/9708 [00:10<00:35, 203.70batch/s]

Training batches on cuda:  25%|██▍       | 2411/9708 [00:11<00:35, 205.82batch/s]

Training batches on cuda:  25%|██▌       | 2433/9708 [00:11<00:35, 207.51batch/s]

Training batches on cuda:  25%|██▌       | 2456/9708 [00:11<00:34, 211.32batch/s]

Training batches on cuda:  26%|██▌       | 2479/9708 [00:11<00:33, 214.07batch/s]

Training batches on cuda:  26%|██▌       | 2501/9708 [00:11<00:33, 212.98batch/s]

Training batches on cuda:  26%|██▌       | 2523/9708 [00:11<00:34, 208.44batch/s]

Training batches on cuda:  26%|██▌       | 2544/9708 [00:11<00:34, 206.20batch/s]

Training batches on cuda:  26%|██▋       | 2566/9708 [00:11<00:34, 208.27batch/s]

Training batches on cuda:  27%|██▋       | 2588/9708 [00:11<00:34, 209.07batch/s]

Training batches on cuda:  27%|██▋       | 2610/9708 [00:12<00:33, 210.03batch/s]

Training batches on cuda:  27%|██▋       | 2632/9708 [00:12<00:33, 209.06batch/s]

Training batches on cuda:  27%|██▋       | 2653/9708 [00:12<00:34, 207.03batch/s]

Training batches on cuda:  28%|██▊       | 2674/9708 [00:12<00:34, 205.02batch/s]

Training batches on cuda:  28%|██▊       | 2696/9708 [00:12<00:33, 207.73batch/s]

Training batches on cuda:  28%|██▊       | 2719/9708 [00:12<00:32, 212.44batch/s]

Training batches on cuda:  28%|██▊       | 2741/9708 [00:12<00:33, 210.13batch/s]

Training batches on cuda:  28%|██▊       | 2764/9708 [00:12<00:32, 215.06batch/s]

Training batches on cuda:  29%|██▊       | 2786/9708 [00:12<00:32, 214.90batch/s]

Training batches on cuda:  29%|██▉       | 2808/9708 [00:12<00:32, 214.10batch/s]

Training batches on cuda:  29%|██▉       | 2830/9708 [00:13<00:32, 209.07batch/s]

Training batches on cuda:  29%|██▉       | 2851/9708 [00:13<00:32, 208.10batch/s]

Training batches on cuda:  30%|██▉       | 2873/9708 [00:13<00:32, 208.92batch/s]

Training batches on cuda:  30%|██▉       | 2894/9708 [00:13<00:33, 206.15batch/s]

Training batches on cuda:  30%|███       | 2915/9708 [00:13<00:32, 207.22batch/s]

Training batches on cuda:  30%|███       | 2936/9708 [00:13<00:33, 204.21batch/s]

Training batches on cuda:  30%|███       | 2957/9708 [00:13<00:33, 202.00batch/s]

Training batches on cuda:  31%|███       | 2978/9708 [00:13<00:33, 198.75batch/s]

Training batches on cuda:  31%|███       | 3000/9708 [00:13<00:33, 202.40batch/s]

Training batches on cuda:  31%|███       | 3021/9708 [00:14<00:32, 203.17batch/s]

Training batches on cuda:  31%|███▏      | 3042/9708 [00:14<00:32, 204.13batch/s]

Training batches on cuda:  32%|███▏      | 3064/9708 [00:14<00:31, 207.80batch/s]

Training batches on cuda:  32%|███▏      | 3085/9708 [00:14<00:32, 206.42batch/s]

Training batches on cuda:  32%|███▏      | 3107/9708 [00:14<00:31, 208.16batch/s]

Training batches on cuda:  32%|███▏      | 3128/9708 [00:14<00:33, 198.93batch/s]

Training batches on cuda:  32%|███▏      | 3148/9708 [00:14<00:33, 194.41batch/s]

Training batches on cuda:  33%|███▎      | 3168/9708 [00:14<00:34, 191.84batch/s]

Training batches on cuda:  33%|███▎      | 3188/9708 [00:14<00:34, 190.39batch/s]

Training batches on cuda:  33%|███▎      | 3208/9708 [00:14<00:34, 190.20batch/s]

Training batches on cuda:  33%|███▎      | 3228/9708 [00:15<00:34, 189.82batch/s]

Training batches on cuda:  33%|███▎      | 3250/9708 [00:15<00:32, 196.20batch/s]

Training batches on cuda:  34%|███▎      | 3270/9708 [00:15<00:33, 192.01batch/s]

Training batches on cuda:  34%|███▍      | 3293/9708 [00:15<00:32, 200.44batch/s]

Training batches on cuda:  34%|███▍      | 3314/9708 [00:15<00:31, 201.96batch/s]

Training batches on cuda:  34%|███▍      | 3336/9708 [00:15<00:31, 205.15batch/s]

Training batches on cuda:  35%|███▍      | 3358/9708 [00:15<00:30, 207.91batch/s]

Training batches on cuda:  35%|███▍      | 3379/9708 [00:15<00:30, 207.39batch/s]

Training batches on cuda:  35%|███▌      | 3403/9708 [00:15<00:29, 215.57batch/s]

Training batches on cuda:  35%|███▌      | 3426/9708 [00:16<00:29, 216.51batch/s]

Training batches on cuda:  36%|███▌      | 3449/9708 [00:16<00:28, 219.07batch/s]

Training batches on cuda:  36%|███▌      | 3476/9708 [00:16<00:27, 230.71batch/s]

Training batches on cuda:  36%|███▌      | 3503/9708 [00:16<00:25, 239.86batch/s]

Training batches on cuda:  36%|███▋      | 3528/9708 [00:16<00:26, 237.25batch/s]

Training batches on cuda:  37%|███▋      | 3552/9708 [00:16<00:26, 234.91batch/s]

Training batches on cuda:  37%|███▋      | 3576/9708 [00:16<00:26, 232.68batch/s]

Training batches on cuda:  37%|███▋      | 3600/9708 [00:16<00:26, 227.82batch/s]

Training batches on cuda:  37%|███▋      | 3623/9708 [00:16<00:28, 214.87batch/s]

Training batches on cuda:  38%|███▊      | 3645/9708 [00:17<00:30, 200.58batch/s]

Training batches on cuda:  38%|███▊      | 3666/9708 [00:17<00:30, 198.20batch/s]

Training batches on cuda:  38%|███▊      | 3687/9708 [00:17<00:30, 199.04batch/s]

Training batches on cuda:  38%|███▊      | 3708/9708 [00:17<00:30, 197.72batch/s]

Training batches on cuda:  38%|███▊      | 3728/9708 [00:17<00:32, 186.69batch/s]

Training batches on cuda:  39%|███▊      | 3748/9708 [00:17<00:31, 188.48batch/s]

Training batches on cuda:  39%|███▉      | 3772/9708 [00:17<00:29, 199.67batch/s]

Training batches on cuda:  39%|███▉      | 3795/9708 [00:17<00:28, 206.88batch/s]

Training batches on cuda:  39%|███▉      | 3818/9708 [00:17<00:27, 211.74batch/s]

Training batches on cuda:  40%|███▉      | 3841/9708 [00:17<00:27, 216.90batch/s]

Training batches on cuda:  40%|███▉      | 3863/9708 [00:18<00:26, 216.50batch/s]

Training batches on cuda:  40%|████      | 3886/9708 [00:18<00:26, 218.57batch/s]

Training batches on cuda:  40%|████      | 3908/9708 [00:18<00:27, 213.21batch/s]

Training batches on cuda:  40%|████      | 3930/9708 [00:18<00:27, 211.40batch/s]

Training batches on cuda:  41%|████      | 3953/9708 [00:18<00:26, 213.92batch/s]

Training batches on cuda:  41%|████      | 3975/9708 [00:18<00:26, 215.56batch/s]

Training batches on cuda:  41%|████      | 3997/9708 [00:18<00:26, 216.04batch/s]

Training batches on cuda:  41%|████▏     | 4020/9708 [00:18<00:26, 217.60batch/s]

Training batches on cuda:  42%|████▏     | 4044/9708 [00:18<00:25, 223.38batch/s]

Training batches on cuda:  42%|████▏     | 4068/9708 [00:18<00:24, 227.98batch/s]

Training batches on cuda:  42%|████▏     | 4092/9708 [00:19<00:24, 230.90batch/s]

Training batches on cuda:  42%|████▏     | 4116/9708 [00:19<00:24, 228.65batch/s]

Training batches on cuda:  43%|████▎     | 4139/9708 [00:19<00:26, 210.03batch/s]

Training batches on cuda:  43%|████▎     | 4161/9708 [00:19<00:26, 210.00batch/s]

Training batches on cuda:  43%|████▎     | 4183/9708 [00:19<00:26, 207.51batch/s]

Training batches on cuda:  43%|████▎     | 4204/9708 [00:19<00:26, 207.25batch/s]

Training batches on cuda:  44%|████▎     | 4226/9708 [00:19<00:26, 209.88batch/s]

Training batches on cuda:  44%|████▍     | 4248/9708 [00:19<00:26, 209.54batch/s]

Training batches on cuda:  44%|████▍     | 4270/9708 [00:19<00:25, 209.37batch/s]

Training batches on cuda:  44%|████▍     | 4291/9708 [00:20<00:25, 208.69batch/s]

Training batches on cuda:  44%|████▍     | 4312/9708 [00:20<00:26, 202.94batch/s]

Training batches on cuda:  45%|████▍     | 4334/9708 [00:20<00:26, 206.37batch/s]

Training batches on cuda:  45%|████▍     | 4355/9708 [00:20<00:26, 201.46batch/s]

Training batches on cuda:  45%|████▌     | 4376/9708 [00:20<00:26, 203.61batch/s]

Training batches on cuda:  45%|████▌     | 4398/9708 [00:20<00:25, 206.58batch/s]

Training batches on cuda:  46%|████▌     | 4419/9708 [00:20<00:27, 191.02batch/s]

Training batches on cuda:  46%|████▌     | 4440/9708 [00:20<00:26, 195.89batch/s]

Training batches on cuda:  46%|████▌     | 4462/9708 [00:20<00:26, 201.26batch/s]

Training batches on cuda:  46%|████▌     | 4483/9708 [00:21<00:26, 200.64batch/s]

Training batches on cuda:  46%|████▋     | 4504/9708 [00:21<00:26, 200.02batch/s]

Training batches on cuda:  47%|████▋     | 4525/9708 [00:21<00:26, 198.98batch/s]

Training batches on cuda:  47%|████▋     | 4546/9708 [00:21<00:25, 199.56batch/s]

Training batches on cuda:  47%|████▋     | 4567/9708 [00:21<00:25, 199.59batch/s]

Training batches on cuda:  47%|████▋     | 4587/9708 [00:21<00:25, 199.66batch/s]

Training batches on cuda:  47%|████▋     | 4607/9708 [00:21<00:25, 198.99batch/s]

Training batches on cuda:  48%|████▊     | 4629/9708 [00:21<00:25, 203.03batch/s]

Training batches on cuda:  48%|████▊     | 4651/9708 [00:21<00:24, 206.19batch/s]

Training batches on cuda:  48%|████▊     | 4674/9708 [00:21<00:23, 212.30batch/s]

Training batches on cuda:  48%|████▊     | 4696/9708 [00:22<00:24, 208.68batch/s]

Training batches on cuda:  49%|████▊     | 4717/9708 [00:22<00:25, 198.22batch/s]

Training batches on cuda:  49%|████▉     | 4737/9708 [00:22<00:26, 191.08batch/s]

Training batches on cuda:  49%|████▉     | 4757/9708 [00:22<00:26, 186.69batch/s]

Training batches on cuda:  49%|████▉     | 4778/9708 [00:22<00:25, 191.30batch/s]

Training batches on cuda:  49%|████▉     | 4798/9708 [00:22<00:25, 192.77batch/s]

Training batches on cuda:  50%|████▉     | 4818/9708 [00:22<00:25, 193.76batch/s]

Training batches on cuda:  50%|████▉     | 4839/9708 [00:22<00:24, 196.87batch/s]

Training batches on cuda:  50%|█████     | 4859/9708 [00:22<00:24, 197.51batch/s]

Training batches on cuda:  50%|█████     | 4880/9708 [00:23<00:24, 198.38batch/s]

Training batches on cuda:  50%|█████     | 4900/9708 [00:23<00:24, 195.55batch/s]

Training batches on cuda:  51%|█████     | 4922/9708 [00:23<00:23, 200.22batch/s]

Training batches on cuda:  51%|█████     | 4943/9708 [00:23<00:23, 202.43batch/s]

Training batches on cuda:  51%|█████     | 4964/9708 [00:23<00:23, 203.54batch/s]

Training batches on cuda:  51%|█████▏    | 4985/9708 [00:23<00:23, 198.07batch/s]

Training batches on cuda:  52%|█████▏    | 5005/9708 [00:23<00:23, 196.02batch/s]

Training batches on cuda:  52%|█████▏    | 5025/9708 [00:23<00:23, 195.63batch/s]

Training batches on cuda:  52%|█████▏    | 5045/9708 [00:23<00:23, 196.49batch/s]

Training batches on cuda:  52%|█████▏    | 5065/9708 [00:23<00:23, 195.91batch/s]

Training batches on cuda:  52%|█████▏    | 5085/9708 [00:24<00:23, 195.73batch/s]

Training batches on cuda:  53%|█████▎    | 5105/9708 [00:24<00:23, 196.52batch/s]

Training batches on cuda:  53%|█████▎    | 5125/9708 [00:24<00:23, 196.00batch/s]

Training batches on cuda:  53%|█████▎    | 5145/9708 [00:24<00:23, 196.41batch/s]

Training batches on cuda:  53%|█████▎    | 5165/9708 [00:24<00:23, 197.00batch/s]

Training batches on cuda:  53%|█████▎    | 5185/9708 [00:24<00:22, 197.01batch/s]

Training batches on cuda:  54%|█████▎    | 5205/9708 [00:24<00:22, 197.01batch/s]

Training batches on cuda:  54%|█████▍    | 5225/9708 [00:24<00:22, 195.96batch/s]

Training batches on cuda:  54%|█████▍    | 5245/9708 [00:24<00:23, 191.80batch/s]

Training batches on cuda:  54%|█████▍    | 5265/9708 [00:24<00:23, 191.42batch/s]

Training batches on cuda:  54%|█████▍    | 5285/9708 [00:25<00:22, 192.45batch/s]

Training batches on cuda:  55%|█████▍    | 5305/9708 [00:25<00:22, 192.29batch/s]

Training batches on cuda:  55%|█████▍    | 5325/9708 [00:25<00:22, 193.42batch/s]

Training batches on cuda:  55%|█████▌    | 5345/9708 [00:25<00:22, 193.57batch/s]

Training batches on cuda:  55%|█████▌    | 5365/9708 [00:25<00:22, 193.70batch/s]

Training batches on cuda:  55%|█████▌    | 5385/9708 [00:25<00:22, 189.91batch/s]

Training batches on cuda:  56%|█████▌    | 5405/9708 [00:25<00:25, 169.74batch/s]

Training batches on cuda:  56%|█████▌    | 5423/9708 [00:25<00:24, 172.01batch/s]

Training batches on cuda:  56%|█████▌    | 5443/9708 [00:25<00:23, 179.29batch/s]

Training batches on cuda:  56%|█████▋    | 5465/9708 [00:26<00:22, 189.37batch/s]

Training batches on cuda:  57%|█████▋    | 5486/9708 [00:26<00:21, 194.13batch/s]

Training batches on cuda:  57%|█████▋    | 5506/9708 [00:26<00:21, 195.09batch/s]

Training batches on cuda:  57%|█████▋    | 5526/9708 [00:26<00:21, 194.66batch/s]

Training batches on cuda:  57%|█████▋    | 5546/9708 [00:26<00:21, 192.45batch/s]

Training batches on cuda:  57%|█████▋    | 5567/9708 [00:26<00:21, 195.65batch/s]

Training batches on cuda:  58%|█████▊    | 5590/9708 [00:26<00:20, 202.08batch/s]

Training batches on cuda:  58%|█████▊    | 5612/9708 [00:26<00:19, 206.28batch/s]

Training batches on cuda:  58%|█████▊    | 5635/9708 [00:26<00:19, 211.06batch/s]

Training batches on cuda:  58%|█████▊    | 5658/9708 [00:26<00:18, 215.61batch/s]

Training batches on cuda:  59%|█████▊    | 5680/9708 [00:27<00:18, 216.62batch/s]

Training batches on cuda:  59%|█████▊    | 5702/9708 [00:27<00:18, 216.43batch/s]

Training batches on cuda:  59%|█████▉    | 5726/9708 [00:27<00:17, 221.36batch/s]

Training batches on cuda:  59%|█████▉    | 5749/9708 [00:27<00:17, 220.47batch/s]

Training batches on cuda:  59%|█████▉    | 5772/9708 [00:27<00:18, 216.83batch/s]

Training batches on cuda:  60%|█████▉    | 5794/9708 [00:27<00:18, 209.10batch/s]

Training batches on cuda:  60%|█████▉    | 5821/9708 [00:27<00:17, 222.33batch/s]

Training batches on cuda:  60%|██████    | 5847/9708 [00:27<00:16, 231.94batch/s]

Training batches on cuda:  60%|██████    | 5872/9708 [00:27<00:16, 235.25batch/s]

Training batches on cuda:  61%|██████    | 5897/9708 [00:28<00:16, 236.97batch/s]

Training batches on cuda:  61%|██████    | 5922/9708 [00:28<00:15, 238.33batch/s]

Training batches on cuda:  61%|██████▏   | 5948/9708 [00:28<00:15, 242.02batch/s]

Training batches on cuda:  62%|██████▏   | 5975/9708 [00:28<00:14, 249.28batch/s]

Training batches on cuda:  62%|██████▏   | 6003/9708 [00:28<00:14, 255.60batch/s]

Training batches on cuda:  62%|██████▏   | 6030/9708 [00:28<00:14, 259.64batch/s]

Training batches on cuda:  62%|██████▏   | 6057/9708 [00:28<00:14, 260.66batch/s]

Training batches on cuda:  63%|██████▎   | 6084/9708 [00:28<00:14, 251.14batch/s]

Training batches on cuda:  63%|██████▎   | 6110/9708 [00:28<00:15, 238.02batch/s]

Training batches on cuda:  63%|██████▎   | 6135/9708 [00:29<00:15, 229.71batch/s]

Training batches on cuda:  63%|██████▎   | 6159/9708 [00:29<00:16, 210.93batch/s]

Training batches on cuda:  64%|██████▎   | 6181/9708 [00:29<00:17, 197.08batch/s]

Training batches on cuda:  64%|██████▍   | 6202/9708 [00:29<00:17, 197.37batch/s]

Training batches on cuda:  64%|██████▍   | 6223/9708 [00:29<00:18, 192.74batch/s]

Training batches on cuda:  64%|██████▍   | 6243/9708 [00:29<00:18, 191.36batch/s]

Training batches on cuda:  65%|██████▍   | 6264/9708 [00:29<00:17, 195.10batch/s]

Training batches on cuda:  65%|██████▍   | 6284/9708 [00:29<00:17, 195.55batch/s]

Training batches on cuda:  65%|██████▍   | 6306/9708 [00:29<00:16, 200.40batch/s]

Training batches on cuda:  65%|██████▌   | 6327/9708 [00:30<00:16, 200.98batch/s]

Training batches on cuda:  65%|██████▌   | 6349/9708 [00:30<00:16, 205.82batch/s]

Training batches on cuda:  66%|██████▌   | 6370/9708 [00:30<00:16, 207.01batch/s]

Training batches on cuda:  66%|██████▌   | 6391/9708 [00:30<00:16, 204.90batch/s]

Training batches on cuda:  66%|██████▌   | 6412/9708 [00:30<00:16, 200.82batch/s]

Training batches on cuda:  66%|██████▋   | 6436/9708 [00:30<00:15, 208.97batch/s]

Training batches on cuda:  67%|██████▋   | 6459/9708 [00:30<00:15, 213.65batch/s]

Training batches on cuda:  67%|██████▋   | 6481/9708 [00:30<00:16, 194.49batch/s]

Training batches on cuda:  67%|██████▋   | 6503/9708 [00:30<00:16, 200.06batch/s]

Training batches on cuda:  67%|██████▋   | 6525/9708 [00:30<00:15, 204.57batch/s]

Training batches on cuda:  67%|██████▋   | 6547/9708 [00:31<00:15, 208.02batch/s]

Training batches on cuda:  68%|██████▊   | 6570/9708 [00:31<00:14, 212.84batch/s]

Training batches on cuda:  68%|██████▊   | 6592/9708 [00:31<00:14, 210.85batch/s]

Training batches on cuda:  68%|██████▊   | 6615/9708 [00:31<00:14, 215.30batch/s]

Training batches on cuda:  68%|██████▊   | 6638/9708 [00:31<00:14, 218.54batch/s]

Training batches on cuda:  69%|██████▊   | 6661/9708 [00:31<00:13, 221.42batch/s]

Training batches on cuda:  69%|██████▉   | 6685/9708 [00:31<00:13, 224.13batch/s]

Training batches on cuda:  69%|██████▉   | 6708/9708 [00:31<00:13, 225.54batch/s]

Training batches on cuda:  69%|██████▉   | 6732/9708 [00:31<00:13, 226.74batch/s]

Training batches on cuda:  70%|██████▉   | 6755/9708 [00:32<00:14, 209.26batch/s]

Training batches on cuda:  70%|██████▉   | 6777/9708 [00:32<00:14, 206.56batch/s]

Training batches on cuda:  70%|███████   | 6798/9708 [00:32<00:14, 204.48batch/s]

Training batches on cuda:  70%|███████   | 6819/9708 [00:32<00:14, 195.17batch/s]

Training batches on cuda:  70%|███████   | 6839/9708 [00:32<00:14, 196.05batch/s]

Training batches on cuda:  71%|███████   | 6860/9708 [00:32<00:14, 199.82batch/s]

Training batches on cuda:  71%|███████   | 6881/9708 [00:32<00:14, 200.96batch/s]

Training batches on cuda:  71%|███████   | 6902/9708 [00:32<00:13, 202.85batch/s]

Training batches on cuda:  71%|███████▏  | 6923/9708 [00:32<00:13, 204.31batch/s]

Training batches on cuda:  72%|███████▏  | 6945/9708 [00:32<00:13, 205.78batch/s]

Training batches on cuda:  72%|███████▏  | 6966/9708 [00:33<00:13, 206.79batch/s]

Training batches on cuda:  72%|███████▏  | 6989/9708 [00:33<00:12, 211.72batch/s]

Training batches on cuda:  72%|███████▏  | 7012/9708 [00:33<00:12, 214.56batch/s]

Training batches on cuda:  72%|███████▏  | 7035/9708 [00:33<00:12, 217.90batch/s]

Training batches on cuda:  73%|███████▎  | 7058/9708 [00:33<00:11, 221.20batch/s]

Training batches on cuda:  73%|███████▎  | 7081/9708 [00:33<00:11, 222.51batch/s]

Training batches on cuda:  73%|███████▎  | 7104/9708 [00:33<00:12, 216.44batch/s]

Training batches on cuda:  73%|███████▎  | 7127/9708 [00:33<00:11, 220.21batch/s]

Training batches on cuda:  74%|███████▎  | 7151/9708 [00:33<00:11, 223.24batch/s]

Training batches on cuda:  74%|███████▍  | 7174/9708 [00:34<00:11, 220.14batch/s]

Training batches on cuda:  74%|███████▍  | 7197/9708 [00:34<00:11, 222.85batch/s]

Training batches on cuda:  74%|███████▍  | 7221/9708 [00:34<00:11, 225.58batch/s]

Training batches on cuda:  75%|███████▍  | 7246/9708 [00:34<00:10, 230.15batch/s]

Training batches on cuda:  75%|███████▍  | 7270/9708 [00:34<00:10, 230.45batch/s]

Training batches on cuda:  75%|███████▌  | 7294/9708 [00:34<00:10, 227.80batch/s]

Training batches on cuda:  75%|███████▌  | 7317/9708 [00:34<00:10, 223.33batch/s]

Training batches on cuda:  76%|███████▌  | 7340/9708 [00:34<00:11, 209.13batch/s]

Training batches on cuda:  76%|███████▌  | 7362/9708 [00:34<00:11, 208.77batch/s]

Training batches on cuda:  76%|███████▌  | 7384/9708 [00:34<00:11, 206.72batch/s]

Training batches on cuda:  76%|███████▋  | 7405/9708 [00:35<00:11, 205.89batch/s]

Training batches on cuda:  76%|███████▋  | 7426/9708 [00:35<00:11, 205.77batch/s]

Training batches on cuda:  77%|███████▋  | 7447/9708 [00:35<00:10, 206.43batch/s]

Training batches on cuda:  77%|███████▋  | 7470/9708 [00:35<00:10, 211.63batch/s]

Training batches on cuda:  77%|███████▋  | 7493/9708 [00:35<00:10, 216.19batch/s]

Training batches on cuda:  77%|███████▋  | 7516/9708 [00:35<00:09, 219.65batch/s]

Training batches on cuda:  78%|███████▊  | 7539/9708 [00:35<00:09, 222.47batch/s]

Training batches on cuda:  78%|███████▊  | 7562/9708 [00:35<00:10, 204.89batch/s]

Training batches on cuda:  78%|███████▊  | 7589/9708 [00:35<00:09, 220.43batch/s]

Training batches on cuda:  78%|███████▊  | 7616/9708 [00:36<00:08, 232.86batch/s]

Training batches on cuda:  79%|███████▊  | 7644/9708 [00:36<00:08, 243.28batch/s]

Training batches on cuda:  79%|███████▉  | 7671/9708 [00:36<00:08, 250.23batch/s]

Training batches on cuda:  79%|███████▉  | 7699/9708 [00:36<00:07, 256.52batch/s]

Training batches on cuda:  80%|███████▉  | 7727/9708 [00:36<00:07, 262.30batch/s]

Training batches on cuda:  80%|███████▉  | 7754/9708 [00:36<00:07, 264.46batch/s]

Training batches on cuda:  80%|████████  | 7781/9708 [00:36<00:07, 266.09batch/s]

Training batches on cuda:  80%|████████  | 7808/9708 [00:36<00:07, 266.70batch/s]

Training batches on cuda:  81%|████████  | 7836/9708 [00:36<00:06, 268.48batch/s]

Training batches on cuda:  81%|████████  | 7864/9708 [00:36<00:06, 270.24batch/s]

Training batches on cuda:  81%|████████▏ | 7892/9708 [00:37<00:06, 269.86batch/s]

Training batches on cuda:  82%|████████▏ | 7920/9708 [00:37<00:06, 269.15batch/s]

Training batches on cuda:  82%|████████▏ | 7947/9708 [00:37<00:06, 268.45batch/s]

Training batches on cuda:  82%|████████▏ | 7974/9708 [00:37<00:06, 268.32batch/s]

Training batches on cuda:  82%|████████▏ | 8002/9708 [00:37<00:06, 269.22batch/s]

Training batches on cuda:  83%|████████▎ | 8029/9708 [00:37<00:06, 269.17batch/s]

Training batches on cuda:  83%|████████▎ | 8056/9708 [00:37<00:06, 269.42batch/s]

Training batches on cuda:  83%|████████▎ | 8083/9708 [00:37<00:06, 268.01batch/s]

Training batches on cuda:  84%|████████▎ | 8110/9708 [00:37<00:05, 267.88batch/s]

Training batches on cuda:  84%|████████▍ | 8138/9708 [00:37<00:05, 268.80batch/s]

Training batches on cuda:  84%|████████▍ | 8165/9708 [00:38<00:05, 262.93batch/s]

Training batches on cuda:  84%|████████▍ | 8192/9708 [00:38<00:06, 250.66batch/s]

Training batches on cuda:  85%|████████▍ | 8218/9708 [00:38<00:06, 242.13batch/s]

Training batches on cuda:  85%|████████▍ | 8243/9708 [00:38<00:06, 229.31batch/s]

Training batches on cuda:  85%|████████▌ | 8267/9708 [00:38<00:06, 220.71batch/s]

Training batches on cuda:  85%|████████▌ | 8290/9708 [00:38<00:06, 215.78batch/s]

Training batches on cuda:  86%|████████▌ | 8312/9708 [00:38<00:06, 209.41batch/s]

Training batches on cuda:  86%|████████▌ | 8334/9708 [00:38<00:06, 204.75batch/s]

Training batches on cuda:  86%|████████▌ | 8355/9708 [00:38<00:06, 205.70batch/s]

Training batches on cuda:  86%|████████▋ | 8376/9708 [00:39<00:06, 206.76batch/s]

Training batches on cuda:  87%|████████▋ | 8398/9708 [00:39<00:06, 208.01batch/s]

Training batches on cuda:  87%|████████▋ | 8419/9708 [00:39<00:06, 207.89batch/s]

Training batches on cuda:  87%|████████▋ | 8440/9708 [00:39<00:06, 208.19batch/s]

Training batches on cuda:  87%|████████▋ | 8461/9708 [00:39<00:05, 208.39batch/s]

Training batches on cuda:  87%|████████▋ | 8482/9708 [00:39<00:05, 208.65batch/s]

Training batches on cuda:  88%|████████▊ | 8503/9708 [00:39<00:05, 208.00batch/s]

Training batches on cuda:  88%|████████▊ | 8524/9708 [00:39<00:05, 207.42batch/s]

Training batches on cuda:  88%|████████▊ | 8545/9708 [00:39<00:05, 207.55batch/s]

Training batches on cuda:  88%|████████▊ | 8566/9708 [00:40<00:05, 197.29batch/s]

Training batches on cuda:  88%|████████▊ | 8587/9708 [00:40<00:05, 199.97batch/s]

Training batches on cuda:  89%|████████▊ | 8610/9708 [00:40<00:05, 206.73batch/s]

Training batches on cuda:  89%|████████▉ | 8634/9708 [00:40<00:05, 213.36batch/s]

Training batches on cuda:  89%|████████▉ | 8657/9708 [00:40<00:04, 217.41batch/s]

Training batches on cuda:  89%|████████▉ | 8679/9708 [00:40<00:04, 217.65batch/s]

Training batches on cuda:  90%|████████▉ | 8702/9708 [00:40<00:04, 220.74batch/s]

Training batches on cuda:  90%|████████▉ | 8726/9708 [00:40<00:04, 223.64batch/s]

Training batches on cuda:  90%|█████████ | 8749/9708 [00:40<00:04, 193.24batch/s]

Training batches on cuda:  90%|█████████ | 8773/9708 [00:40<00:04, 203.73batch/s]

Training batches on cuda:  91%|█████████ | 8797/9708 [00:41<00:04, 212.35batch/s]

Training batches on cuda:  91%|█████████ | 8821/9708 [00:41<00:04, 218.94batch/s]

Training batches on cuda:  91%|█████████ | 8845/9708 [00:41<00:03, 223.14batch/s]

Training batches on cuda:  91%|█████████▏| 8869/9708 [00:41<00:03, 227.57batch/s]

Training batches on cuda:  92%|█████████▏| 8893/9708 [00:41<00:03, 229.27batch/s]

Training batches on cuda:  92%|█████████▏| 8917/9708 [00:41<00:03, 227.13batch/s]

Training batches on cuda:  92%|█████████▏| 8940/9708 [00:41<00:03, 223.46batch/s]

Training batches on cuda:  92%|█████████▏| 8963/9708 [00:41<00:03, 216.39batch/s]

Training batches on cuda:  93%|█████████▎| 8985/9708 [00:41<00:03, 215.27batch/s]

Training batches on cuda:  93%|█████████▎| 9008/9708 [00:42<00:03, 218.06batch/s]

Training batches on cuda:  93%|█████████▎| 9030/9708 [00:42<00:03, 214.90batch/s]

Training batches on cuda:  93%|█████████▎| 9053/9708 [00:42<00:03, 217.64batch/s]

Training batches on cuda:  93%|█████████▎| 9075/9708 [00:42<00:02, 217.39batch/s]

Training batches on cuda:  94%|█████████▎| 9097/9708 [00:42<00:02, 214.22batch/s]

Training batches on cuda:  94%|█████████▍| 9119/9708 [00:42<00:02, 213.82batch/s]

Training batches on cuda:  94%|█████████▍| 9141/9708 [00:42<00:02, 214.39batch/s]

Training batches on cuda:  94%|█████████▍| 9164/9708 [00:42<00:02, 217.67batch/s]

Training batches on cuda:  95%|█████████▍| 9188/9708 [00:42<00:02, 221.45batch/s]

Training batches on cuda:  95%|█████████▍| 9212/9708 [00:42<00:02, 226.51batch/s]

Training batches on cuda:  95%|█████████▌| 9236/9708 [00:43<00:02, 229.37batch/s]

Training batches on cuda:  95%|█████████▌| 9260/9708 [00:43<00:01, 231.22batch/s]

Training batches on cuda:  96%|█████████▌| 9286/9708 [00:43<00:01, 237.37batch/s]

Training batches on cuda:  96%|█████████▌| 9310/9708 [00:43<00:01, 238.13batch/s]

Training batches on cuda:  96%|█████████▌| 9334/9708 [00:43<00:01, 235.45batch/s]

Training batches on cuda:  96%|█████████▋| 9360/9708 [00:43<00:01, 240.21batch/s]

Training batches on cuda:  97%|█████████▋| 9385/9708 [00:43<00:01, 241.97batch/s]

Training batches on cuda:  97%|█████████▋| 9410/9708 [00:43<00:01, 239.08batch/s]

Training batches on cuda:  97%|█████████▋| 9435/9708 [00:43<00:01, 240.59batch/s]

Training batches on cuda:  97%|█████████▋| 9460/9708 [00:43<00:01, 243.10batch/s]

Training batches on cuda:  98%|█████████▊| 9485/9708 [00:44<00:00, 241.65batch/s]

Training batches on cuda:  98%|█████████▊| 9510/9708 [00:44<00:00, 239.63batch/s]

Training batches on cuda:  98%|█████████▊| 9537/9708 [00:44<00:00, 246.56batch/s]

Training batches on cuda:  98%|█████████▊| 9562/9708 [00:44<00:00, 246.01batch/s]

Training batches on cuda:  99%|█████████▉| 9587/9708 [00:44<00:00, 241.15batch/s]

Training batches on cuda:  99%|█████████▉| 9613/9708 [00:44<00:00, 243.96batch/s]

Training batches on cuda:  99%|█████████▉| 9639/9708 [00:44<00:00, 246.40batch/s]

Training batches on cuda: 100%|█████████▉| 9664/9708 [00:44<00:00, 243.05batch/s]

Training batches on cuda: 100%|█████████▉| 9689/9708 [00:44<00:00, 243.52batch/s]

                                                                                 
Training epochs on cuda:  70%|███████   | 7/10 [06:08<02:18, 46.03s/epoch, loss=0.096, prev_loss=0.096]
Training epochs on cuda:  80%|████████  | 8/10 [06:08<01:31, 45.77s/epoch, loss=0.096, prev_loss=0.096]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 19/9708 [00:00<00:51, 188.54batch/s]

Training batches on cuda:   0%|          | 44/9708 [00:00<00:47, 203.46batch/s]

Training batches on cuda:   1%|          | 68/9708 [00:00<00:45, 211.70batch/s]

Training batches on cuda:   1%|          | 92/9708 [00:00<00:44, 218.00batch/s]

Training batches on cuda:   1%|          | 116/9708 [00:00<00:43, 222.10batch/s]

Training batches on cuda:   1%|▏         | 138/9708 [00:00<00:43, 219.97batch/s]

Training batches on cuda:   2%|▏         | 158/9708 [00:00<00:49, 193.79batch/s]

Training batches on cuda:   2%|▏         | 180/9708 [00:00<00:47, 200.05batch/s]

Training batches on cuda:   2%|▏         | 202/9708 [00:00<00:46, 203.95batch/s]

Training batches on cuda:   2%|▏         | 224/9708 [00:01<00:45, 206.95batch/s]

Training batches on cuda:   3%|▎         | 246/9708 [00:01<00:45, 208.85batch/s]

Training batches on cuda:   3%|▎         | 268/9708 [00:01<00:45, 209.67batch/s]

Training batches on cuda:   3%|▎         | 290/9708 [00:01<00:44, 210.75batch/s]

Training batches on cuda:   3%|▎         | 312/9708 [00:01<00:44, 211.95batch/s]

Training batches on cuda:   3%|▎         | 334/9708 [00:01<00:45, 208.12batch/s]

Training batches on cuda:   4%|▎         | 355/9708 [00:01<00:46, 201.94batch/s]

Training batches on cuda:   4%|▍         | 376/9708 [00:01<00:47, 195.38batch/s]

Training batches on cuda:   4%|▍         | 397/9708 [00:01<00:47, 196.95batch/s]

Training batches on cuda:   4%|▍         | 418/9708 [00:02<00:46, 200.64batch/s]

Training batches on cuda:   5%|▍         | 439/9708 [00:02<00:45, 202.19batch/s]

Training batches on cuda:   5%|▍         | 460/9708 [00:02<00:45, 202.90batch/s]

Training batches on cuda:   5%|▍         | 481/9708 [00:02<00:46, 198.49batch/s]

Training batches on cuda:   5%|▌         | 501/9708 [00:02<00:47, 194.37batch/s]

Training batches on cuda:   5%|▌         | 521/9708 [00:02<00:47, 191.95batch/s]

Training batches on cuda:   6%|▌         | 541/9708 [00:02<00:48, 189.54batch/s]

Training batches on cuda:   6%|▌         | 560/9708 [00:02<00:48, 188.27batch/s]

Training batches on cuda:   6%|▌         | 579/9708 [00:02<00:48, 186.98batch/s]

Training batches on cuda:   6%|▌         | 599/9708 [00:02<00:48, 188.45batch/s]

Training batches on cuda:   6%|▋         | 620/9708 [00:03<00:47, 193.21batch/s]

Training batches on cuda:   7%|▋         | 641/9708 [00:03<00:45, 197.49batch/s]

Training batches on cuda:   7%|▋         | 662/9708 [00:03<00:45, 200.72batch/s]

Training batches on cuda:   7%|▋         | 683/9708 [00:03<00:44, 202.22batch/s]

Training batches on cuda:   7%|▋         | 707/9708 [00:03<00:42, 211.33batch/s]

Training batches on cuda:   8%|▊         | 731/9708 [00:03<00:41, 217.23batch/s]

Training batches on cuda:   8%|▊         | 755/9708 [00:03<00:40, 222.96batch/s]

Training batches on cuda:   8%|▊         | 779/9708 [00:03<00:39, 226.59batch/s]

Training batches on cuda:   8%|▊         | 803/9708 [00:03<00:38, 228.63batch/s]

Training batches on cuda:   9%|▊         | 827/9708 [00:03<00:38, 229.63batch/s]

Training batches on cuda:   9%|▉         | 851/9708 [00:04<00:38, 231.17batch/s]

Training batches on cuda:   9%|▉         | 875/9708 [00:04<00:37, 233.01batch/s]

Training batches on cuda:   9%|▉         | 899/9708 [00:04<00:37, 232.57batch/s]

Training batches on cuda:  10%|▉         | 923/9708 [00:04<00:37, 233.10batch/s]

Training batches on cuda:  10%|▉         | 947/9708 [00:04<00:37, 233.27batch/s]

Training batches on cuda:  10%|█         | 971/9708 [00:04<00:37, 232.75batch/s]

Training batches on cuda:  10%|█         | 995/9708 [00:04<00:37, 232.50batch/s]

Training batches on cuda:  10%|█         | 1019/9708 [00:04<00:38, 227.78batch/s]

Training batches on cuda:  11%|█         | 1042/9708 [00:04<00:38, 227.23batch/s]

Training batches on cuda:  11%|█         | 1066/9708 [00:05<00:37, 228.40batch/s]

Training batches on cuda:  11%|█         | 1090/9708 [00:05<00:37, 230.68batch/s]

Training batches on cuda:  11%|█▏        | 1114/9708 [00:05<00:37, 231.59batch/s]

Training batches on cuda:  12%|█▏        | 1138/9708 [00:05<00:36, 231.93batch/s]

Training batches on cuda:  12%|█▏        | 1162/9708 [00:05<00:36, 231.62batch/s]

Training batches on cuda:  12%|█▏        | 1187/9708 [00:05<00:36, 235.25batch/s]

Training batches on cuda:  12%|█▏        | 1211/9708 [00:05<00:36, 235.12batch/s]

Training batches on cuda:  13%|█▎        | 1235/9708 [00:05<00:43, 194.81batch/s]

Training batches on cuda:  13%|█▎        | 1259/9708 [00:05<00:41, 206.06batch/s]

Training batches on cuda:  13%|█▎        | 1283/9708 [00:06<00:39, 213.60batch/s]

Training batches on cuda:  13%|█▎        | 1306/9708 [00:06<00:39, 214.27batch/s]

Training batches on cuda:  14%|█▎        | 1329/9708 [00:06<00:38, 216.87batch/s]

Training batches on cuda:  14%|█▍        | 1352/9708 [00:06<00:38, 218.67batch/s]

Training batches on cuda:  14%|█▍        | 1375/9708 [00:06<00:38, 217.32batch/s]

Training batches on cuda:  14%|█▍        | 1398/9708 [00:06<00:38, 218.20batch/s]

Training batches on cuda:  15%|█▍        | 1420/9708 [00:06<00:38, 217.57batch/s]

Training batches on cuda:  15%|█▍        | 1442/9708 [00:06<00:37, 217.93batch/s]

Training batches on cuda:  15%|█▌        | 1464/9708 [00:06<00:38, 216.89batch/s]

Training batches on cuda:  15%|█▌        | 1486/9708 [00:06<00:38, 216.29batch/s]

Training batches on cuda:  16%|█▌        | 1508/9708 [00:07<00:38, 214.06batch/s]

Training batches on cuda:  16%|█▌        | 1530/9708 [00:07<00:38, 212.72batch/s]

Training batches on cuda:  16%|█▌        | 1554/9708 [00:07<00:37, 217.54batch/s]

Training batches on cuda:  16%|█▌        | 1577/9708 [00:07<00:36, 220.07batch/s]

Training batches on cuda:  16%|█▋        | 1600/9708 [00:07<00:37, 217.67batch/s]

Training batches on cuda:  17%|█▋        | 1622/9708 [00:07<00:37, 213.10batch/s]

Training batches on cuda:  17%|█▋        | 1649/9708 [00:07<00:35, 226.86batch/s]

Training batches on cuda:  17%|█▋        | 1674/9708 [00:07<00:34, 233.04batch/s]

Training batches on cuda:  17%|█▋        | 1698/9708 [00:07<00:35, 224.97batch/s]

Training batches on cuda:  18%|█▊        | 1721/9708 [00:08<00:39, 203.33batch/s]

Training batches on cuda:  18%|█▊        | 1742/9708 [00:08<00:39, 203.79batch/s]

Training batches on cuda:  18%|█▊        | 1763/9708 [00:08<00:39, 201.17batch/s]

Training batches on cuda:  18%|█▊        | 1784/9708 [00:08<00:40, 197.67batch/s]

Training batches on cuda:  19%|█▊        | 1807/9708 [00:08<00:38, 206.31batch/s]

Training batches on cuda:  19%|█▉        | 1828/9708 [00:08<00:40, 192.70batch/s]

Training batches on cuda:  19%|█▉        | 1848/9708 [00:08<00:42, 183.42batch/s]

Training batches on cuda:  19%|█▉        | 1867/9708 [00:08<00:44, 176.95batch/s]

Training batches on cuda:  19%|█▉        | 1885/9708 [00:08<00:45, 173.14batch/s]

Training batches on cuda:  20%|█▉        | 1904/9708 [00:09<00:44, 176.47batch/s]

Training batches on cuda:  20%|█▉        | 1925/9708 [00:09<00:42, 184.01batch/s]

Training batches on cuda:  20%|██        | 1946/9708 [00:09<00:40, 191.01batch/s]

Training batches on cuda:  20%|██        | 1966/9708 [00:09<00:40, 192.46batch/s]

Training batches on cuda:  20%|██        | 1988/9708 [00:09<00:38, 198.35batch/s]

Training batches on cuda:  21%|██        | 2009/9708 [00:09<00:38, 200.57batch/s]

Training batches on cuda:  21%|██        | 2030/9708 [00:09<00:38, 200.98batch/s]

Training batches on cuda:  21%|██        | 2053/9708 [00:09<00:37, 206.27batch/s]

Training batches on cuda:  21%|██▏       | 2074/9708 [00:09<00:38, 200.85batch/s]

Training batches on cuda:  22%|██▏       | 2095/9708 [00:09<00:37, 200.74batch/s]

Training batches on cuda:  22%|██▏       | 2116/9708 [00:10<00:37, 201.58batch/s]

Training batches on cuda:  22%|██▏       | 2137/9708 [00:10<00:39, 191.17batch/s]

Training batches on cuda:  22%|██▏       | 2157/9708 [00:10<00:39, 191.57batch/s]

Training batches on cuda:  22%|██▏       | 2178/9708 [00:10<00:38, 195.87batch/s]

Training batches on cuda:  23%|██▎       | 2198/9708 [00:10<00:39, 188.86batch/s]

Training batches on cuda:  23%|██▎       | 2218/9708 [00:10<00:39, 189.90batch/s]

Training batches on cuda:  23%|██▎       | 2239/9708 [00:10<00:38, 192.88batch/s]

Training batches on cuda:  23%|██▎       | 2260/9708 [00:10<00:37, 196.70batch/s]

Training batches on cuda:  24%|██▎       | 2283/9708 [00:10<00:36, 204.91batch/s]

Training batches on cuda:  24%|██▎       | 2304/9708 [00:11<00:36, 200.19batch/s]

Training batches on cuda:  24%|██▍       | 2328/9708 [00:11<00:35, 210.64batch/s]

Training batches on cuda:  24%|██▍       | 2353/9708 [00:11<00:33, 218.80batch/s]

Training batches on cuda:  24%|██▍       | 2377/9708 [00:11<00:32, 223.52batch/s]

Training batches on cuda:  25%|██▍       | 2400/9708 [00:11<00:34, 213.08batch/s]

Training batches on cuda:  25%|██▍       | 2423/9708 [00:11<00:33, 216.19batch/s]

Training batches on cuda:  25%|██▌       | 2445/9708 [00:11<00:35, 204.57batch/s]

Training batches on cuda:  25%|██▌       | 2466/9708 [00:11<00:38, 189.78batch/s]

Training batches on cuda:  26%|██▌       | 2488/9708 [00:11<00:36, 195.72batch/s]

Training batches on cuda:  26%|██▌       | 2511/9708 [00:12<00:35, 203.07batch/s]

Training batches on cuda:  26%|██▌       | 2535/9708 [00:12<00:33, 211.59batch/s]

Training batches on cuda:  26%|██▋       | 2559/9708 [00:12<00:32, 217.61batch/s]

Training batches on cuda:  27%|██▋       | 2582/9708 [00:12<00:32, 219.72batch/s]

Training batches on cuda:  27%|██▋       | 2605/9708 [00:12<00:32, 221.80batch/s]

Training batches on cuda:  27%|██▋       | 2629/9708 [00:12<00:31, 224.50batch/s]

Training batches on cuda:  27%|██▋       | 2653/9708 [00:12<00:31, 226.21batch/s]

Training batches on cuda:  28%|██▊       | 2676/9708 [00:12<00:31, 226.43batch/s]

Training batches on cuda:  28%|██▊       | 2699/9708 [00:12<00:30, 226.92batch/s]

Training batches on cuda:  28%|██▊       | 2722/9708 [00:12<00:30, 227.73batch/s]

Training batches on cuda:  28%|██▊       | 2745/9708 [00:13<00:31, 221.00batch/s]

Training batches on cuda:  29%|██▊       | 2768/9708 [00:13<00:32, 210.69batch/s]

Training batches on cuda:  29%|██▊       | 2790/9708 [00:13<00:34, 201.54batch/s]

Training batches on cuda:  29%|██▉       | 2811/9708 [00:13<00:34, 198.36batch/s]

Training batches on cuda:  29%|██▉       | 2831/9708 [00:13<00:35, 195.05batch/s]

Training batches on cuda:  29%|██▉       | 2852/9708 [00:13<00:34, 197.98batch/s]

Training batches on cuda:  30%|██▉       | 2873/9708 [00:13<00:34, 199.26batch/s]

Training batches on cuda:  30%|██▉       | 2894/9708 [00:13<00:33, 200.73batch/s]

Training batches on cuda:  30%|███       | 2915/9708 [00:13<00:34, 198.95batch/s]

Training batches on cuda:  30%|███       | 2935/9708 [00:14<00:33, 199.25batch/s]

Training batches on cuda:  30%|███       | 2955/9708 [00:14<00:35, 190.12batch/s]

Training batches on cuda:  31%|███       | 2980/9708 [00:14<00:33, 203.45batch/s]

Training batches on cuda:  31%|███       | 3005/9708 [00:14<00:31, 213.64batch/s]

Training batches on cuda:  31%|███       | 3031/9708 [00:14<00:29, 224.58batch/s]

Training batches on cuda:  31%|███▏      | 3058/9708 [00:14<00:28, 234.70batch/s]

Training batches on cuda:  32%|███▏      | 3085/9708 [00:14<00:27, 242.47batch/s]

Training batches on cuda:  32%|███▏      | 3110/9708 [00:14<00:27, 243.89batch/s]

Training batches on cuda:  32%|███▏      | 3135/9708 [00:14<00:29, 223.16batch/s]

Training batches on cuda:  33%|███▎      | 3158/9708 [00:14<00:30, 216.43batch/s]

Training batches on cuda:  33%|███▎      | 3181/9708 [00:15<00:30, 212.74batch/s]

Training batches on cuda:  33%|███▎      | 3203/9708 [00:15<00:31, 205.72batch/s]

Training batches on cuda:  33%|███▎      | 3224/9708 [00:15<00:31, 203.82batch/s]

Training batches on cuda:  33%|███▎      | 3245/9708 [00:15<00:31, 202.32batch/s]

Training batches on cuda:  34%|███▎      | 3268/9708 [00:15<00:30, 209.06batch/s]

Training batches on cuda:  34%|███▍      | 3291/9708 [00:15<00:30, 212.30batch/s]

Training batches on cuda:  34%|███▍      | 3313/9708 [00:15<00:30, 210.94batch/s]

Training batches on cuda:  34%|███▍      | 3335/9708 [00:15<00:30, 209.04batch/s]

Training batches on cuda:  35%|███▍      | 3357/9708 [00:15<00:30, 210.73batch/s]

Training batches on cuda:  35%|███▍      | 3379/9708 [00:16<00:30, 208.44batch/s]

Training batches on cuda:  35%|███▌      | 3400/9708 [00:16<00:31, 200.89batch/s]

Training batches on cuda:  35%|███▌      | 3421/9708 [00:16<00:30, 203.39batch/s]

Training batches on cuda:  35%|███▌      | 3443/9708 [00:16<00:30, 206.26batch/s]

Training batches on cuda:  36%|███▌      | 3465/9708 [00:16<00:29, 208.26batch/s]

Training batches on cuda:  36%|███▌      | 3487/9708 [00:16<00:29, 209.02batch/s]

Training batches on cuda:  36%|███▌      | 3509/9708 [00:16<00:29, 209.85batch/s]

Training batches on cuda:  36%|███▋      | 3531/9708 [00:16<00:30, 203.86batch/s]

Training batches on cuda:  37%|███▋      | 3552/9708 [00:16<00:30, 203.45batch/s]

Training batches on cuda:  37%|███▋      | 3573/9708 [00:17<00:30, 204.38batch/s]

Training batches on cuda:  37%|███▋      | 3594/9708 [00:17<00:30, 200.97batch/s]

Training batches on cuda:  37%|███▋      | 3615/9708 [00:17<00:31, 195.74batch/s]

Training batches on cuda:  37%|███▋      | 3635/9708 [00:17<00:31, 194.95batch/s]

Training batches on cuda:  38%|███▊      | 3656/9708 [00:17<00:30, 196.73batch/s]

Training batches on cuda:  38%|███▊      | 3677/9708 [00:17<00:30, 198.05batch/s]

Training batches on cuda:  38%|███▊      | 3698/9708 [00:17<00:30, 199.77batch/s]

Training batches on cuda:  38%|███▊      | 3718/9708 [00:17<00:30, 197.21batch/s]

Training batches on cuda:  39%|███▊      | 3738/9708 [00:17<00:30, 195.85batch/s]

Training batches on cuda:  39%|███▊      | 3760/9708 [00:17<00:29, 200.19batch/s]

Training batches on cuda:  39%|███▉      | 3782/9708 [00:18<00:29, 203.97batch/s]

Training batches on cuda:  39%|███▉      | 3806/9708 [00:18<00:27, 211.47batch/s]

Training batches on cuda:  39%|███▉      | 3829/9708 [00:18<00:27, 215.75batch/s]

Training batches on cuda:  40%|███▉      | 3852/9708 [00:18<00:26, 218.72batch/s]

Training batches on cuda:  40%|███▉      | 3875/9708 [00:18<00:26, 221.33batch/s]

Training batches on cuda:  40%|████      | 3898/9708 [00:18<00:26, 222.92batch/s]

Training batches on cuda:  40%|████      | 3921/9708 [00:18<00:26, 218.62batch/s]

Training batches on cuda:  41%|████      | 3943/9708 [00:18<00:27, 206.79batch/s]

Training batches on cuda:  41%|████      | 3964/9708 [00:18<00:29, 196.07batch/s]

Training batches on cuda:  41%|████      | 3984/9708 [00:19<00:29, 196.84batch/s]

Training batches on cuda:  41%|████▏     | 4006/9708 [00:19<00:28, 203.00batch/s]

Training batches on cuda:  42%|████▏     | 4029/9708 [00:19<00:27, 209.24batch/s]

Training batches on cuda:  42%|████▏     | 4052/9708 [00:19<00:26, 212.37batch/s]

Training batches on cuda:  42%|████▏     | 4074/9708 [00:19<00:28, 199.30batch/s]

Training batches on cuda:  42%|████▏     | 4096/9708 [00:19<00:27, 202.87batch/s]

Training batches on cuda:  42%|████▏     | 4117/9708 [00:19<00:27, 202.81batch/s]

Training batches on cuda:  43%|████▎     | 4138/9708 [00:19<00:27, 201.39batch/s]

Training batches on cuda:  43%|████▎     | 4159/9708 [00:19<00:30, 183.33batch/s]

Training batches on cuda:  43%|████▎     | 4178/9708 [00:20<00:30, 178.85batch/s]

Training batches on cuda:  43%|████▎     | 4202/9708 [00:20<00:28, 192.27batch/s]

Training batches on cuda:  44%|████▎     | 4226/9708 [00:20<00:27, 202.60batch/s]

Training batches on cuda:  44%|████▎     | 4247/9708 [00:20<00:26, 202.87batch/s]

Training batches on cuda:  44%|████▍     | 4268/9708 [00:20<00:27, 197.46batch/s]

Training batches on cuda:  44%|████▍     | 4290/9708 [00:20<00:26, 202.17batch/s]

Training batches on cuda:  44%|████▍     | 4313/9708 [00:20<00:26, 207.43batch/s]

Training batches on cuda:  45%|████▍     | 4334/9708 [00:20<00:26, 205.05batch/s]

Training batches on cuda:  45%|████▍     | 4355/9708 [00:20<00:26, 204.79batch/s]

Training batches on cuda:  45%|████▌     | 4376/9708 [00:20<00:26, 203.59batch/s]

Training batches on cuda:  45%|████▌     | 4397/9708 [00:21<00:26, 202.08batch/s]

Training batches on cuda:  46%|████▌     | 4418/9708 [00:21<00:27, 195.12batch/s]

Training batches on cuda:  46%|████▌     | 4438/9708 [00:21<00:27, 193.36batch/s]

Training batches on cuda:  46%|████▌     | 4458/9708 [00:21<00:27, 192.99batch/s]

Training batches on cuda:  46%|████▌     | 4482/9708 [00:21<00:25, 203.73batch/s]

Training batches on cuda:  46%|████▋     | 4507/9708 [00:21<00:24, 213.29batch/s]

Training batches on cuda:  47%|████▋     | 4531/9708 [00:21<00:23, 219.72batch/s]

Training batches on cuda:  47%|████▋     | 4554/9708 [00:21<00:23, 218.92batch/s]

Training batches on cuda:  47%|████▋     | 4579/9708 [00:21<00:22, 226.84batch/s]

Training batches on cuda:  47%|████▋     | 4604/9708 [00:22<00:21, 232.83batch/s]

Training batches on cuda:  48%|████▊     | 4628/9708 [00:22<00:21, 232.21batch/s]

Training batches on cuda:  48%|████▊     | 4652/9708 [00:22<00:22, 229.11batch/s]

Training batches on cuda:  48%|████▊     | 4676/9708 [00:22<00:22, 224.38batch/s]

Training batches on cuda:  48%|████▊     | 4699/9708 [00:22<00:22, 220.92batch/s]

Training batches on cuda:  49%|████▊     | 4722/9708 [00:22<00:22, 218.17batch/s]

Training batches on cuda:  49%|████▉     | 4744/9708 [00:22<00:22, 216.99batch/s]

Training batches on cuda:  49%|████▉     | 4766/9708 [00:22<00:22, 216.83batch/s]

Training batches on cuda:  49%|████▉     | 4789/9708 [00:22<00:22, 220.61batch/s]

Training batches on cuda:  50%|████▉     | 4812/9708 [00:22<00:22, 221.76batch/s]

Training batches on cuda:  50%|████▉     | 4835/9708 [00:23<00:21, 222.89batch/s]

Training batches on cuda:  50%|█████     | 4858/9708 [00:23<00:21, 222.48batch/s]

Training batches on cuda:  50%|█████     | 4882/9708 [00:23<00:21, 225.67batch/s]

Training batches on cuda:  51%|█████     | 4906/9708 [00:23<00:21, 227.82batch/s]

Training batches on cuda:  51%|█████     | 4930/9708 [00:23<00:20, 228.77batch/s]

Training batches on cuda:  51%|█████     | 4953/9708 [00:23<00:20, 228.48batch/s]

Training batches on cuda:  51%|█████▏    | 4977/9708 [00:23<00:20, 229.88batch/s]

Training batches on cuda:  52%|█████▏    | 5002/9708 [00:23<00:20, 233.91batch/s]

Training batches on cuda:  52%|█████▏    | 5026/9708 [00:23<00:21, 221.30batch/s]

Training batches on cuda:  52%|█████▏    | 5049/9708 [00:24<00:21, 214.93batch/s]

Training batches on cuda:  52%|█████▏    | 5071/9708 [00:24<00:23, 200.80batch/s]

Training batches on cuda:  52%|█████▏    | 5092/9708 [00:24<00:22, 203.39batch/s]

Training batches on cuda:  53%|█████▎    | 5116/9708 [00:24<00:21, 211.53batch/s]

Training batches on cuda:  53%|█████▎    | 5138/9708 [00:24<00:21, 211.86batch/s]

Training batches on cuda:  53%|█████▎    | 5160/9708 [00:24<00:22, 205.44batch/s]

Training batches on cuda:  53%|█████▎    | 5181/9708 [00:24<00:22, 200.87batch/s]

Training batches on cuda:  54%|█████▎    | 5202/9708 [00:24<00:22, 198.02batch/s]

Training batches on cuda:  54%|█████▍    | 5223/9708 [00:24<00:22, 201.07batch/s]

Training batches on cuda:  54%|█████▍    | 5244/9708 [00:24<00:22, 200.78batch/s]

Training batches on cuda:  54%|█████▍    | 5265/9708 [00:25<00:22, 201.81batch/s]

Training batches on cuda:  54%|█████▍    | 5286/9708 [00:25<00:21, 202.75batch/s]

Training batches on cuda:  55%|█████▍    | 5307/9708 [00:25<00:21, 202.39batch/s]

Training batches on cuda:  55%|█████▍    | 5328/9708 [00:25<00:21, 203.75batch/s]

Training batches on cuda:  55%|█████▌    | 5350/9708 [00:25<00:20, 207.94batch/s]

Training batches on cuda:  55%|█████▌    | 5372/9708 [00:25<00:20, 210.89batch/s]

Training batches on cuda:  56%|█████▌    | 5394/9708 [00:25<00:20, 212.26batch/s]

Training batches on cuda:  56%|█████▌    | 5416/9708 [00:25<00:20, 212.78batch/s]

Training batches on cuda:  56%|█████▌    | 5438/9708 [00:25<00:22, 187.84batch/s]

Training batches on cuda:  56%|█████▌    | 5460/9708 [00:26<00:21, 196.27batch/s]

Training batches on cuda:  56%|█████▋    | 5482/9708 [00:26<00:21, 200.62batch/s]

Training batches on cuda:  57%|█████▋    | 5503/9708 [00:26<00:20, 200.41batch/s]

Training batches on cuda:  57%|█████▋    | 5524/9708 [00:26<00:21, 197.61batch/s]

Training batches on cuda:  57%|█████▋    | 5544/9708 [00:26<00:21, 192.69batch/s]

Training batches on cuda:  57%|█████▋    | 5564/9708 [00:26<00:21, 192.69batch/s]

Training batches on cuda:  58%|█████▊    | 5584/9708 [00:26<00:21, 189.62batch/s]

Training batches on cuda:  58%|█████▊    | 5605/9708 [00:26<00:21, 194.26batch/s]

Training batches on cuda:  58%|█████▊    | 5625/9708 [00:26<00:21, 191.42batch/s]

Training batches on cuda:  58%|█████▊    | 5647/9708 [00:27<00:20, 197.58batch/s]

Training batches on cuda:  58%|█████▊    | 5670/9708 [00:27<00:19, 204.69batch/s]

Training batches on cuda:  59%|█████▊    | 5696/9708 [00:27<00:18, 217.43batch/s]

Training batches on cuda:  59%|█████▉    | 5719/9708 [00:27<00:18, 218.83batch/s]

Training batches on cuda:  59%|█████▉    | 5742/9708 [00:27<00:18, 219.91batch/s]

Training batches on cuda:  59%|█████▉    | 5765/9708 [00:27<00:18, 216.42batch/s]

Training batches on cuda:  60%|█████▉    | 5787/9708 [00:27<00:18, 212.22batch/s]

Training batches on cuda:  60%|█████▉    | 5809/9708 [00:27<00:18, 211.40batch/s]

Training batches on cuda:  60%|██████    | 5831/9708 [00:27<00:18, 207.53batch/s]

Training batches on cuda:  60%|██████    | 5854/9708 [00:27<00:18, 212.82batch/s]

Training batches on cuda:  61%|██████    | 5877/9708 [00:28<00:17, 215.09batch/s]

Training batches on cuda:  61%|██████    | 5899/9708 [00:28<00:18, 208.76batch/s]

Training batches on cuda:  61%|██████    | 5920/9708 [00:28<00:18, 208.87batch/s]

Training batches on cuda:  61%|██████    | 5942/9708 [00:28<00:17, 209.54batch/s]

Training batches on cuda:  61%|██████▏   | 5963/9708 [00:28<00:17, 209.31batch/s]

Training batches on cuda:  62%|██████▏   | 5984/9708 [00:28<00:18, 206.28batch/s]

Training batches on cuda:  62%|██████▏   | 6005/9708 [00:28<00:18, 204.68batch/s]

Training batches on cuda:  62%|██████▏   | 6026/9708 [00:28<00:18, 203.22batch/s]

Training batches on cuda:  62%|██████▏   | 6048/9708 [00:28<00:17, 205.48batch/s]

Training batches on cuda:  63%|██████▎   | 6069/9708 [00:28<00:17, 205.99batch/s]

Training batches on cuda:  63%|██████▎   | 6090/9708 [00:29<00:18, 199.25batch/s]

Training batches on cuda:  63%|██████▎   | 6114/9708 [00:29<00:17, 208.97batch/s]

Training batches on cuda:  63%|██████▎   | 6139/9708 [00:29<00:16, 217.79batch/s]

Training batches on cuda:  63%|██████▎   | 6164/9708 [00:29<00:15, 224.39batch/s]

Training batches on cuda:  64%|██████▍   | 6190/9708 [00:29<00:15, 232.56batch/s]

Training batches on cuda:  64%|██████▍   | 6214/9708 [00:29<00:15, 231.33batch/s]

Training batches on cuda:  64%|██████▍   | 6238/9708 [00:29<00:15, 225.29batch/s]

Training batches on cuda:  64%|██████▍   | 6261/9708 [00:29<00:15, 220.61batch/s]

Training batches on cuda:  65%|██████▍   | 6284/9708 [00:29<00:16, 207.92batch/s]

Training batches on cuda:  65%|██████▍   | 6306/9708 [00:30<00:17, 198.27batch/s]

Training batches on cuda:  65%|██████▌   | 6327/9708 [00:30<00:17, 190.58batch/s]

Training batches on cuda:  65%|██████▌   | 6347/9708 [00:30<00:17, 190.23batch/s]

Training batches on cuda:  66%|██████▌   | 6368/9708 [00:30<00:17, 194.16batch/s]

Training batches on cuda:  66%|██████▌   | 6390/9708 [00:30<00:16, 199.21batch/s]

Training batches on cuda:  66%|██████▌   | 6411/9708 [00:30<00:17, 190.78batch/s]

Training batches on cuda:  66%|██████▌   | 6431/9708 [00:30<00:17, 188.25batch/s]

Training batches on cuda:  66%|██████▋   | 6450/9708 [00:30<00:17, 186.37batch/s]

Training batches on cuda:  67%|██████▋   | 6469/9708 [00:31<00:19, 162.40batch/s]

Training batches on cuda:  67%|██████▋   | 6491/9708 [00:31<00:18, 176.18batch/s]

Training batches on cuda:  67%|██████▋   | 6511/9708 [00:31<00:17, 181.71batch/s]

Training batches on cuda:  67%|██████▋   | 6532/9708 [00:31<00:16, 187.95batch/s]

Training batches on cuda:  68%|██████▊   | 6555/9708 [00:31<00:16, 196.55batch/s]

Training batches on cuda:  68%|██████▊   | 6577/9708 [00:31<00:15, 201.49batch/s]

Training batches on cuda:  68%|██████▊   | 6598/9708 [00:31<00:15, 203.27batch/s]

Training batches on cuda:  68%|██████▊   | 6623/9708 [00:31<00:14, 212.56batch/s]

Training batches on cuda:  68%|██████▊   | 6645/9708 [00:31<00:14, 211.34batch/s]

Training batches on cuda:  69%|██████▊   | 6667/9708 [00:31<00:14, 207.16batch/s]

Training batches on cuda:  69%|██████▉   | 6690/9708 [00:32<00:14, 212.00batch/s]

Training batches on cuda:  69%|██████▉   | 6712/9708 [00:32<00:14, 206.30batch/s]

Training batches on cuda:  69%|██████▉   | 6733/9708 [00:32<00:14, 199.11batch/s]

Training batches on cuda:  70%|██████▉   | 6754/9708 [00:32<00:14, 198.99batch/s]

Training batches on cuda:  70%|██████▉   | 6777/9708 [00:32<00:14, 206.68batch/s]

Training batches on cuda:  70%|███████   | 6800/9708 [00:32<00:13, 210.93batch/s]

Training batches on cuda:  70%|███████   | 6822/9708 [00:32<00:13, 211.78batch/s]

Training batches on cuda:  70%|███████   | 6844/9708 [00:32<00:13, 212.86batch/s]

Training batches on cuda:  71%|███████   | 6867/9708 [00:32<00:13, 216.82batch/s]

Training batches on cuda:  71%|███████   | 6890/9708 [00:33<00:12, 219.88batch/s]

Training batches on cuda:  71%|███████   | 6913/9708 [00:33<00:12, 215.93batch/s]

Training batches on cuda:  71%|███████▏  | 6937/9708 [00:33<00:12, 221.54batch/s]

Training batches on cuda:  72%|███████▏  | 6961/9708 [00:33<00:12, 224.60batch/s]

Training batches on cuda:  72%|███████▏  | 6984/9708 [00:33<00:12, 225.56batch/s]

Training batches on cuda:  72%|███████▏  | 7007/9708 [00:33<00:12, 214.30batch/s]

Training batches on cuda:  72%|███████▏  | 7029/9708 [00:33<00:12, 210.35batch/s]

Training batches on cuda:  73%|███████▎  | 7051/9708 [00:33<00:12, 206.50batch/s]

Training batches on cuda:  73%|███████▎  | 7072/9708 [00:33<00:12, 203.17batch/s]

Training batches on cuda:  73%|███████▎  | 7093/9708 [00:33<00:12, 203.69batch/s]

Training batches on cuda:  73%|███████▎  | 7114/9708 [00:34<00:12, 202.71batch/s]

Training batches on cuda:  73%|███████▎  | 7135/9708 [00:34<00:13, 195.64batch/s]

Training batches on cuda:  74%|███████▎  | 7155/9708 [00:34<00:13, 192.03batch/s]

Training batches on cuda:  74%|███████▍  | 7176/9708 [00:34<00:12, 195.35batch/s]

Training batches on cuda:  74%|███████▍  | 7197/9708 [00:34<00:12, 198.31batch/s]

Training batches on cuda:  74%|███████▍  | 7220/9708 [00:34<00:12, 205.71batch/s]

Training batches on cuda:  75%|███████▍  | 7241/9708 [00:34<00:12, 204.67batch/s]

Training batches on cuda:  75%|███████▍  | 7263/9708 [00:34<00:11, 206.56batch/s]

Training batches on cuda:  75%|███████▌  | 7284/9708 [00:34<00:11, 205.59batch/s]

Training batches on cuda:  75%|███████▌  | 7306/9708 [00:35<00:11, 207.25batch/s]

Training batches on cuda:  75%|███████▌  | 7327/9708 [00:35<00:11, 203.72batch/s]

Training batches on cuda:  76%|███████▌  | 7348/9708 [00:35<00:11, 201.04batch/s]

Training batches on cuda:  76%|███████▌  | 7369/9708 [00:35<00:11, 197.29batch/s]

Training batches on cuda:  76%|███████▌  | 7389/9708 [00:35<00:11, 196.00batch/s]

Training batches on cuda:  76%|███████▋  | 7410/9708 [00:35<00:11, 199.81batch/s]

Training batches on cuda:  77%|███████▋  | 7431/9708 [00:35<00:11, 199.81batch/s]

Training batches on cuda:  77%|███████▋  | 7452/9708 [00:35<00:11, 201.52batch/s]

Training batches on cuda:  77%|███████▋  | 7473/9708 [00:35<00:11, 195.47batch/s]

Training batches on cuda:  77%|███████▋  | 7493/9708 [00:36<00:12, 172.37batch/s]

Training batches on cuda:  77%|███████▋  | 7514/9708 [00:36<00:12, 181.79batch/s]

Training batches on cuda:  78%|███████▊  | 7533/9708 [00:36<00:11, 182.83batch/s]

Training batches on cuda:  78%|███████▊  | 7553/9708 [00:36<00:11, 185.05batch/s]

Training batches on cuda:  78%|███████▊  | 7573/9708 [00:36<00:11, 187.03batch/s]

Training batches on cuda:  78%|███████▊  | 7593/9708 [00:36<00:11, 190.40batch/s]

Training batches on cuda:  78%|███████▊  | 7616/9708 [00:36<00:10, 200.63batch/s]

Training batches on cuda:  79%|███████▊  | 7638/9708 [00:36<00:10, 205.24batch/s]

Training batches on cuda:  79%|███████▉  | 7659/9708 [00:36<00:10, 197.02batch/s]

Training batches on cuda:  79%|███████▉  | 7679/9708 [00:36<00:10, 194.75batch/s]

Training batches on cuda:  79%|███████▉  | 7700/9708 [00:37<00:10, 197.30batch/s]

Training batches on cuda:  80%|███████▉  | 7721/9708 [00:37<00:09, 199.01batch/s]

Training batches on cuda:  80%|███████▉  | 7743/9708 [00:37<00:09, 204.07batch/s]

Training batches on cuda:  80%|████████  | 7767/9708 [00:37<00:09, 212.35batch/s]

Training batches on cuda:  80%|████████  | 7791/9708 [00:37<00:08, 217.71batch/s]

Training batches on cuda:  81%|████████  | 7815/9708 [00:37<00:08, 222.57batch/s]

Training batches on cuda:  81%|████████  | 7839/9708 [00:37<00:08, 225.00batch/s]

Training batches on cuda:  81%|████████  | 7863/9708 [00:37<00:08, 227.96batch/s]

Training batches on cuda:  81%|████████  | 7886/9708 [00:37<00:08, 227.74batch/s]

Training batches on cuda:  81%|████████▏ | 7909/9708 [00:37<00:07, 227.53batch/s]

Training batches on cuda:  82%|████████▏ | 7932/9708 [00:38<00:07, 228.07batch/s]

Training batches on cuda:  82%|████████▏ | 7956/9708 [00:38<00:07, 231.03batch/s]

Training batches on cuda:  82%|████████▏ | 7980/9708 [00:38<00:07, 230.95batch/s]

Training batches on cuda:  82%|████████▏ | 8004/9708 [00:38<00:07, 227.65batch/s]

Training batches on cuda:  83%|████████▎ | 8027/9708 [00:38<00:07, 227.00batch/s]

Training batches on cuda:  83%|████████▎ | 8051/9708 [00:38<00:07, 229.62batch/s]

Training batches on cuda:  83%|████████▎ | 8074/9708 [00:38<00:07, 228.88batch/s]

Training batches on cuda:  83%|████████▎ | 8097/9708 [00:38<00:07, 218.91batch/s]

Training batches on cuda:  84%|████████▎ | 8119/9708 [00:38<00:07, 219.16batch/s]

Training batches on cuda:  84%|████████▍ | 8141/9708 [00:39<00:07, 217.72batch/s]

Training batches on cuda:  84%|████████▍ | 8164/9708 [00:39<00:07, 218.77batch/s]

Training batches on cuda:  84%|████████▍ | 8187/9708 [00:39<00:06, 221.82batch/s]

Training batches on cuda:  85%|████████▍ | 8210/9708 [00:39<00:06, 221.46batch/s]

Training batches on cuda:  85%|████████▍ | 8233/9708 [00:39<00:06, 216.46batch/s]

Training batches on cuda:  85%|████████▌ | 8255/9708 [00:39<00:06, 215.17batch/s]

Training batches on cuda:  85%|████████▌ | 8277/9708 [00:39<00:06, 216.35batch/s]

Training batches on cuda:  85%|████████▌ | 8299/9708 [00:39<00:06, 215.65batch/s]

Training batches on cuda:  86%|████████▌ | 8321/9708 [00:39<00:06, 210.86batch/s]

Training batches on cuda:  86%|████████▌ | 8343/9708 [00:39<00:06, 204.73batch/s]

Training batches on cuda:  86%|████████▌ | 8364/9708 [00:40<00:06, 202.36batch/s]

Training batches on cuda:  86%|████████▋ | 8386/9708 [00:40<00:06, 204.96batch/s]

Training batches on cuda:  87%|████████▋ | 8407/9708 [00:40<00:06, 205.38batch/s]

Training batches on cuda:  87%|████████▋ | 8429/9708 [00:40<00:06, 208.53batch/s]

Training batches on cuda:  87%|████████▋ | 8450/9708 [00:40<00:06, 208.85batch/s]

Training batches on cuda:  87%|████████▋ | 8471/9708 [00:40<00:05, 207.35batch/s]

Training batches on cuda:  87%|████████▋ | 8492/9708 [00:40<00:05, 203.18batch/s]

Training batches on cuda:  88%|████████▊ | 8514/9708 [00:40<00:05, 205.72batch/s]

Training batches on cuda:  88%|████████▊ | 8535/9708 [00:40<00:05, 202.55batch/s]

Training batches on cuda:  88%|████████▊ | 8556/9708 [00:41<00:06, 191.25batch/s]

Training batches on cuda:  88%|████████▊ | 8576/9708 [00:41<00:05, 190.13batch/s]

Training batches on cuda:  89%|████████▊ | 8598/9708 [00:41<00:05, 196.37batch/s]

Training batches on cuda:  89%|████████▉ | 8622/9708 [00:41<00:05, 205.75batch/s]

Training batches on cuda:  89%|████████▉ | 8646/9708 [00:41<00:04, 213.33batch/s]

Training batches on cuda:  89%|████████▉ | 8669/9708 [00:41<00:04, 217.73batch/s]

Training batches on cuda:  90%|████████▉ | 8693/9708 [00:41<00:04, 221.42batch/s]

Training batches on cuda:  90%|████████▉ | 8716/9708 [00:41<00:04, 223.30batch/s]

Training batches on cuda:  90%|█████████ | 8739/9708 [00:41<00:04, 212.72batch/s]

Training batches on cuda:  90%|█████████ | 8761/9708 [00:41<00:04, 203.71batch/s]

Training batches on cuda:  90%|█████████ | 8782/9708 [00:42<00:04, 202.51batch/s]

Training batches on cuda:  91%|█████████ | 8803/9708 [00:42<00:04, 203.29batch/s]

Training batches on cuda:  91%|█████████ | 8825/9708 [00:42<00:04, 205.61batch/s]

Training batches on cuda:  91%|█████████ | 8847/9708 [00:42<00:04, 208.12batch/s]

Training batches on cuda:  91%|█████████▏| 8868/9708 [00:42<00:04, 208.52batch/s]

Training batches on cuda:  92%|█████████▏| 8890/9708 [00:42<00:03, 210.46batch/s]

Training batches on cuda:  92%|█████████▏| 8912/9708 [00:42<00:03, 208.98batch/s]

Training batches on cuda:  92%|█████████▏| 8933/9708 [00:42<00:03, 206.21batch/s]

Training batches on cuda:  92%|█████████▏| 8954/9708 [00:42<00:03, 204.57batch/s]

Training batches on cuda:  92%|█████████▏| 8975/9708 [00:43<00:03, 203.98batch/s]

Training batches on cuda:  93%|█████████▎| 8996/9708 [00:43<00:03, 204.61batch/s]

Training batches on cuda:  93%|█████████▎| 9017/9708 [00:43<00:03, 203.20batch/s]

Training batches on cuda:  93%|█████████▎| 9039/9708 [00:43<00:03, 205.20batch/s]

Training batches on cuda:  93%|█████████▎| 9061/9708 [00:43<00:03, 208.41batch/s]

Training batches on cuda:  94%|█████████▎| 9084/9708 [00:43<00:02, 212.59batch/s]

Training batches on cuda:  94%|█████████▍| 9107/9708 [00:43<00:02, 214.26batch/s]

Training batches on cuda:  94%|█████████▍| 9129/9708 [00:43<00:02, 213.59batch/s]

Training batches on cuda:  94%|█████████▍| 9151/9708 [00:43<00:02, 213.65batch/s]

Training batches on cuda:  94%|█████████▍| 9173/9708 [00:43<00:02, 213.31batch/s]

Training batches on cuda:  95%|█████████▍| 9195/9708 [00:44<00:02, 212.95batch/s]

Training batches on cuda:  95%|█████████▍| 9217/9708 [00:44<00:02, 212.50batch/s]

Training batches on cuda:  95%|█████████▌| 9239/9708 [00:44<00:02, 212.66batch/s]

Training batches on cuda:  95%|█████████▌| 9261/9708 [00:44<00:02, 212.09batch/s]

Training batches on cuda:  96%|█████████▌| 9284/9708 [00:44<00:01, 214.36batch/s]

Training batches on cuda:  96%|█████████▌| 9307/9708 [00:44<00:01, 217.05batch/s]

Training batches on cuda:  96%|█████████▌| 9331/9708 [00:44<00:01, 221.88batch/s]

Training batches on cuda:  96%|█████████▋| 9355/9708 [00:44<00:01, 226.55batch/s]

Training batches on cuda:  97%|█████████▋| 9379/9708 [00:44<00:01, 229.55batch/s]

Training batches on cuda:  97%|█████████▋| 9403/9708 [00:44<00:01, 230.88batch/s]

Training batches on cuda:  97%|█████████▋| 9427/9708 [00:45<00:01, 232.44batch/s]

Training batches on cuda:  97%|█████████▋| 9451/9708 [00:45<00:01, 233.72batch/s]

Training batches on cuda:  98%|█████████▊| 9475/9708 [00:45<00:00, 233.99batch/s]

Training batches on cuda:  98%|█████████▊| 9499/9708 [00:45<00:00, 231.79batch/s]

Training batches on cuda:  98%|█████████▊| 9523/9708 [00:45<00:00, 231.14batch/s]

Training batches on cuda:  98%|█████████▊| 9547/9708 [00:45<00:00, 229.90batch/s]

Training batches on cuda:  99%|█████████▊| 9571/9708 [00:45<00:00, 229.64batch/s]

Training batches on cuda:  99%|█████████▉| 9594/9708 [00:45<00:00, 228.06batch/s]

Training batches on cuda:  99%|█████████▉| 9617/9708 [00:45<00:00, 227.84batch/s]

Training batches on cuda:  99%|█████████▉| 9640/9708 [00:46<00:00, 196.37batch/s]

Training batches on cuda: 100%|█████████▉| 9663/9708 [00:46<00:00, 205.26batch/s]

Training batches on cuda: 100%|█████████▉| 9686/9708 [00:46<00:00, 210.81batch/s]

Training batches on cuda: 100%|██████████| 9708/9708 [00:46<00:00, 212.48batch/s]

                                                                                 
Training epochs on cuda:  80%|████████  | 8/10 [06:55<01:31, 45.77s/epoch, loss=0.096, prev_loss=0.096]
Training epochs on cuda:  90%|█████████ | 9/10 [06:55<00:46, 46.03s/epoch, loss=0.096, prev_loss=0.096]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 19/9708 [00:00<00:52, 184.06batch/s]

Training batches on cuda:   0%|          | 43/9708 [00:00<00:49, 195.86batch/s]

Training batches on cuda:   1%|          | 67/9708 [00:00<00:46, 206.19batch/s]

Training batches on cuda:   1%|          | 91/9708 [00:00<00:44, 213.97batch/s]

Training batches on cuda:   1%|          | 115/9708 [00:00<00:43, 218.67batch/s]

Training batches on cuda:   1%|▏         | 139/9708 [00:00<00:42, 223.37batch/s]

Training batches on cuda:   2%|▏         | 163/9708 [00:00<00:42, 226.02batch/s]

Training batches on cuda:   2%|▏         | 187/9708 [00:00<00:41, 227.91batch/s]

Training batches on cuda:   2%|▏         | 211/9708 [00:00<00:41, 229.20batch/s]

Training batches on cuda:   2%|▏         | 234/9708 [00:01<00:41, 228.24batch/s]

Training batches on cuda:   3%|▎         | 257/9708 [00:01<00:41, 228.00batch/s]

Training batches on cuda:   3%|▎         | 280/9708 [00:01<00:41, 227.49batch/s]

Training batches on cuda:   3%|▎         | 303/9708 [00:01<00:41, 227.71batch/s]

Training batches on cuda:   3%|▎         | 327/9708 [00:01<00:40, 229.36batch/s]

Training batches on cuda:   4%|▎         | 350/9708 [00:01<00:41, 226.81batch/s]

Training batches on cuda:   4%|▍         | 373/9708 [00:01<00:42, 220.83batch/s]

Training batches on cuda:   4%|▍         | 396/9708 [00:01<00:42, 217.04batch/s]

Training batches on cuda:   4%|▍         | 418/9708 [00:01<00:43, 213.09batch/s]

Training batches on cuda:   5%|▍         | 440/9708 [00:01<00:43, 214.51batch/s]

Training batches on cuda:   5%|▍         | 462/9708 [00:02<00:43, 213.45batch/s]

Training batches on cuda:   5%|▍         | 484/9708 [00:02<00:43, 212.13batch/s]

Training batches on cuda:   5%|▌         | 506/9708 [00:02<00:43, 210.78batch/s]

Training batches on cuda:   5%|▌         | 528/9708 [00:02<00:43, 210.83batch/s]

Training batches on cuda:   6%|▌         | 550/9708 [00:02<00:44, 204.79batch/s]

Training batches on cuda:   6%|▌         | 572/9708 [00:02<00:43, 207.84batch/s]

Training batches on cuda:   6%|▌         | 594/9708 [00:02<00:43, 209.76batch/s]

Training batches on cuda:   6%|▋         | 616/9708 [00:02<00:43, 211.11batch/s]

Training batches on cuda:   7%|▋         | 638/9708 [00:02<00:42, 212.25batch/s]

Training batches on cuda:   7%|▋         | 660/9708 [00:03<00:42, 213.43batch/s]

Training batches on cuda:   7%|▋         | 682/9708 [00:03<00:42, 213.71batch/s]

Training batches on cuda:   7%|▋         | 705/9708 [00:03<00:41, 217.59batch/s]

Training batches on cuda:   7%|▋         | 728/9708 [00:03<00:40, 219.99batch/s]

Training batches on cuda:   8%|▊         | 751/9708 [00:03<00:40, 220.97batch/s]

Training batches on cuda:   8%|▊         | 774/9708 [00:03<00:40, 222.50batch/s]

Training batches on cuda:   8%|▊         | 797/9708 [00:03<00:39, 224.18batch/s]

Training batches on cuda:   8%|▊         | 821/9708 [00:03<00:39, 225.87batch/s]

Training batches on cuda:   9%|▊         | 844/9708 [00:03<00:39, 224.11batch/s]

Training batches on cuda:   9%|▉         | 868/9708 [00:03<00:38, 227.90batch/s]

Training batches on cuda:   9%|▉         | 891/9708 [00:04<00:39, 225.63batch/s]

Training batches on cuda:   9%|▉         | 914/9708 [00:04<00:39, 222.86batch/s]

Training batches on cuda:  10%|▉         | 937/9708 [00:04<00:39, 224.83batch/s]

Training batches on cuda:  10%|▉         | 961/9708 [00:04<00:38, 226.97batch/s]

Training batches on cuda:  10%|█         | 984/9708 [00:04<00:41, 208.62batch/s]

Training batches on cuda:  10%|█         | 1006/9708 [00:04<00:41, 208.88batch/s]

Training batches on cuda:  11%|█         | 1028/9708 [00:04<00:41, 210.89batch/s]

Training batches on cuda:  11%|█         | 1050/9708 [00:04<00:40, 211.35batch/s]

Training batches on cuda:  11%|█         | 1072/9708 [00:04<00:40, 211.87batch/s]

Training batches on cuda:  11%|█▏        | 1094/9708 [00:04<00:40, 212.01batch/s]

Training batches on cuda:  11%|█▏        | 1116/9708 [00:05<00:40, 212.56batch/s]

Training batches on cuda:  12%|█▏        | 1140/9708 [00:05<00:39, 218.01batch/s]

Training batches on cuda:  12%|█▏        | 1162/9708 [00:05<00:40, 210.14batch/s]

Training batches on cuda:  12%|█▏        | 1184/9708 [00:05<00:41, 205.51batch/s]

Training batches on cuda:  12%|█▏        | 1205/9708 [00:05<00:43, 195.78batch/s]

Training batches on cuda:  13%|█▎        | 1227/9708 [00:05<00:42, 201.60batch/s]

Training batches on cuda:  13%|█▎        | 1249/9708 [00:05<00:40, 206.73batch/s]

Training batches on cuda:  13%|█▎        | 1270/9708 [00:05<00:40, 206.90batch/s]

Training batches on cuda:  13%|█▎        | 1291/9708 [00:05<00:41, 204.75batch/s]

Training batches on cuda:  14%|█▎        | 1312/9708 [00:06<00:41, 203.42batch/s]

Training batches on cuda:  14%|█▎        | 1334/9708 [00:06<00:40, 206.97batch/s]

Training batches on cuda:  14%|█▍        | 1356/9708 [00:06<00:39, 209.69batch/s]

Training batches on cuda:  14%|█▍        | 1378/9708 [00:06<00:39, 210.09batch/s]

Training batches on cuda:  14%|█▍        | 1400/9708 [00:06<00:39, 208.85batch/s]

Training batches on cuda:  15%|█▍        | 1421/9708 [00:06<00:39, 208.91batch/s]

Training batches on cuda:  15%|█▍        | 1442/9708 [00:06<00:39, 208.15batch/s]

Training batches on cuda:  15%|█▌        | 1463/9708 [00:06<00:40, 205.91batch/s]

Training batches on cuda:  15%|█▌        | 1484/9708 [00:06<00:40, 204.86batch/s]

Training batches on cuda:  16%|█▌        | 1505/9708 [00:06<00:40, 204.04batch/s]

Training batches on cuda:  16%|█▌        | 1526/9708 [00:07<00:40, 202.73batch/s]

Training batches on cuda:  16%|█▌        | 1547/9708 [00:07<00:40, 202.32batch/s]

Training batches on cuda:  16%|█▌        | 1568/9708 [00:07<00:40, 201.28batch/s]

Training batches on cuda:  16%|█▋        | 1589/9708 [00:07<00:40, 201.26batch/s]

Training batches on cuda:  17%|█▋        | 1610/9708 [00:07<00:40, 200.95batch/s]

Training batches on cuda:  17%|█▋        | 1631/9708 [00:07<00:40, 200.13batch/s]

Training batches on cuda:  17%|█▋        | 1652/9708 [00:07<00:40, 200.14batch/s]

Training batches on cuda:  17%|█▋        | 1673/9708 [00:07<00:40, 200.84batch/s]

Training batches on cuda:  17%|█▋        | 1694/9708 [00:07<00:39, 201.77batch/s]

Training batches on cuda:  18%|█▊        | 1716/9708 [00:08<00:39, 204.82batch/s]

Training batches on cuda:  18%|█▊        | 1739/9708 [00:08<00:37, 209.75batch/s]

Training batches on cuda:  18%|█▊        | 1763/9708 [00:08<00:36, 215.86batch/s]

Training batches on cuda:  18%|█▊        | 1787/9708 [00:08<00:36, 219.84batch/s]

Training batches on cuda:  19%|█▊        | 1810/9708 [00:08<00:35, 222.38batch/s]

Training batches on cuda:  19%|█▉        | 1833/9708 [00:08<00:35, 224.31batch/s]

Training batches on cuda:  19%|█▉        | 1857/9708 [00:08<00:34, 226.41batch/s]

Training batches on cuda:  19%|█▉        | 1880/9708 [00:08<00:34, 225.94batch/s]

Training batches on cuda:  20%|█▉        | 1903/9708 [00:08<00:35, 218.32batch/s]

Training batches on cuda:  20%|█▉        | 1925/9708 [00:08<00:37, 207.23batch/s]

Training batches on cuda:  20%|██        | 1946/9708 [00:09<00:37, 206.20batch/s]

Training batches on cuda:  20%|██        | 1968/9708 [00:09<00:36, 209.20batch/s]

Training batches on cuda:  20%|██        | 1990/9708 [00:09<00:36, 208.87batch/s]

Training batches on cuda:  21%|██        | 2011/9708 [00:09<00:37, 206.19batch/s]

Training batches on cuda:  21%|██        | 2032/9708 [00:09<00:40, 188.50batch/s]

Training batches on cuda:  21%|██        | 2053/9708 [00:09<00:39, 194.47batch/s]

Training batches on cuda:  21%|██▏       | 2075/9708 [00:09<00:38, 199.06batch/s]

Training batches on cuda:  22%|██▏       | 2096/9708 [00:09<00:37, 200.56batch/s]

Training batches on cuda:  22%|██▏       | 2117/9708 [00:09<00:38, 198.69batch/s]

Training batches on cuda:  22%|██▏       | 2138/9708 [00:10<00:37, 199.34batch/s]

Training batches on cuda:  22%|██▏       | 2159/9708 [00:10<00:38, 197.95batch/s]

Training batches on cuda:  22%|██▏       | 2182/9708 [00:10<00:36, 204.99batch/s]

Training batches on cuda:  23%|██▎       | 2205/9708 [00:10<00:35, 210.54batch/s]

Training batches on cuda:  23%|██▎       | 2227/9708 [00:10<00:35, 207.82batch/s]

Training batches on cuda:  23%|██▎       | 2248/9708 [00:10<00:37, 200.83batch/s]

Training batches on cuda:  23%|██▎       | 2269/9708 [00:10<00:36, 201.34batch/s]

Training batches on cuda:  24%|██▎       | 2290/9708 [00:10<00:39, 185.82batch/s]

Training batches on cuda:  24%|██▍       | 2311/9708 [00:10<00:38, 190.97batch/s]

Training batches on cuda:  24%|██▍       | 2331/9708 [00:11<00:39, 187.94batch/s]

Training batches on cuda:  24%|██▍       | 2353/9708 [00:11<00:37, 194.40batch/s]

Training batches on cuda:  24%|██▍       | 2373/9708 [00:11<00:37, 194.77batch/s]

Training batches on cuda:  25%|██▍       | 2393/9708 [00:11<00:38, 191.92batch/s]

Training batches on cuda:  25%|██▍       | 2413/9708 [00:11<00:38, 190.74batch/s]

Training batches on cuda:  25%|██▌       | 2433/9708 [00:11<00:38, 189.52batch/s]

Training batches on cuda:  25%|██▌       | 2453/9708 [00:11<00:38, 188.52batch/s]

Training batches on cuda:  25%|██▌       | 2472/9708 [00:11<00:38, 188.53batch/s]

Training batches on cuda:  26%|██▌       | 2494/9708 [00:11<00:36, 195.06batch/s]

Training batches on cuda:  26%|██▌       | 2516/9708 [00:11<00:36, 199.55batch/s]

Training batches on cuda:  26%|██▌       | 2537/9708 [00:12<00:35, 202.04batch/s]

Training batches on cuda:  26%|██▋       | 2558/9708 [00:12<00:36, 195.53batch/s]

Training batches on cuda:  27%|██▋       | 2578/9708 [00:12<00:36, 196.42batch/s]

Training batches on cuda:  27%|██▋       | 2598/9708 [00:12<00:36, 194.99batch/s]

Training batches on cuda:  27%|██▋       | 2620/9708 [00:12<00:35, 201.73batch/s]

Training batches on cuda:  27%|██▋       | 2642/9708 [00:12<00:34, 205.13batch/s]

Training batches on cuda:  27%|██▋       | 2663/9708 [00:12<00:35, 199.69batch/s]

Training batches on cuda:  28%|██▊       | 2684/9708 [00:12<00:35, 199.97batch/s]

Training batches on cuda:  28%|██▊       | 2706/9708 [00:12<00:34, 204.36batch/s]

Training batches on cuda:  28%|██▊       | 2727/9708 [00:13<00:34, 203.83batch/s]

Training batches on cuda:  28%|██▊       | 2748/9708 [00:13<00:34, 201.49batch/s]

Training batches on cuda:  29%|██▊       | 2772/9708 [00:13<00:32, 210.30batch/s]

Training batches on cuda:  29%|██▉       | 2794/9708 [00:13<00:32, 211.37batch/s]

Training batches on cuda:  29%|██▉       | 2816/9708 [00:13<00:33, 207.83batch/s]

Training batches on cuda:  29%|██▉       | 2839/9708 [00:13<00:32, 212.62batch/s]

Training batches on cuda:  29%|██▉       | 2861/9708 [00:13<00:32, 209.57batch/s]

Training batches on cuda:  30%|██▉       | 2885/9708 [00:13<00:31, 216.06batch/s]

Training batches on cuda:  30%|██▉       | 2909/9708 [00:13<00:30, 221.13batch/s]

Training batches on cuda:  30%|███       | 2933/9708 [00:13<00:30, 223.42batch/s]

Training batches on cuda:  30%|███       | 2956/9708 [00:14<00:31, 215.22batch/s]

Training batches on cuda:  31%|███       | 2978/9708 [00:14<00:31, 215.19batch/s]

Training batches on cuda:  31%|███       | 3000/9708 [00:14<00:31, 214.69batch/s]

Training batches on cuda:  31%|███       | 3022/9708 [00:14<00:31, 215.65batch/s]

Training batches on cuda:  31%|███▏      | 3044/9708 [00:14<00:30, 216.09batch/s]

Training batches on cuda:  32%|███▏      | 3066/9708 [00:14<00:32, 204.32batch/s]

Training batches on cuda:  32%|███▏      | 3088/9708 [00:14<00:31, 207.47batch/s]

Training batches on cuda:  32%|███▏      | 3110/9708 [00:14<00:31, 209.79batch/s]

Training batches on cuda:  32%|███▏      | 3132/9708 [00:14<00:31, 210.95batch/s]

Training batches on cuda:  33%|███▎      | 3156/9708 [00:15<00:30, 217.09batch/s]

Training batches on cuda:  33%|███▎      | 3178/9708 [00:15<00:30, 217.21batch/s]

Training batches on cuda:  33%|███▎      | 3202/9708 [00:15<00:29, 222.43batch/s]

Training batches on cuda:  33%|███▎      | 3227/9708 [00:15<00:28, 228.16batch/s]

Training batches on cuda:  33%|███▎      | 3252/9708 [00:15<00:27, 231.68batch/s]

Training batches on cuda:  34%|███▎      | 3276/9708 [00:15<00:27, 230.97batch/s]

Training batches on cuda:  34%|███▍      | 3300/9708 [00:15<00:27, 231.41batch/s]

Training batches on cuda:  34%|███▍      | 3324/9708 [00:15<00:27, 232.76batch/s]

Training batches on cuda:  34%|███▍      | 3348/9708 [00:15<00:27, 231.61batch/s]

Training batches on cuda:  35%|███▍      | 3372/9708 [00:15<00:27, 232.77batch/s]

Training batches on cuda:  35%|███▍      | 3396/9708 [00:16<00:27, 232.41batch/s]

Training batches on cuda:  35%|███▌      | 3420/9708 [00:16<00:27, 230.50batch/s]

Training batches on cuda:  35%|███▌      | 3444/9708 [00:16<00:27, 228.04batch/s]

Training batches on cuda:  36%|███▌      | 3468/9708 [00:16<00:27, 230.38batch/s]

Training batches on cuda:  36%|███▌      | 3492/9708 [00:16<00:26, 230.35batch/s]

Training batches on cuda:  36%|███▌      | 3516/9708 [00:16<00:26, 231.01batch/s]

Training batches on cuda:  36%|███▋      | 3540/9708 [00:16<00:26, 233.34batch/s]

Training batches on cuda:  37%|███▋      | 3565/9708 [00:16<00:26, 235.47batch/s]

Training batches on cuda:  37%|███▋      | 3589/9708 [00:16<00:26, 235.03batch/s]

Training batches on cuda:  37%|███▋      | 3613/9708 [00:16<00:25, 235.85batch/s]

Training batches on cuda:  37%|███▋      | 3637/9708 [00:17<00:25, 236.85batch/s]

Training batches on cuda:  38%|███▊      | 3661/9708 [00:17<00:25, 236.78batch/s]

Training batches on cuda:  38%|███▊      | 3685/9708 [00:17<00:26, 229.45batch/s]

Training batches on cuda:  38%|███▊      | 3708/9708 [00:17<00:26, 224.60batch/s]

Training batches on cuda:  38%|███▊      | 3731/9708 [00:17<00:27, 216.74batch/s]

Training batches on cuda:  39%|███▊      | 3755/9708 [00:17<00:26, 222.88batch/s]

Training batches on cuda:  39%|███▉      | 3780/9708 [00:17<00:25, 228.08batch/s]

Training batches on cuda:  39%|███▉      | 3804/9708 [00:17<00:25, 229.54batch/s]

Training batches on cuda:  39%|███▉      | 3829/9708 [00:17<00:25, 234.57batch/s]

Training batches on cuda:  40%|███▉      | 3854/9708 [00:18<00:24, 238.62batch/s]

Training batches on cuda:  40%|███▉      | 3878/9708 [00:18<00:25, 232.23batch/s]

Training batches on cuda:  40%|████      | 3902/9708 [00:18<00:25, 228.20batch/s]

Training batches on cuda:  40%|████      | 3925/9708 [00:18<00:25, 228.28batch/s]

Training batches on cuda:  41%|████      | 3949/9708 [00:18<00:24, 230.82batch/s]

Training batches on cuda:  41%|████      | 3973/9708 [00:18<00:24, 230.32batch/s]

Training batches on cuda:  41%|████      | 3997/9708 [00:18<00:25, 222.22batch/s]

Training batches on cuda:  41%|████▏     | 4020/9708 [00:18<00:26, 218.72batch/s]

Training batches on cuda:  42%|████▏     | 4042/9708 [00:18<00:26, 213.73batch/s]

Training batches on cuda:  42%|████▏     | 4064/9708 [00:19<00:26, 211.72batch/s]

Training batches on cuda:  42%|████▏     | 4086/9708 [00:19<00:26, 211.47batch/s]

Training batches on cuda:  42%|████▏     | 4108/9708 [00:19<00:26, 211.63batch/s]

Training batches on cuda:  43%|████▎     | 4130/9708 [00:19<00:26, 211.30batch/s]

Training batches on cuda:  43%|████▎     | 4152/9708 [00:19<00:26, 210.19batch/s]

Training batches on cuda:  43%|████▎     | 4174/9708 [00:19<00:27, 203.04batch/s]

Training batches on cuda:  43%|████▎     | 4195/9708 [00:19<00:30, 179.27batch/s]

Training batches on cuda:  43%|████▎     | 4216/9708 [00:19<00:29, 185.66batch/s]

Training batches on cuda:  44%|████▎     | 4238/9708 [00:19<00:28, 193.82batch/s]

Training batches on cuda:  44%|████▍     | 4259/9708 [00:19<00:27, 196.06batch/s]

Training batches on cuda:  44%|████▍     | 4279/9708 [00:20<00:28, 192.58batch/s]

Training batches on cuda:  44%|████▍     | 4299/9708 [00:20<00:27, 193.23batch/s]

Training batches on cuda:  44%|████▍     | 4319/9708 [00:20<00:28, 190.53batch/s]

Training batches on cuda:  45%|████▍     | 4339/9708 [00:20<00:28, 189.26batch/s]

Training batches on cuda:  45%|████▍     | 4361/9708 [00:20<00:27, 195.49batch/s]

Training batches on cuda:  45%|████▌     | 4383/9708 [00:20<00:26, 200.80batch/s]

Training batches on cuda:  45%|████▌     | 4405/9708 [00:20<00:25, 204.67batch/s]

Training batches on cuda:  46%|████▌     | 4427/9708 [00:20<00:25, 207.88batch/s]

Training batches on cuda:  46%|████▌     | 4449/9708 [00:20<00:25, 208.89batch/s]

Training batches on cuda:  46%|████▌     | 4470/9708 [00:21<00:25, 203.56batch/s]

Training batches on cuda:  46%|████▋     | 4491/9708 [00:21<00:25, 202.56batch/s]

Training batches on cuda:  46%|████▋     | 4512/9708 [00:21<00:26, 197.41batch/s]

Training batches on cuda:  47%|████▋     | 4533/9708 [00:21<00:26, 198.83batch/s]

Training batches on cuda:  47%|████▋     | 4553/9708 [00:21<00:25, 199.10batch/s]

Training batches on cuda:  47%|████▋     | 4574/9708 [00:21<00:25, 199.92batch/s]

Training batches on cuda:  47%|████▋     | 4595/9708 [00:21<00:25, 199.73batch/s]

Training batches on cuda:  48%|████▊     | 4616/9708 [00:21<00:25, 200.43batch/s]

Training batches on cuda:  48%|████▊     | 4637/9708 [00:21<00:25, 199.82batch/s]

Training batches on cuda:  48%|████▊     | 4658/9708 [00:21<00:25, 200.32batch/s]

Training batches on cuda:  48%|████▊     | 4680/9708 [00:22<00:24, 204.26batch/s]

Training batches on cuda:  48%|████▊     | 4702/9708 [00:22<00:24, 207.22batch/s]

Training batches on cuda:  49%|████▊     | 4724/9708 [00:22<00:23, 209.35batch/s]

Training batches on cuda:  49%|████▉     | 4747/9708 [00:22<00:23, 213.55batch/s]

Training batches on cuda:  49%|████▉     | 4769/9708 [00:22<00:23, 214.31batch/s]

Training batches on cuda:  49%|████▉     | 4791/9708 [00:22<00:22, 214.46batch/s]

Training batches on cuda:  50%|████▉     | 4813/9708 [00:22<00:23, 210.85batch/s]

Training batches on cuda:  50%|████▉     | 4835/9708 [00:22<00:23, 206.31batch/s]

Training batches on cuda:  50%|█████     | 4856/9708 [00:22<00:23, 206.18batch/s]

Training batches on cuda:  50%|█████     | 4878/9708 [00:23<00:23, 208.49batch/s]

Training batches on cuda:  50%|█████     | 4899/9708 [00:23<00:23, 208.35batch/s]

Training batches on cuda:  51%|█████     | 4920/9708 [00:23<00:23, 203.23batch/s]

Training batches on cuda:  51%|█████     | 4942/9708 [00:23<00:23, 206.58batch/s]

Training batches on cuda:  51%|█████     | 4965/9708 [00:23<00:22, 212.96batch/s]

Training batches on cuda:  51%|█████▏    | 4987/9708 [00:23<00:22, 209.59batch/s]

Training batches on cuda:  52%|█████▏    | 5009/9708 [00:23<00:22, 211.00batch/s]

Training batches on cuda:  52%|█████▏    | 5033/9708 [00:23<00:21, 217.57batch/s]

Training batches on cuda:  52%|█████▏    | 5055/9708 [00:23<00:21, 217.44batch/s]

Training batches on cuda:  52%|█████▏    | 5077/9708 [00:23<00:23, 198.32batch/s]

Training batches on cuda:  53%|█████▎    | 5099/9708 [00:24<00:22, 203.53batch/s]

Training batches on cuda:  53%|█████▎    | 5121/9708 [00:24<00:22, 208.19batch/s]

Training batches on cuda:  53%|█████▎    | 5143/9708 [00:24<00:22, 204.54batch/s]

Training batches on cuda:  53%|█████▎    | 5164/9708 [00:24<00:22, 199.89batch/s]

Training batches on cuda:  53%|█████▎    | 5186/9708 [00:24<00:22, 204.10batch/s]

Training batches on cuda:  54%|█████▎    | 5207/9708 [00:24<00:21, 205.07batch/s]

Training batches on cuda:  54%|█████▍    | 5228/9708 [00:24<00:22, 197.70batch/s]

Training batches on cuda:  54%|█████▍    | 5248/9708 [00:24<00:22, 196.30batch/s]

Training batches on cuda:  54%|█████▍    | 5269/9708 [00:24<00:22, 198.69batch/s]

Training batches on cuda:  55%|█████▍    | 5291/9708 [00:25<00:21, 204.53batch/s]

Training batches on cuda:  55%|█████▍    | 5312/9708 [00:25<00:21, 206.05batch/s]

Training batches on cuda:  55%|█████▍    | 5334/9708 [00:25<00:21, 207.87batch/s]

Training batches on cuda:  55%|█████▌    | 5356/9708 [00:25<00:20, 210.64batch/s]

Training batches on cuda:  55%|█████▌    | 5378/9708 [00:25<00:20, 209.47batch/s]

Training batches on cuda:  56%|█████▌    | 5400/9708 [00:25<00:20, 209.91batch/s]

Training batches on cuda:  56%|█████▌    | 5422/9708 [00:25<00:20, 205.93batch/s]

Training batches on cuda:  56%|█████▌    | 5443/9708 [00:25<00:20, 205.87batch/s]

Training batches on cuda:  56%|█████▋    | 5465/9708 [00:25<00:20, 207.89batch/s]

Training batches on cuda:  57%|█████▋    | 5487/9708 [00:25<00:20, 209.86batch/s]

Training batches on cuda:  57%|█████▋    | 5509/9708 [00:26<00:20, 208.83batch/s]

Training batches on cuda:  57%|█████▋    | 5530/9708 [00:26<00:19, 209.14batch/s]

Training batches on cuda:  57%|█████▋    | 5551/9708 [00:26<00:20, 202.80batch/s]

Training batches on cuda:  57%|█████▋    | 5572/9708 [00:26<00:20, 200.68batch/s]

Training batches on cuda:  58%|█████▊    | 5593/9708 [00:26<00:20, 200.52batch/s]

Training batches on cuda:  58%|█████▊    | 5616/9708 [00:26<00:19, 207.18batch/s]

Training batches on cuda:  58%|█████▊    | 5637/9708 [00:26<00:20, 203.25batch/s]

Training batches on cuda:  58%|█████▊    | 5658/9708 [00:26<00:20, 196.01batch/s]

Training batches on cuda:  58%|█████▊    | 5678/9708 [00:26<00:21, 190.22batch/s]

Training batches on cuda:  59%|█████▊    | 5701/9708 [00:27<00:20, 199.21batch/s]

Training batches on cuda:  59%|█████▉    | 5722/9708 [00:27<00:19, 199.45batch/s]

Training batches on cuda:  59%|█████▉    | 5745/9708 [00:27<00:19, 207.71batch/s]

Training batches on cuda:  59%|█████▉    | 5768/9708 [00:27<00:18, 212.58batch/s]

Training batches on cuda:  60%|█████▉    | 5792/9708 [00:27<00:17, 219.36batch/s]

Training batches on cuda:  60%|█████▉    | 5819/9708 [00:27<00:16, 230.52batch/s]

Training batches on cuda:  60%|██████    | 5846/9708 [00:27<00:16, 239.45batch/s]

Training batches on cuda:  60%|██████    | 5873/9708 [00:27<00:15, 245.65batch/s]

Training batches on cuda:  61%|██████    | 5898/9708 [00:27<00:15, 243.34batch/s]

Training batches on cuda:  61%|██████    | 5923/9708 [00:27<00:16, 233.70batch/s]

Training batches on cuda:  61%|██████▏   | 5947/9708 [00:28<00:16, 233.50batch/s]

Training batches on cuda:  62%|██████▏   | 5972/9708 [00:28<00:15, 236.20batch/s]

Training batches on cuda:  62%|██████▏   | 5996/9708 [00:28<00:16, 231.19batch/s]

Training batches on cuda:  62%|██████▏   | 6020/9708 [00:28<00:16, 225.07batch/s]

Training batches on cuda:  62%|██████▏   | 6043/9708 [00:28<00:16, 219.54batch/s]

Training batches on cuda:  62%|██████▏   | 6066/9708 [00:28<00:16, 220.37batch/s]

Training batches on cuda:  63%|██████▎   | 6089/9708 [00:28<00:16, 220.60batch/s]

Training batches on cuda:  63%|██████▎   | 6113/9708 [00:28<00:15, 224.79batch/s]

Training batches on cuda:  63%|██████▎   | 6136/9708 [00:28<00:15, 226.24batch/s]

Training batches on cuda:  63%|██████▎   | 6160/9708 [00:29<00:15, 229.25batch/s]

Training batches on cuda:  64%|██████▎   | 6184/9708 [00:29<00:15, 229.82batch/s]

Training batches on cuda:  64%|██████▍   | 6208/9708 [00:29<00:15, 230.72batch/s]

Training batches on cuda:  64%|██████▍   | 6232/9708 [00:29<00:14, 232.37batch/s]

Training batches on cuda:  64%|██████▍   | 6256/9708 [00:29<00:15, 228.75batch/s]

Training batches on cuda:  65%|██████▍   | 6279/9708 [00:29<00:16, 213.31batch/s]

Training batches on cuda:  65%|██████▍   | 6301/9708 [00:29<00:16, 203.15batch/s]

Training batches on cuda:  65%|██████▌   | 6322/9708 [00:29<00:18, 186.63batch/s]

Training batches on cuda:  65%|██████▌   | 6342/9708 [00:29<00:18, 186.67batch/s]

Training batches on cuda:  66%|██████▌   | 6361/9708 [00:30<00:17, 186.01batch/s]

Training batches on cuda:  66%|██████▌   | 6381/9708 [00:30<00:17, 188.77batch/s]

Training batches on cuda:  66%|██████▌   | 6401/9708 [00:30<00:17, 185.80batch/s]

Training batches on cuda:  66%|██████▌   | 6422/9708 [00:30<00:17, 192.28batch/s]

Training batches on cuda:  66%|██████▋   | 6442/9708 [00:30<00:16, 194.41batch/s]

Training batches on cuda:  67%|██████▋   | 6464/9708 [00:30<00:16, 199.50batch/s]

Training batches on cuda:  67%|██████▋   | 6485/9708 [00:30<00:15, 202.39batch/s]

Training batches on cuda:  67%|██████▋   | 6506/9708 [00:30<00:15, 204.12batch/s]

Training batches on cuda:  67%|██████▋   | 6527/9708 [00:30<00:15, 205.06batch/s]

Training batches on cuda:  67%|██████▋   | 6549/9708 [00:30<00:15, 206.62batch/s]

Training batches on cuda:  68%|██████▊   | 6572/9708 [00:31<00:14, 211.28batch/s]

Training batches on cuda:  68%|██████▊   | 6594/9708 [00:31<00:14, 209.89batch/s]

Training batches on cuda:  68%|██████▊   | 6616/9708 [00:31<00:15, 194.43batch/s]

Training batches on cuda:  68%|██████▊   | 6638/9708 [00:31<00:15, 200.00batch/s]

Training batches on cuda:  69%|██████▊   | 6660/9708 [00:31<00:14, 203.24batch/s]

Training batches on cuda:  69%|██████▉   | 6682/9708 [00:31<00:14, 206.66batch/s]

Training batches on cuda:  69%|██████▉   | 6705/9708 [00:31<00:14, 212.72batch/s]

Training batches on cuda:  69%|██████▉   | 6727/9708 [00:31<00:13, 214.14batch/s]

Training batches on cuda:  70%|██████▉   | 6749/9708 [00:31<00:14, 210.91batch/s]

Training batches on cuda:  70%|██████▉   | 6772/9708 [00:32<00:13, 214.41batch/s]

Training batches on cuda:  70%|██████▉   | 6794/9708 [00:32<00:13, 212.23batch/s]

Training batches on cuda:  70%|███████   | 6816/9708 [00:32<00:13, 206.78batch/s]

Training batches on cuda:  70%|███████   | 6837/9708 [00:32<00:13, 205.41batch/s]

Training batches on cuda:  71%|███████   | 6858/9708 [00:32<00:14, 202.82batch/s]

Training batches on cuda:  71%|███████   | 6880/9708 [00:32<00:13, 206.28batch/s]

Training batches on cuda:  71%|███████   | 6901/9708 [00:32<00:13, 206.32batch/s]

Training batches on cuda:  71%|███████▏  | 6922/9708 [00:32<00:13, 207.37batch/s]

Training batches on cuda:  72%|███████▏  | 6943/9708 [00:32<00:13, 207.67batch/s]

Training batches on cuda:  72%|███████▏  | 6964/9708 [00:32<00:13, 207.12batch/s]

Training batches on cuda:  72%|███████▏  | 6986/9708 [00:33<00:13, 208.49batch/s]

Training batches on cuda:  72%|███████▏  | 7007/9708 [00:33<00:13, 206.87batch/s]

Training batches on cuda:  72%|███████▏  | 7028/9708 [00:33<00:12, 207.58batch/s]

Training batches on cuda:  73%|███████▎  | 7049/9708 [00:33<00:12, 207.90batch/s]

Training batches on cuda:  73%|███████▎  | 7070/9708 [00:33<00:12, 208.30batch/s]

Training batches on cuda:  73%|███████▎  | 7091/9708 [00:33<00:12, 205.38batch/s]

Training batches on cuda:  73%|███████▎  | 7112/9708 [00:33<00:13, 199.63batch/s]

Training batches on cuda:  73%|███████▎  | 7133/9708 [00:33<00:13, 197.87batch/s]

Training batches on cuda:  74%|███████▎  | 7156/9708 [00:33<00:12, 205.46batch/s]

Training batches on cuda:  74%|███████▍  | 7178/9708 [00:34<00:12, 208.79batch/s]

Training batches on cuda:  74%|███████▍  | 7200/9708 [00:34<00:11, 210.58batch/s]

Training batches on cuda:  74%|███████▍  | 7223/9708 [00:34<00:11, 213.60batch/s]

Training batches on cuda:  75%|███████▍  | 7246/9708 [00:34<00:11, 217.32batch/s]

Training batches on cuda:  75%|███████▍  | 7268/9708 [00:34<00:11, 216.18batch/s]

Training batches on cuda:  75%|███████▌  | 7291/9708 [00:34<00:11, 219.25batch/s]

Training batches on cuda:  75%|███████▌  | 7314/9708 [00:34<00:10, 221.00batch/s]

Training batches on cuda:  76%|███████▌  | 7337/9708 [00:34<00:11, 214.35batch/s]

Training batches on cuda:  76%|███████▌  | 7359/9708 [00:34<00:11, 210.56batch/s]

Training batches on cuda:  76%|███████▌  | 7381/9708 [00:34<00:11, 210.59batch/s]

Training batches on cuda:  76%|███████▋  | 7403/9708 [00:35<00:10, 211.38batch/s]

Training batches on cuda:  76%|███████▋  | 7426/9708 [00:35<00:10, 216.12batch/s]

Training batches on cuda:  77%|███████▋  | 7449/9708 [00:35<00:10, 219.43batch/s]

Training batches on cuda:  77%|███████▋  | 7471/9708 [00:35<00:10, 219.54batch/s]

Training batches on cuda:  77%|███████▋  | 7494/9708 [00:35<00:10, 221.15batch/s]

Training batches on cuda:  77%|███████▋  | 7517/9708 [00:35<00:09, 221.30batch/s]

Training batches on cuda:  78%|███████▊  | 7541/9708 [00:35<00:09, 223.83batch/s]

Training batches on cuda:  78%|███████▊  | 7564/9708 [00:35<00:09, 224.68batch/s]

Training batches on cuda:  78%|███████▊  | 7587/9708 [00:35<00:09, 221.76batch/s]

Training batches on cuda:  78%|███████▊  | 7610/9708 [00:35<00:09, 216.06batch/s]

Training batches on cuda:  79%|███████▊  | 7632/9708 [00:36<00:09, 215.64batch/s]

Training batches on cuda:  79%|███████▉  | 7654/9708 [00:36<00:10, 201.11batch/s]

Training batches on cuda:  79%|███████▉  | 7675/9708 [00:36<00:10, 199.41batch/s]

Training batches on cuda:  79%|███████▉  | 7696/9708 [00:36<00:09, 201.54batch/s]

Training batches on cuda:  80%|███████▉  | 7718/9708 [00:36<00:09, 204.41batch/s]

Training batches on cuda:  80%|███████▉  | 7739/9708 [00:36<00:09, 201.02batch/s]

Training batches on cuda:  80%|███████▉  | 7760/9708 [00:36<00:10, 191.99batch/s]

Training batches on cuda:  80%|████████  | 7780/9708 [00:36<00:10, 189.69batch/s]

Training batches on cuda:  80%|████████  | 7800/9708 [00:36<00:09, 190.84batch/s]

Training batches on cuda:  81%|████████  | 7822/9708 [00:37<00:09, 196.79batch/s]

Training batches on cuda:  81%|████████  | 7842/9708 [00:37<00:09, 195.93batch/s]

Training batches on cuda:  81%|████████  | 7862/9708 [00:37<00:09, 194.14batch/s]

Training batches on cuda:  81%|████████  | 7882/9708 [00:37<00:09, 185.76batch/s]

Training batches on cuda:  81%|████████▏ | 7902/9708 [00:37<00:09, 187.88batch/s]

Training batches on cuda:  82%|████████▏ | 7923/9708 [00:37<00:09, 193.57batch/s]

Training batches on cuda:  82%|████████▏ | 7943/9708 [00:37<00:09, 189.61batch/s]

Training batches on cuda:  82%|████████▏ | 7963/9708 [00:37<00:09, 192.37batch/s]

Training batches on cuda:  82%|████████▏ | 7984/9708 [00:37<00:08, 196.51batch/s]

Training batches on cuda:  82%|████████▏ | 8004/9708 [00:38<00:08, 195.16batch/s]

Training batches on cuda:  83%|████████▎ | 8024/9708 [00:38<00:08, 195.03batch/s]

Training batches on cuda:  83%|████████▎ | 8044/9708 [00:38<00:08, 195.47batch/s]

Training batches on cuda:  83%|████████▎ | 8067/9708 [00:38<00:08, 204.66batch/s]

Training batches on cuda:  83%|████████▎ | 8089/9708 [00:38<00:07, 208.97batch/s]

Training batches on cuda:  84%|████████▎ | 8111/9708 [00:38<00:07, 210.02batch/s]

Training batches on cuda:  84%|████████▍ | 8135/9708 [00:38<00:07, 216.31batch/s]

Training batches on cuda:  84%|████████▍ | 8157/9708 [00:38<00:07, 216.04batch/s]

Training batches on cuda:  84%|████████▍ | 8179/9708 [00:38<00:07, 216.19batch/s]

Training batches on cuda:  84%|████████▍ | 8201/9708 [00:38<00:07, 215.02batch/s]

Training batches on cuda:  85%|████████▍ | 8223/9708 [00:39<00:06, 213.10batch/s]

Training batches on cuda:  85%|████████▍ | 8245/9708 [00:39<00:06, 211.97batch/s]

Training batches on cuda:  85%|████████▌ | 8267/9708 [00:39<00:06, 213.26batch/s]

Training batches on cuda:  85%|████████▌ | 8289/9708 [00:39<00:06, 212.45batch/s]

Training batches on cuda:  86%|████████▌ | 8311/9708 [00:39<00:06, 208.48batch/s]

Training batches on cuda:  86%|████████▌ | 8334/9708 [00:39<00:06, 212.20batch/s]

Training batches on cuda:  86%|████████▌ | 8357/9708 [00:39<00:06, 215.06batch/s]

Training batches on cuda:  86%|████████▋ | 8380/9708 [00:39<00:06, 218.05batch/s]

Training batches on cuda:  87%|████████▋ | 8402/9708 [00:39<00:06, 202.69batch/s]

Training batches on cuda:  87%|████████▋ | 8424/9708 [00:40<00:06, 205.07batch/s]

Training batches on cuda:  87%|████████▋ | 8445/9708 [00:40<00:06, 204.70batch/s]

Training batches on cuda:  87%|████████▋ | 8467/9708 [00:40<00:05, 207.73batch/s]

Training batches on cuda:  87%|████████▋ | 8488/9708 [00:40<00:05, 206.85batch/s]

Training batches on cuda:  88%|████████▊ | 8509/9708 [00:40<00:05, 206.43batch/s]

Training batches on cuda:  88%|████████▊ | 8530/9708 [00:40<00:05, 207.02batch/s]

Training batches on cuda:  88%|████████▊ | 8553/9708 [00:40<00:05, 210.93batch/s]

Training batches on cuda:  88%|████████▊ | 8576/9708 [00:40<00:05, 215.77batch/s]

Training batches on cuda:  89%|████████▊ | 8598/9708 [00:40<00:05, 215.76batch/s]

Training batches on cuda:  89%|████████▉ | 8621/9708 [00:40<00:04, 218.14batch/s]

Training batches on cuda:  89%|████████▉ | 8644/9708 [00:41<00:04, 220.33batch/s]

Training batches on cuda:  89%|████████▉ | 8667/9708 [00:41<00:04, 223.14batch/s]

Training batches on cuda:  90%|████████▉ | 8693/9708 [00:41<00:04, 232.20batch/s]

Training batches on cuda:  90%|████████▉ | 8720/9708 [00:41<00:04, 241.21batch/s]

Training batches on cuda:  90%|█████████ | 8747/9708 [00:41<00:03, 247.10batch/s]

Training batches on cuda:  90%|█████████ | 8772/9708 [00:41<00:03, 246.84batch/s]

Training batches on cuda:  91%|█████████ | 8797/9708 [00:41<00:03, 237.18batch/s]

Training batches on cuda:  91%|█████████ | 8821/9708 [00:41<00:03, 223.20batch/s]

Training batches on cuda:  91%|█████████ | 8844/9708 [00:41<00:03, 216.93batch/s]

Training batches on cuda:  91%|█████████▏| 8866/9708 [00:41<00:03, 216.45batch/s]

Training batches on cuda:  92%|█████████▏| 8888/9708 [00:42<00:04, 204.30batch/s]

Training batches on cuda:  92%|█████████▏| 8909/9708 [00:42<00:04, 199.21batch/s]

Training batches on cuda:  92%|█████████▏| 8932/9708 [00:42<00:03, 207.25batch/s]

Training batches on cuda:  92%|█████████▏| 8954/9708 [00:42<00:03, 208.54batch/s]

Training batches on cuda:  92%|█████████▏| 8978/9708 [00:42<00:03, 216.81batch/s]

Training batches on cuda:  93%|█████████▎| 9002/9708 [00:42<00:03, 220.83batch/s]

Training batches on cuda:  93%|█████████▎| 9027/9708 [00:42<00:02, 227.11batch/s]

Training batches on cuda:  93%|█████████▎| 9050/9708 [00:42<00:02, 227.18batch/s]

Training batches on cuda:  93%|█████████▎| 9073/9708 [00:42<00:02, 218.49batch/s]

Training batches on cuda:  94%|█████████▎| 9095/9708 [00:43<00:02, 206.00batch/s]

Training batches on cuda:  94%|█████████▍| 9116/9708 [00:43<00:02, 203.09batch/s]

Training batches on cuda:  94%|█████████▍| 9137/9708 [00:43<00:02, 197.13batch/s]

Training batches on cuda:  94%|█████████▍| 9157/9708 [00:43<00:02, 190.45batch/s]

Training batches on cuda:  95%|█████████▍| 9177/9708 [00:43<00:02, 186.67batch/s]

Training batches on cuda:  95%|█████████▍| 9197/9708 [00:43<00:02, 188.24batch/s]

Training batches on cuda:  95%|█████████▍| 9219/9708 [00:43<00:02, 195.89batch/s]

Training batches on cuda:  95%|█████████▌| 9241/9708 [00:43<00:02, 201.13batch/s]

Training batches on cuda:  95%|█████████▌| 9262/9708 [00:43<00:02, 200.09batch/s]

Training batches on cuda:  96%|█████████▌| 9283/9708 [00:44<00:02, 202.38batch/s]

Training batches on cuda:  96%|█████████▌| 9304/9708 [00:44<00:02, 197.58batch/s]

Training batches on cuda:  96%|█████████▌| 9324/9708 [00:44<00:01, 195.14batch/s]

Training batches on cuda:  96%|█████████▋| 9344/9708 [00:44<00:01, 194.34batch/s]

Training batches on cuda:  96%|█████████▋| 9364/9708 [00:44<00:01, 188.56batch/s]

Training batches on cuda:  97%|█████████▋| 9384/9708 [00:44<00:01, 189.66batch/s]

Training batches on cuda:  97%|█████████▋| 9406/9708 [00:44<00:01, 196.61batch/s]

Training batches on cuda:  97%|█████████▋| 9427/9708 [00:44<00:01, 199.37batch/s]

Training batches on cuda:  97%|█████████▋| 9448/9708 [00:44<00:01, 184.65batch/s]

Training batches on cuda:  98%|█████████▊| 9471/9708 [00:45<00:01, 194.75batch/s]

Training batches on cuda:  98%|█████████▊| 9494/9708 [00:45<00:01, 203.75batch/s]

Training batches on cuda:  98%|█████████▊| 9515/9708 [00:45<00:00, 203.81batch/s]

Training batches on cuda:  98%|█████████▊| 9536/9708 [00:45<00:00, 205.22batch/s]

Training batches on cuda:  98%|█████████▊| 9557/9708 [00:45<00:00, 204.19batch/s]

Training batches on cuda:  99%|█████████▊| 9580/9708 [00:45<00:00, 211.29batch/s]

Training batches on cuda:  99%|█████████▉| 9602/9708 [00:45<00:00, 213.41batch/s]

Training batches on cuda:  99%|█████████▉| 9624/9708 [00:45<00:00, 211.93batch/s]

Training batches on cuda:  99%|█████████▉| 9646/9708 [00:45<00:00, 211.88batch/s]

Training batches on cuda: 100%|█████████▉| 9668/9708 [00:45<00:00, 211.19batch/s]

Training batches on cuda: 100%|█████████▉| 9690/9708 [00:46<00:00, 210.82batch/s]

                                                                                 
Training epochs on cuda:  90%|█████████ | 9/10 [07:41<00:46, 46.03s/epoch, loss=0.096, prev_loss=0.096]2021-09-16:17:42:54,107 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:42:55,97 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:43:10,41 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:17:43:10,195 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [07:57<00:00, 50.93s/epoch, loss=0.096, prev_loss=0.096]
Training epochs on cuda: 100%|██████████| 10/10 [07:57<00:00, 47.77s/epoch, loss=0.096, prev_loss=0.096]
2021-09-16:17:43:10,196 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp70n966h3'
2021-09-16:17:43:10,238 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp70n966h3' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.25ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.36ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.50ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:43:25,19 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 17:43:25,024] Trial 10 finished with value: -0.009810833766599858 and parameters: {'loss.margin': 3, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.2970866543722897, 'optimizer.lr': 0.025147354117093117, 'negative_sampler.num_negs_per_pos': 46, 'training.batch_size': 32}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:43:25,91 WARNING  [api.py:823] No random seed is specified. Setting to 2700869626.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/217 [00:00<00:03, 58.54batch/s]

Training batches on cuda:   7%|▋         | 16/217 [00:00<00:03, 65.40batch/s]

Training batches on cuda:  12%|█▏        | 26/217 [00:00<00:02, 72.42batch/s]

Training batches on cuda:  17%|█▋        | 37/217 [00:00<00:02, 79.25batch/s]

Training batches on cuda:  22%|██▏       | 47/217 [00:00<00:02, 83.01batch/s]

Training batches on cuda:  26%|██▌       | 56/217 [00:00<00:01, 84.20batch/s]

Training batches on cuda:  31%|███▏      | 68/217 [00:00<00:01, 91.45batch/s]

Training batches on cuda:  36%|███▋      | 79/217 [00:00<00:01, 95.93batch/s]

Training batches on cuda:  41%|████      | 89/217 [00:00<00:01, 96.32batch/s]

Training batches on cuda:  46%|████▌     | 99/217 [00:01<00:01, 94.45batch/s]

Training batches on cuda:  50%|█████     | 109/217 [00:01<00:01, 92.76batch/s]

Training batches on cuda:  55%|█████▌    | 120/217 [00:01<00:01, 95.77batch/s]

Training batches on cuda:  61%|██████    | 132/217 [00:01<00:00, 101.24batch/s]

Training batches on cuda:  66%|██████▋   | 144/217 [00:01<00:00, 105.53batch/s]

Training batches on cuda:  72%|███████▏  | 156/217 [00:01<00:00, 107.66batch/s]

Training batches on cuda:  77%|███████▋  | 168/217 [00:01<00:00, 109.38batch/s]

Training batches on cuda:  83%|████████▎ | 180/217 [00:01<00:00, 109.98batch/s]

Training batches on cuda:  88%|████████▊ | 192/217 [00:01<00:00, 110.38batch/s]

Training batches on cuda:  94%|█████████▍| 204/217 [00:02<00:00, 110.08batch/s]

Training batches on cuda: 100%|█████████▉| 216/217 [00:02<00:00, 109.17batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.00177, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:20,  2.30s/epoch, loss=0.00177, prev_loss=nan]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 9/217 [00:00<00:02, 83.38batch/s]

Training batches on cuda:   9%|▉         | 20/217 [00:00<00:02, 89.17batch/s]

Training batches on cuda:  14%|█▍        | 30/217 [00:00<00:02, 90.10batch/s]

Training batches on cuda:  18%|█▊        | 39/217 [00:00<00:02, 87.74batch/s]

Training batches on cuda:  22%|██▏       | 48/217 [00:00<00:01, 86.69batch/s]

Training batches on cuda:  27%|██▋       | 58/217 [00:00<00:01, 89.15batch/s]

Training batches on cuda:  31%|███▏      | 68/217 [00:00<00:01, 91.36batch/s]

Training batches on cuda:  35%|███▌      | 77/217 [00:00<00:01, 81.93batch/s]

Training batches on cuda:  40%|████      | 87/217 [00:00<00:01, 85.91batch/s]

Training batches on cuda:  45%|████▍     | 97/217 [00:01<00:01, 88.70batch/s]

Training batches on cuda:  49%|████▉     | 107/217 [00:01<00:01, 91.42batch/s]

Training batches on cuda:  54%|█████▍    | 117/217 [00:01<00:01, 93.48batch/s]

Training batches on cuda:  59%|█████▊    | 127/217 [00:01<00:00, 94.51batch/s]

Training batches on cuda:  63%|██████▎   | 137/217 [00:01<00:00, 95.09batch/s]

Training batches on cuda:  68%|██████▊   | 147/217 [00:01<00:00, 95.45batch/s]

Training batches on cuda:  73%|███████▎  | 158/217 [00:01<00:00, 97.08batch/s]

Training batches on cuda:  77%|███████▋  | 168/217 [00:01<00:00, 97.15batch/s]

Training batches on cuda:  82%|████████▏ | 178/217 [00:01<00:00, 97.33batch/s]

Training batches on cuda:  87%|████████▋ | 188/217 [00:02<00:00, 97.26batch/s]

Training batches on cuda:  91%|█████████ | 198/217 [00:02<00:00, 97.20batch/s]

Training batches on cuda:  96%|█████████▌| 208/217 [00:02<00:00, 95.98batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:20,  2.30s/epoch, loss=0.00149, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:18,  2.36s/epoch, loss=0.00149, prev_loss=nan]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 9/217 [00:00<00:02, 83.79batch/s]

Training batches on cuda:  10%|▉         | 21/217 [00:00<00:02, 90.61batch/s]

Training batches on cuda:  15%|█▌        | 33/217 [00:00<00:01, 96.79batch/s]

Training batches on cuda:  20%|█▉        | 43/217 [00:00<00:01, 95.18batch/s]

Training batches on cuda:  25%|██▌       | 55/217 [00:00<00:01, 100.39batch/s]

Training batches on cuda:  31%|███       | 67/217 [00:00<00:01, 104.11batch/s]

Training batches on cuda:  36%|███▋      | 79/217 [00:00<00:01, 106.63batch/s]

Training batches on cuda:  42%|████▏     | 91/217 [00:00<00:01, 109.33batch/s]

Training batches on cuda:  47%|████▋     | 103/217 [00:00<00:01, 110.81batch/s]

Training batches on cuda:  53%|█████▎    | 115/217 [00:01<00:00, 110.88batch/s]

Training batches on cuda:  59%|█████▊    | 127/217 [00:01<00:00, 111.69batch/s]

Training batches on cuda:  64%|██████▍   | 139/217 [00:01<00:00, 112.49batch/s]

Training batches on cuda:  70%|██████▉   | 151/217 [00:01<00:00, 111.87batch/s]

Training batches on cuda:  75%|███████▌  | 163/217 [00:01<00:00, 110.92batch/s]

Training batches on cuda:  81%|████████  | 175/217 [00:01<00:00, 109.90batch/s]

Training batches on cuda:  86%|████████▌ | 186/217 [00:01<00:00, 108.67batch/s]

Training batches on cuda:  91%|█████████ | 197/217 [00:01<00:00, 108.43batch/s]

Training batches on cuda:  96%|█████████▌| 208/217 [00:01<00:00, 107.63batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:18,  2.36s/epoch, loss=0.00149, prev_loss=0.00149]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:16,  2.31s/epoch, loss=0.00149, prev_loss=0.00149]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 8/217 [00:00<00:02, 76.60batch/s]

Training batches on cuda:   9%|▉         | 19/217 [00:00<00:02, 83.41batch/s]

Training batches on cuda:  14%|█▍        | 31/217 [00:00<00:02, 90.72batch/s]

Training batches on cuda:  19%|█▉        | 42/217 [00:00<00:01, 95.16batch/s]

Training batches on cuda:  25%|██▍       | 54/217 [00:00<00:01, 99.58batch/s]

Training batches on cuda:  30%|██▉       | 65/217 [00:00<00:01, 101.66batch/s]

Training batches on cuda:  35%|███▌      | 77/217 [00:00<00:01, 104.81batch/s]

Training batches on cuda:  41%|████      | 89/217 [00:00<00:01, 106.29batch/s]

Training batches on cuda:  46%|████▌     | 100/217 [00:00<00:01, 106.89batch/s]

Training batches on cuda:  51%|█████     | 111/217 [00:01<00:01, 103.95batch/s]

Training batches on cuda:  56%|█████▌    | 122/217 [00:01<00:00, 95.66batch/s] 

Training batches on cuda:  61%|██████    | 132/217 [00:01<00:01, 81.39batch/s]

Training batches on cuda:  65%|██████▌   | 142/217 [00:01<00:00, 84.52batch/s]

Training batches on cuda:  70%|███████   | 152/217 [00:01<00:00, 88.00batch/s]

Training batches on cuda:  75%|███████▍  | 162/217 [00:01<00:00, 90.35batch/s]

Training batches on cuda:  79%|███████▉  | 172/217 [00:01<00:00, 91.32batch/s]

Training batches on cuda:  84%|████████▍ | 183/217 [00:01<00:00, 95.15batch/s]

Training batches on cuda:  89%|████████▉ | 194/217 [00:01<00:00, 96.93batch/s]

Training batches on cuda:  94%|█████████▍| 204/217 [00:02<00:00, 96.27batch/s]

Training batches on cuda:  99%|█████████▊| 214/217 [00:02<00:00, 95.93batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:16,  2.31s/epoch, loss=0.00149, prev_loss=0.00149]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:14,  2.34s/epoch, loss=0.00149, prev_loss=0.00149]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/217 [00:00<00:03, 66.25batch/s]

Training batches on cuda:   9%|▉         | 19/217 [00:00<00:02, 75.26batch/s]

Training batches on cuda:  14%|█▍        | 30/217 [00:00<00:02, 82.56batch/s]

Training batches on cuda:  19%|█▉        | 42/217 [00:00<00:01, 89.11batch/s]

Training batches on cuda:  24%|██▍       | 53/217 [00:00<00:01, 93.31batch/s]

Training batches on cuda:  29%|██▉       | 63/217 [00:00<00:01, 94.47batch/s]

Training batches on cuda:  34%|███▎      | 73/217 [00:00<00:01, 95.28batch/s]

Training batches on cuda:  39%|███▊      | 84/217 [00:00<00:01, 96.46batch/s]

Training batches on cuda:  43%|████▎     | 94/217 [00:00<00:01, 94.80batch/s]

Training batches on cuda:  48%|████▊     | 104/217 [00:01<00:01, 94.49batch/s]

Training batches on cuda:  53%|█████▎    | 115/217 [00:01<00:01, 97.69batch/s]

Training batches on cuda:  58%|█████▊    | 125/217 [00:01<00:00, 97.09batch/s]

Training batches on cuda:  62%|██████▏   | 135/217 [00:01<00:00, 96.57batch/s]

Training batches on cuda:  67%|██████▋   | 145/217 [00:01<00:00, 96.98batch/s]

Training batches on cuda:  71%|███████▏  | 155/217 [00:01<00:00, 96.32batch/s]

Training batches on cuda:  76%|███████▋  | 166/217 [00:01<00:00, 97.64batch/s]

Training batches on cuda:  82%|████████▏ | 177/217 [00:01<00:00, 98.96batch/s]

Training batches on cuda:  87%|████████▋ | 188/217 [00:01<00:00, 99.87batch/s]

Training batches on cuda:  92%|█████████▏| 199/217 [00:02<00:00, 100.58batch/s]

Training batches on cuda:  97%|█████████▋| 210/217 [00:02<00:00, 99.67batch/s] 

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:14,  2.34s/epoch, loss=0.00149, prev_loss=0.00149]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.35s/epoch, loss=0.00149, prev_loss=0.00149]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/217 [00:00<00:03, 52.93batch/s]

Training batches on cuda:   7%|▋         | 15/217 [00:00<00:03, 60.09batch/s]

Training batches on cuda:  12%|█▏        | 25/217 [00:00<00:02, 67.40batch/s]

Training batches on cuda:  16%|█▌        | 35/217 [00:00<00:02, 73.06batch/s]

Training batches on cuda:  21%|██        | 45/217 [00:00<00:02, 77.50batch/s]

Training batches on cuda:  26%|██▌       | 56/217 [00:00<00:01, 84.61batch/s]

Training batches on cuda:  31%|███       | 67/217 [00:00<00:01, 90.25batch/s]

Training batches on cuda:  36%|███▌      | 78/217 [00:00<00:01, 92.49batch/s]

Training batches on cuda:  41%|████      | 88/217 [00:00<00:01, 93.23batch/s]

Training batches on cuda:  45%|████▌     | 98/217 [00:01<00:01, 91.51batch/s]

Training batches on cuda:  50%|████▉     | 108/217 [00:01<00:01, 89.42batch/s]

Training batches on cuda:  54%|█████▍    | 117/217 [00:01<00:01, 88.27batch/s]

Training batches on cuda:  59%|█████▊    | 127/217 [00:01<00:01, 89.94batch/s]

Training batches on cuda:  63%|██████▎   | 137/217 [00:01<00:00, 90.48batch/s]

Training batches on cuda:  68%|██████▊   | 147/217 [00:01<00:00, 91.78batch/s]

Training batches on cuda:  72%|███████▏  | 157/217 [00:01<00:00, 90.30batch/s]

Training batches on cuda:  77%|███████▋  | 167/217 [00:01<00:00, 89.75batch/s]

Training batches on cuda:  81%|████████  | 176/217 [00:01<00:00, 89.13batch/s]

Training batches on cuda:  85%|████████▌ | 185/217 [00:02<00:00, 89.12batch/s]

Training batches on cuda:  90%|████████▉ | 195/217 [00:02<00:00, 89.85batch/s]

Training batches on cuda:  94%|█████████▍| 205/217 [00:02<00:00, 91.00batch/s]

Training batches on cuda:  99%|█████████▉| 215/217 [00:02<00:00, 89.00batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:11,  2.35s/epoch, loss=0.00149, prev_loss=0.00149]
Training epochs on cuda:  60%|██████    | 6/10 [00:14<00:09,  2.43s/epoch, loss=0.00149, prev_loss=0.00149]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/217 [00:00<00:03, 68.71batch/s]

Training batches on cuda:   8%|▊         | 17/217 [00:00<00:02, 74.00batch/s]

Training batches on cuda:  12%|█▏        | 26/217 [00:00<00:02, 76.27batch/s]

Training batches on cuda:  16%|█▌        | 34/217 [00:00<00:02, 77.15batch/s]

Training batches on cuda:  20%|█▉        | 43/217 [00:00<00:02, 79.04batch/s]

Training batches on cuda:  25%|██▍       | 54/217 [00:00<00:01, 84.86batch/s]

Training batches on cuda:  29%|██▉       | 63/217 [00:00<00:01, 85.54batch/s]

Training batches on cuda:  33%|███▎      | 72/217 [00:00<00:01, 82.20batch/s]

Training batches on cuda:  38%|███▊      | 83/217 [00:00<00:01, 87.05batch/s]

Training batches on cuda:  43%|████▎     | 94/217 [00:01<00:01, 91.28batch/s]

Training batches on cuda:  48%|████▊     | 104/217 [00:01<00:01, 93.03batch/s]

Training batches on cuda:  53%|█████▎    | 114/217 [00:01<00:01, 77.83batch/s]

Training batches on cuda:  57%|█████▋    | 123/217 [00:01<00:01, 79.85batch/s]

Training batches on cuda:  61%|██████    | 132/217 [00:01<00:01, 78.17batch/s]

Training batches on cuda:  65%|██████▍   | 141/217 [00:01<00:00, 79.71batch/s]

Training batches on cuda:  69%|██████▉   | 150/217 [00:01<00:00, 78.54batch/s]

Training batches on cuda:  73%|███████▎  | 159/217 [00:01<00:00, 80.46batch/s]

Training batches on cuda:  78%|███████▊  | 169/217 [00:02<00:00, 83.59batch/s]

Training batches on cuda:  82%|████████▏ | 178/217 [00:02<00:00, 84.84batch/s]

Training batches on cuda:  86%|████████▌ | 187/217 [00:02<00:00, 85.50batch/s]

Training batches on cuda:  90%|█████████ | 196/217 [00:02<00:00, 83.88batch/s]

Training batches on cuda:  94%|█████████▍| 205/217 [00:02<00:00, 80.34batch/s]

Training batches on cuda:  99%|█████████▊| 214/217 [00:02<00:00, 78.57batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:09,  2.43s/epoch, loss=0.00149, prev_loss=0.00149]
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:07,  2.55s/epoch, loss=0.00149, prev_loss=0.00149]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/217 [00:00<00:03, 55.76batch/s]

Training batches on cuda:   8%|▊         | 17/217 [00:00<00:03, 64.56batch/s]

Training batches on cuda:  12%|█▏        | 26/217 [00:00<00:02, 68.49batch/s]

Training batches on cuda:  15%|█▌        | 33/217 [00:00<00:02, 67.75batch/s]

Training batches on cuda:  19%|█▉        | 41/217 [00:00<00:02, 69.61batch/s]

Training batches on cuda:  23%|██▎       | 50/217 [00:00<00:02, 74.18batch/s]

Training batches on cuda:  27%|██▋       | 59/217 [00:00<00:02, 77.11batch/s]

Training batches on cuda:  31%|███▏      | 68/217 [00:00<00:01, 78.83batch/s]

Training batches on cuda:  36%|███▌      | 78/217 [00:00<00:01, 79.96batch/s]

Training batches on cuda:  40%|███▉      | 86/217 [00:01<00:01, 69.85batch/s]

Training batches on cuda:  43%|████▎     | 94/217 [00:01<00:01, 72.53batch/s]

Training batches on cuda:  47%|████▋     | 103/217 [00:01<00:01, 75.65batch/s]

Training batches on cuda:  51%|█████     | 111/217 [00:01<00:01, 75.34batch/s]

Training batches on cuda:  55%|█████▍    | 119/217 [00:01<00:01, 72.80batch/s]

Training batches on cuda:  59%|█████▉    | 128/217 [00:01<00:01, 75.16batch/s]

Training batches on cuda:  63%|██████▎   | 136/217 [00:01<00:01, 75.82batch/s]

Training batches on cuda:  67%|██████▋   | 146/217 [00:01<00:00, 81.73batch/s]

Training batches on cuda:  72%|███████▏  | 156/217 [00:01<00:00, 84.24batch/s]

Training batches on cuda:  76%|███████▋  | 166/217 [00:02<00:00, 86.62batch/s]

Training batches on cuda:  81%|████████  | 176/217 [00:02<00:00, 88.97batch/s]

Training batches on cuda:  85%|████████▌ | 185/217 [00:02<00:00, 87.12batch/s]

Training batches on cuda:  89%|████████▉ | 194/217 [00:02<00:00, 85.18batch/s]

Training batches on cuda:  94%|█████████▍| 204/217 [00:02<00:00, 87.18batch/s]

Training batches on cuda:  99%|█████████▉| 215/217 [00:02<00:00, 90.76batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:07,  2.55s/epoch, loss=0.00149, prev_loss=0.00149]
Training epochs on cuda:  80%|████████  | 8/10 [00:20<00:05,  2.64s/epoch, loss=0.00149, prev_loss=0.00149]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/217 [00:00<00:03, 54.31batch/s]

Training batches on cuda:   7%|▋         | 15/217 [00:00<00:03, 60.53batch/s]

Training batches on cuda:  11%|█         | 24/217 [00:00<00:02, 66.98batch/s]

Training batches on cuda:  16%|█▌        | 35/217 [00:00<00:02, 74.43batch/s]

Training batches on cuda:  21%|██        | 46/217 [00:00<00:02, 81.00batch/s]

Training batches on cuda:  26%|██▌       | 56/217 [00:00<00:01, 84.42batch/s]

Training batches on cuda:  30%|███       | 66/217 [00:00<00:01, 86.87batch/s]

Training batches on cuda:  35%|███▍      | 75/217 [00:00<00:01, 78.15batch/s]

Training batches on cuda:  39%|███▉      | 85/217 [00:00<00:01, 83.40batch/s]

Training batches on cuda:  43%|████▎     | 94/217 [00:01<00:01, 84.38batch/s]

Training batches on cuda:  48%|████▊     | 104/217 [00:01<00:01, 87.76batch/s]

Training batches on cuda:  53%|█████▎    | 114/217 [00:01<00:01, 88.87batch/s]

Training batches on cuda:  58%|█████▊    | 125/217 [00:01<00:00, 93.99batch/s]

Training batches on cuda:  63%|██████▎   | 136/217 [00:01<00:00, 96.77batch/s]

Training batches on cuda:  67%|██████▋   | 146/217 [00:01<00:00, 94.71batch/s]

Training batches on cuda:  72%|███████▏  | 156/217 [00:01<00:00, 96.04batch/s]

Training batches on cuda:  77%|███████▋  | 168/217 [00:01<00:00, 100.34batch/s]

Training batches on cuda:  82%|████████▏ | 179/217 [00:01<00:00, 101.61batch/s]

Training batches on cuda:  88%|████████▊ | 190/217 [00:02<00:00, 98.98batch/s] 

Training batches on cuda:  92%|█████████▏| 200/217 [00:02<00:00, 96.95batch/s]

Training batches on cuda:  97%|█████████▋| 210/217 [00:02<00:00, 93.94batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:22<00:05,  2.64s/epoch, loss=0.00149, prev_loss=0.00149]
Training epochs on cuda:  90%|█████████ | 9/10 [00:22<00:02,  2.64s/epoch, loss=0.00149, prev_loss=0.00149]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/217 [00:00<00:03, 58.62batch/s]

Training batches on cuda:   8%|▊         | 17/217 [00:00<00:02, 67.93batch/s]

Training batches on cuda:  13%|█▎        | 28/217 [00:00<00:02, 76.52batch/s]

Training batches on cuda:  18%|█▊        | 38/217 [00:00<00:02, 82.11batch/s]

Training batches on cuda:  23%|██▎       | 50/217 [00:00<00:01, 89.25batch/s]

Training batches on cuda:  28%|██▊       | 61/217 [00:00<00:01, 94.40batch/s]

Training batches on cuda:  33%|███▎      | 72/217 [00:00<00:01, 98.40batch/s]

Training batches on cuda:  38%|███▊      | 83/217 [00:00<00:01, 100.67batch/s]

Training batches on cuda:  44%|████▍     | 95/217 [00:00<00:01, 103.56batch/s]

Training batches on cuda:  49%|████▉     | 107/217 [00:01<00:01, 105.95batch/s]

Training batches on cuda:  55%|█████▍    | 119/217 [00:01<00:00, 107.34batch/s]

Training batches on cuda:  60%|█████▉    | 130/217 [00:01<00:00, 107.51batch/s]

Training batches on cuda:  65%|██████▍   | 141/217 [00:01<00:00, 103.02batch/s]

Training batches on cuda:  70%|███████   | 152/217 [00:01<00:00, 100.71batch/s]

Training batches on cuda:  75%|███████▌  | 163/217 [00:01<00:00, 99.49batch/s] 

Training batches on cuda:  80%|███████▉  | 173/217 [00:01<00:00, 98.87batch/s]

Training batches on cuda:  84%|████████▍ | 183/217 [00:01<00:00, 97.39batch/s]

Training batches on cuda:  89%|████████▉ | 193/217 [00:01<00:00, 96.72batch/s]

Training batches on cuda:  94%|█████████▎| 203/217 [00:02<00:00, 97.35batch/s]

Training batches on cuda:  99%|█████████▊| 214/217 [00:02<00:00, 100.59batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:25<00:02,  2.64s/epoch, loss=0.00149, prev_loss=0.00149]2021-09-16:17:43:50,587 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:43:51,583 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:44:06,485 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
2021-09-16:17:44:06,635 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00,  7.36s/epoch, loss=0.00149, prev_loss=0.00149]
Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00,  4.11s/epoch, loss=0.00149, prev_loss=0.00149]
2021-09-16:17:44:06,635 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpj6xa5enm'
2021-09-16:17:44:06,693 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj6xa5enm' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.32ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.42ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.48ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.53ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:08, 2.57ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.59ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.05ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:12<00:03, 2.20ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:03, 2.32ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.41ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.47ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.53ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.56ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.59ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.61ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.59ktriple/s]
2021-09-16:17:44:21,686 INFO     [evaluator.py:657] Evaluation took 14.98s seconds
[I 2021-09-16 17:44:21,691] Trial 11 finished with value: 0.003729993210033089 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.6204069436414972, 'optimizer.lr': 0.0013005278307563414, 'negative_sampler.num_negs_per_pos': 3, 'training.batch_size': 1432}. Best is trial 1 with value: 0.29090778411927365.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:44:21,710 WARNING  [api.py:823] No random seed is specified. Setting to 2407467682.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 9/234 [00:00<00:02, 86.84batch/s]

Training batches on cuda:   9%|▊         | 20/234 [00:00<00:02, 92.44batch/s]

Training batches on cuda:  14%|█▎        | 32/234 [00:00<00:02, 97.73batch/s]

Training batches on cuda:  19%|█▉        | 44/234 [00:00<00:01, 101.30batch/s]

Training batches on cuda:  24%|██▎       | 55/234 [00:00<00:01, 102.80batch/s]

Training batches on cuda:  28%|██▊       | 66/234 [00:00<00:01, 102.46batch/s]

Training batches on cuda:  33%|███▎      | 78/234 [00:00<00:01, 105.00batch/s]

Training batches on cuda:  38%|███▊      | 90/234 [00:00<00:01, 106.48batch/s]

Training batches on cuda:  44%|████▎     | 102/234 [00:00<00:01, 107.92batch/s]

Training batches on cuda:  49%|████▊     | 114/234 [00:01<00:01, 109.29batch/s]

Training batches on cuda:  54%|█████▍    | 126/234 [00:01<00:00, 109.98batch/s]

Training batches on cuda:  59%|█████▊    | 137/234 [00:01<00:00, 109.71batch/s]

Training batches on cuda:  63%|██████▎   | 148/234 [00:01<00:00, 104.89batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:01<00:00, 106.27batch/s]

Training batches on cuda:  73%|███████▎  | 170/234 [00:01<00:00, 106.01batch/s]

Training batches on cuda:  77%|███████▋  | 181/234 [00:01<00:00, 102.48batch/s]

Training batches on cuda:  82%|████████▏ | 192/234 [00:01<00:00, 89.18batch/s] 

Training batches on cuda:  86%|████████▋ | 202/234 [00:01<00:00, 88.68batch/s]

Training batches on cuda:  91%|█████████ | 212/234 [00:02<00:00, 91.70batch/s]

Training batches on cuda:  95%|█████████▍| 222/234 [00:02<00:00, 83.79batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:02<00:00, 83.48batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.0016, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:22,  2.55s/epoch, loss=0.0016, prev_loss=nan]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:03, 65.39batch/s]

Training batches on cuda:   7%|▋         | 16/234 [00:00<00:03, 70.63batch/s]

Training batches on cuda:  11%|█         | 26/234 [00:00<00:02, 76.60batch/s]

Training batches on cuda:  15%|█▌        | 36/234 [00:00<00:02, 80.70batch/s]

Training batches on cuda:  20%|█▉        | 46/234 [00:00<00:02, 84.90batch/s]

Training batches on cuda:  24%|██▍       | 56/234 [00:00<00:02, 87.54batch/s]

Training batches on cuda:  28%|██▊       | 65/234 [00:00<00:01, 87.32batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:00<00:01, 87.83batch/s]

Training batches on cuda:  36%|███▌      | 84/234 [00:00<00:01, 87.73batch/s]

Training batches on cuda:  40%|████      | 94/234 [00:01<00:01, 89.59batch/s]

Training batches on cuda:  44%|████▍     | 103/234 [00:01<00:01, 88.13batch/s]

Training batches on cuda:  48%|████▊     | 113/234 [00:01<00:01, 90.56batch/s]

Training batches on cuda:  53%|█████▎    | 123/234 [00:01<00:01, 92.40batch/s]

Training batches on cuda:  57%|█████▋    | 133/234 [00:01<00:01, 94.07batch/s]

Training batches on cuda:  61%|██████    | 143/234 [00:01<00:00, 94.72batch/s]

Training batches on cuda:  66%|██████▌   | 154/234 [00:01<00:00, 98.20batch/s]

Training batches on cuda:  71%|███████   | 165/234 [00:01<00:00, 100.57batch/s]

Training batches on cuda:  75%|███████▌  | 176/234 [00:01<00:00, 100.90batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:01<00:00, 103.25batch/s]

Training batches on cuda:  85%|████████▍ | 198/234 [00:02<00:00, 104.63batch/s]

Training batches on cuda:  89%|████████▉ | 209/234 [00:02<00:00, 105.33batch/s]

Training batches on cuda:  94%|█████████▍| 220/234 [00:02<00:00, 106.03batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:02<00:00, 107.19batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:22,  2.55s/epoch, loss=0.00158, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:20,  2.57s/epoch, loss=0.00158, prev_loss=nan]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:03, 67.50batch/s]

Training batches on cuda:   8%|▊         | 18/234 [00:00<00:02, 75.81batch/s]

Training batches on cuda:  12%|█▏        | 29/234 [00:00<00:02, 83.17batch/s]

Training batches on cuda:  18%|█▊        | 41/234 [00:00<00:02, 89.77batch/s]

Training batches on cuda:  21%|██▏       | 50/234 [00:00<00:02, 89.36batch/s]

Training batches on cuda:  26%|██▌       | 61/234 [00:00<00:01, 94.56batch/s]

Training batches on cuda:  31%|███       | 72/234 [00:00<00:01, 97.83batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:00<00:01, 101.14batch/s]

Training batches on cuda:  40%|████      | 94/234 [00:00<00:01, 102.73batch/s]

Training batches on cuda:  45%|████▍     | 105/234 [00:01<00:01, 104.44batch/s]

Training batches on cuda:  50%|████▉     | 116/234 [00:01<00:01, 105.21batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:01<00:01, 106.10batch/s]

Training batches on cuda:  59%|█████▉    | 138/234 [00:01<00:00, 105.52batch/s]

Training batches on cuda:  64%|██████▎   | 149/234 [00:01<00:00, 105.59batch/s]

Training batches on cuda:  68%|██████▊   | 160/234 [00:01<00:00, 105.24batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:01<00:00, 101.57batch/s]

Training batches on cuda:  78%|███████▊  | 182/234 [00:01<00:00, 98.66batch/s] 

Training batches on cuda:  82%|████████▏ | 192/234 [00:01<00:00, 96.07batch/s]

Training batches on cuda:  86%|████████▋ | 202/234 [00:01<00:00, 95.87batch/s]

Training batches on cuda:  91%|█████████ | 212/234 [00:02<00:00, 95.17batch/s]

Training batches on cuda:  95%|█████████▍| 222/234 [00:02<00:00, 95.62batch/s]

Training batches on cuda: 100%|█████████▉| 233/234 [00:02<00:00, 99.16batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:20,  2.57s/epoch, loss=0.00156, prev_loss=0.00158]
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:17,  2.55s/epoch, loss=0.00156, prev_loss=0.00158]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:03, 69.24batch/s]

Training batches on cuda:   8%|▊         | 18/234 [00:00<00:02, 76.48batch/s]

Training batches on cuda:  12%|█▏        | 29/234 [00:00<00:02, 83.25batch/s]

Training batches on cuda:  17%|█▋        | 40/234 [00:00<00:02, 89.35batch/s]

Training batches on cuda:  21%|██        | 49/234 [00:00<00:02, 88.91batch/s]

Training batches on cuda:  26%|██▌       | 60/234 [00:00<00:01, 93.54batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:00<00:01, 97.56batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:00<00:01, 101.06batch/s]

Training batches on cuda:  40%|████      | 94/234 [00:00<00:01, 103.57batch/s]

Training batches on cuda:  45%|████▍     | 105/234 [00:01<00:01, 104.39batch/s]

Training batches on cuda:  50%|█████     | 117/234 [00:01<00:01, 106.59batch/s]

Training batches on cuda:  55%|█████▍    | 128/234 [00:01<00:00, 106.21batch/s]

Training batches on cuda:  59%|█████▉    | 139/234 [00:01<00:00, 106.36batch/s]

Training batches on cuda:  64%|██████▍   | 150/234 [00:01<00:00, 104.54batch/s]

Training batches on cuda:  69%|██████▉   | 161/234 [00:01<00:00, 105.30batch/s]

Training batches on cuda:  74%|███████▎  | 172/234 [00:01<00:00, 104.41batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:01<00:00, 105.83batch/s]

Training batches on cuda:  83%|████████▎ | 195/234 [00:01<00:00, 107.63batch/s]

Training batches on cuda:  88%|████████▊ | 206/234 [00:01<00:00, 107.47batch/s]

Training batches on cuda:  93%|█████████▎| 217/234 [00:02<00:00, 107.98batch/s]

Training batches on cuda:  98%|█████████▊| 229/234 [00:02<00:00, 109.28batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:17,  2.55s/epoch, loss=0.00148, prev_loss=0.00156]
Training epochs on cuda:  40%|████      | 4/10 [00:10<00:15,  2.50s/epoch, loss=0.00148, prev_loss=0.00156]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 8/234 [00:00<00:02, 76.10batch/s]

Training batches on cuda:   8%|▊         | 19/234 [00:00<00:02, 82.57batch/s]

Training batches on cuda:  12%|█▏        | 28/234 [00:00<00:02, 84.43batch/s]

Training batches on cuda:  16%|█▌        | 37/234 [00:00<00:02, 84.95batch/s]

Training batches on cuda:  21%|██        | 48/234 [00:00<00:02, 90.10batch/s]

Training batches on cuda:  25%|██▍       | 58/234 [00:00<00:01, 92.19batch/s]

Training batches on cuda:  29%|██▉       | 69/234 [00:00<00:01, 96.49batch/s]

Training batches on cuda:  34%|███▍      | 80/234 [00:00<00:01, 99.07batch/s]

Training batches on cuda:  39%|███▉      | 91/234 [00:00<00:01, 101.06batch/s]

Training batches on cuda:  43%|████▎     | 101/234 [00:01<00:01, 99.81batch/s]

Training batches on cuda:  47%|████▋     | 111/234 [00:01<00:01, 94.28batch/s]

Training batches on cuda:  52%|█████▏    | 121/234 [00:01<00:01, 94.12batch/s]

Training batches on cuda:  56%|█████▌    | 131/234 [00:01<00:01, 84.16batch/s]

Training batches on cuda:  60%|██████    | 141/234 [00:01<00:01, 86.68batch/s]

Training batches on cuda:  65%|██████▍   | 151/234 [00:01<00:00, 89.56batch/s]

Training batches on cuda:  69%|██████▉   | 161/234 [00:01<00:00, 90.59batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:01<00:00, 92.10batch/s]

Training batches on cuda:  77%|███████▋  | 181/234 [00:01<00:00, 93.26batch/s]

Training batches on cuda:  82%|████████▏ | 191/234 [00:02<00:00, 93.41batch/s]

Training batches on cuda:  86%|████████▋ | 202/234 [00:02<00:00, 96.97batch/s]

Training batches on cuda:  91%|█████████ | 213/234 [00:02<00:00, 99.12batch/s]

Training batches on cuda:  96%|█████████▌| 224/234 [00:02<00:00, 101.70batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:15,  2.50s/epoch, loss=0.00146, prev_loss=0.00148]
Training epochs on cuda:  50%|█████     | 5/10 [00:12<00:12,  2.54s/epoch, loss=0.00146, prev_loss=0.00148]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:03, 68.54batch/s]

Training batches on cuda:   7%|▋         | 16/234 [00:00<00:02, 73.06batch/s]

Training batches on cuda:  11%|█         | 26/234 [00:00<00:02, 78.05batch/s]

Training batches on cuda:  15%|█▌        | 36/234 [00:00<00:02, 83.09batch/s]

Training batches on cuda:  19%|█▉        | 44/234 [00:00<00:02, 81.47batch/s]

Training batches on cuda:  22%|██▏       | 52/234 [00:00<00:02, 77.27batch/s]

Training batches on cuda:  26%|██▋       | 62/234 [00:00<00:02, 81.51batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:00<00:02, 81.08batch/s]

Training batches on cuda:  34%|███▍      | 80/234 [00:00<00:01, 82.35batch/s]

Training batches on cuda:  38%|███▊      | 90/234 [00:01<00:01, 86.64batch/s]

Training batches on cuda:  43%|████▎     | 100/234 [00:01<00:01, 87.14batch/s]

Training batches on cuda:  47%|████▋     | 109/234 [00:01<00:01, 85.35batch/s]

Training batches on cuda:  51%|█████     | 119/234 [00:01<00:01, 88.11batch/s]

Training batches on cuda:  56%|█████▌    | 130/234 [00:01<00:01, 92.38batch/s]

Training batches on cuda:  60%|██████    | 141/234 [00:01<00:00, 96.59batch/s]

Training batches on cuda:  65%|██████▌   | 153/234 [00:01<00:00, 100.56batch/s]

Training batches on cuda:  71%|███████   | 165/234 [00:01<00:00, 103.41batch/s]

Training batches on cuda:  75%|███████▌  | 176/234 [00:01<00:00, 104.23batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:02<00:00, 103.80batch/s]

Training batches on cuda:  85%|████████▍ | 198/234 [00:02<00:00, 102.69batch/s]

Training batches on cuda:  89%|████████▉ | 209/234 [00:02<00:00, 94.24batch/s] 

Training batches on cuda:  94%|█████████▍| 221/234 [00:02<00:00, 99.01batch/s]

Training batches on cuda: 100%|█████████▉| 233/234 [00:02<00:00, 101.44batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:12,  2.54s/epoch, loss=0.00143, prev_loss=0.00146]
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:10,  2.58s/epoch, loss=0.00143, prev_loss=0.00146]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 9/234 [00:00<00:02, 86.48batch/s]

Training batches on cuda:   9%|▊         | 20/234 [00:00<00:02, 91.23batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:02, 95.55batch/s]

Training batches on cuda:  18%|█▊        | 42/234 [00:00<00:01, 98.92batch/s]

Training batches on cuda:  23%|██▎       | 53/234 [00:00<00:01, 100.92batch/s]

Training batches on cuda:  27%|██▋       | 64/234 [00:00<00:01, 100.79batch/s]

Training batches on cuda:  32%|███▏      | 74/234 [00:00<00:01, 100.00batch/s]

Training batches on cuda:  36%|███▌      | 84/234 [00:00<00:01, 98.20batch/s] 

Training batches on cuda:  40%|████      | 94/234 [00:00<00:01, 96.68batch/s]

Training batches on cuda:  44%|████▍     | 104/234 [00:01<00:01, 95.65batch/s]

Training batches on cuda:  49%|████▊     | 114/234 [00:01<00:01, 94.78batch/s]

Training batches on cuda:  53%|█████▎    | 124/234 [00:01<00:01, 92.71batch/s]

Training batches on cuda:  57%|█████▋    | 134/234 [00:01<00:01, 94.04batch/s]

Training batches on cuda:  62%|██████▏   | 146/234 [00:01<00:00, 98.95batch/s]

Training batches on cuda:  67%|██████▋   | 157/234 [00:01<00:00, 101.36batch/s]

Training batches on cuda:  72%|███████▏  | 168/234 [00:01<00:00, 101.97batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:01<00:00, 97.85batch/s] 

Training batches on cuda:  81%|████████  | 189/234 [00:01<00:00, 94.39batch/s]

Training batches on cuda:  85%|████████▌ | 199/234 [00:02<00:00, 94.87batch/s]

Training batches on cuda:  90%|████████▉ | 210/234 [00:02<00:00, 98.28batch/s]

Training batches on cuda:  94%|█████████▍| 220/234 [00:02<00:00, 98.08batch/s]

Training batches on cuda:  98%|█████████▊| 230/234 [00:02<00:00, 94.79batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:10,  2.58s/epoch, loss=0.00138, prev_loss=0.00143]
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:07,  2.58s/epoch, loss=0.00138, prev_loss=0.00143]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:03, 65.21batch/s]

Training batches on cuda:   7%|▋         | 16/234 [00:00<00:03, 70.30batch/s]

Training batches on cuda:  11%|█         | 26/234 [00:00<00:02, 75.76batch/s]

Training batches on cuda:  16%|█▌        | 37/234 [00:00<00:02, 81.72batch/s]

Training batches on cuda:  20%|██        | 47/234 [00:00<00:02, 85.43batch/s]

Training batches on cuda:  24%|██▍       | 57/234 [00:00<00:01, 88.99batch/s]

Training batches on cuda:  29%|██▊       | 67/234 [00:00<00:01, 90.74batch/s]

Training batches on cuda:  33%|███▎      | 77/234 [00:00<00:01, 91.87batch/s]

Training batches on cuda:  37%|███▋      | 87/234 [00:00<00:01, 92.20batch/s]

Training batches on cuda:  41%|████▏     | 97/234 [00:01<00:01, 93.30batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:01<00:01, 92.03batch/s]

Training batches on cuda:  50%|█████     | 117/234 [00:01<00:01, 93.04batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:01<00:01, 90.49batch/s]

Training batches on cuda:  59%|█████▊    | 137/234 [00:01<00:01, 92.12batch/s]

Training batches on cuda:  63%|██████▎   | 147/234 [00:01<00:00, 93.97batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:01<00:00, 98.70batch/s]

Training batches on cuda:  73%|███████▎  | 170/234 [00:01<00:00, 101.28batch/s]

Training batches on cuda:  77%|███████▋  | 181/234 [00:01<00:00, 103.49batch/s]

Training batches on cuda:  82%|████████▏ | 193/234 [00:02<00:00, 105.98batch/s]

Training batches on cuda:  88%|████████▊ | 205/234 [00:02<00:00, 108.42batch/s]

Training batches on cuda:  92%|█████████▏| 216/234 [00:02<00:00, 106.50batch/s]

Training batches on cuda:  97%|█████████▋| 227/234 [00:02<00:00, 105.92batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:07,  2.58s/epoch, loss=0.00128, prev_loss=0.00138]
Training epochs on cuda:  80%|████████  | 8/10 [00:20<00:05,  2.58s/epoch, loss=0.00128, prev_loss=0.00138]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:03, 65.25batch/s]

Training batches on cuda:   8%|▊         | 18/234 [00:00<00:02, 73.47batch/s]

Training batches on cuda:  12%|█▏        | 29/234 [00:00<00:02, 81.53batch/s]

Training batches on cuda:  17%|█▋        | 39/234 [00:00<00:02, 85.52batch/s]

Training batches on cuda:  21%|██        | 49/234 [00:00<00:02, 89.01batch/s]

Training batches on cuda:  25%|██▌       | 59/234 [00:00<00:01, 90.78batch/s]

Training batches on cuda:  29%|██▉       | 69/234 [00:00<00:01, 92.62batch/s]

Training batches on cuda:  34%|███▍      | 79/234 [00:00<00:01, 93.44batch/s]

Training batches on cuda:  38%|███▊      | 89/234 [00:00<00:01, 94.43batch/s]

Training batches on cuda:  43%|████▎     | 101/234 [00:01<00:01, 98.83batch/s]

Training batches on cuda:  48%|████▊     | 113/234 [00:01<00:01, 103.49batch/s]

Training batches on cuda:  53%|█████▎    | 125/234 [00:01<00:01, 106.94batch/s]

Training batches on cuda:  59%|█████▊    | 137/234 [00:01<00:00, 108.83batch/s]

Training batches on cuda:  64%|██████▎   | 149/234 [00:01<00:00, 109.89batch/s]

Training batches on cuda:  69%|██████▉   | 161/234 [00:01<00:00, 109.58batch/s]

Training batches on cuda:  74%|███████▎  | 172/234 [00:01<00:00, 109.09batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:01<00:00, 105.51batch/s]

Training batches on cuda:  83%|████████▎ | 194/234 [00:01<00:00, 93.70batch/s] 

Training batches on cuda:  87%|████████▋ | 204/234 [00:02<00:00, 88.52batch/s]

Training batches on cuda:  91%|█████████▏| 214/234 [00:02<00:00, 85.73batch/s]

Training batches on cuda:  96%|█████████▌| 224/234 [00:02<00:00, 87.34batch/s]

Training batches on cuda: 100%|██████████| 234/234 [00:02<00:00, 88.69batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:05,  2.58s/epoch, loss=0.000761, prev_loss=0.00128]
Training epochs on cuda:  90%|█████████ | 9/10 [00:23<00:02,  2.58s/epoch, loss=0.000761, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 8/234 [00:00<00:03, 74.16batch/s]

Training batches on cuda:   8%|▊         | 19/234 [00:00<00:02, 81.99batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:02, 88.83batch/s]

Training batches on cuda:  18%|█▊        | 42/234 [00:00<00:02, 93.03batch/s]

Training batches on cuda:  23%|██▎       | 54/234 [00:00<00:01, 98.16batch/s]

Training batches on cuda:  27%|██▋       | 64/234 [00:00<00:01, 98.07batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:00<00:01, 99.97batch/s]

Training batches on cuda:  36%|███▋      | 85/234 [00:00<00:01, 98.57batch/s]

Training batches on cuda:  41%|████      | 96/234 [00:00<00:01, 100.80batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:01<00:01, 102.58batch/s]

Training batches on cuda:  50%|█████     | 118/234 [00:01<00:01, 102.77batch/s]

Training batches on cuda:  55%|█████▌    | 129/234 [00:01<00:01, 97.39batch/s] 

Training batches on cuda:  59%|█████▉    | 139/234 [00:01<00:00, 95.13batch/s]

Training batches on cuda:  64%|██████▎   | 149/234 [00:01<00:00, 91.26batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:01<00:00, 85.61batch/s]

Training batches on cuda:  72%|███████▏  | 168/234 [00:01<00:00, 81.81batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:01<00:00, 87.62batch/s]

Training batches on cuda:  81%|████████  | 190/234 [00:01<00:00, 91.48batch/s]

Training batches on cuda:  85%|████████▌ | 200/234 [00:02<00:00, 89.39batch/s]

Training batches on cuda:  90%|█████████ | 211/234 [00:02<00:00, 92.65batch/s]

Training batches on cuda:  95%|█████████▍| 222/234 [00:02<00:00, 95.18batch/s]

Training batches on cuda:  99%|█████████▉| 232/234 [00:02<00:00, 96.40batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:25<00:02,  2.58s/epoch, loss=0.000553, prev_loss=0.000761]2021-09-16:17:44:47,903 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:44:48,907 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:45:03,804 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
2021-09-16:17:45:03,921 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00,  7.41s/epoch, loss=0.000553, prev_loss=0.000761]
Training epochs on cuda: 100%|██████████| 10/10 [00:41<00:00,  4.18s/epoch, loss=0.000553, prev_loss=0.000761]
2021-09-16:17:45:03,922 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmphnhnsj1x'
2021-09-16:17:45:03,999 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmphnhnsj1x' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.52ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.61ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.62ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.66ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.66ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.52ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.56ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.59ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.61ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.62ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:45:18,784 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 17:45:18,788] Trial 12 finished with value: 0.9534737906551205 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.01037352199327964, 'optimizer.lr': 0.0187117625151004, 'negative_sampler.num_negs_per_pos': 6, 'training.batch_size': 1332}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:45:18,808 WARNING  [api.py:823] No random seed is specified. Setting to 3380451122.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 9/584 [00:00<00:06, 87.49batch/s]

Training batches on cuda:   4%|▍         | 24/584 [00:00<00:05, 99.15batch/s]

Training batches on cuda:   6%|▋         | 37/584 [00:00<00:05, 105.47batch/s]

Training batches on cuda:   9%|▉         | 54/584 [00:00<00:04, 118.79batch/s]

Training batches on cuda:  12%|█▏        | 71/584 [00:00<00:03, 128.97batch/s]

Training batches on cuda:  15%|█▌        | 88/584 [00:00<00:03, 137.04batch/s]

Training batches on cuda:  18%|█▊        | 105/584 [00:00<00:03, 143.64batch/s]

Training batches on cuda:  21%|██        | 122/584 [00:00<00:03, 148.80batch/s]

Training batches on cuda:  24%|██▍       | 140/584 [00:00<00:02, 155.76batch/s]

Training batches on cuda:  27%|██▋       | 160/584 [00:01<00:02, 164.99batch/s]

Training batches on cuda:  31%|███       | 179/584 [00:01<00:02, 171.42batch/s]

Training batches on cuda:  34%|███▍      | 198/584 [00:01<00:02, 176.50batch/s]

Training batches on cuda:  37%|███▋      | 218/584 [00:01<00:02, 180.74batch/s]

Training batches on cuda:  41%|████      | 237/584 [00:01<00:01, 181.05batch/s]

Training batches on cuda:  44%|████▍     | 256/584 [00:01<00:01, 183.03batch/s]

Training batches on cuda:  47%|████▋     | 275/584 [00:01<00:01, 184.24batch/s]

Training batches on cuda:  50%|█████     | 294/584 [00:01<00:01, 183.19batch/s]

Training batches on cuda:  54%|█████▎    | 313/584 [00:01<00:01, 184.70batch/s]

Training batches on cuda:  57%|█████▋    | 332/584 [00:01<00:01, 184.49batch/s]

Training batches on cuda:  60%|██████    | 351/584 [00:02<00:01, 185.01batch/s]

Training batches on cuda:  63%|██████▎   | 370/584 [00:02<00:01, 184.59batch/s]

Training batches on cuda:  67%|██████▋   | 389/584 [00:02<00:01, 183.40batch/s]

Training batches on cuda:  70%|██████▉   | 408/584 [00:02<00:01, 169.38batch/s]

Training batches on cuda:  73%|███████▎  | 426/584 [00:02<00:00, 162.95batch/s]

Training batches on cuda:  76%|███████▌  | 443/584 [00:02<00:00, 161.20batch/s]

Training batches on cuda:  79%|███████▉  | 460/584 [00:02<00:00, 158.86batch/s]

Training batches on cuda:  82%|████████▏ | 476/584 [00:02<00:00, 157.92batch/s]

Training batches on cuda:  84%|████████▍ | 492/584 [00:02<00:00, 143.71batch/s]

Training batches on cuda:  87%|████████▋ | 507/584 [00:03<00:00, 139.33batch/s]

Training batches on cuda:  89%|████████▉ | 522/584 [00:03<00:00, 137.70batch/s]

Training batches on cuda:  92%|█████████▏| 537/584 [00:03<00:00, 139.04batch/s]

Training batches on cuda:  95%|█████████▍| 552/584 [00:03<00:00, 134.18batch/s]

Training batches on cuda:  97%|█████████▋| 566/584 [00:03<00:00, 131.87batch/s]

Training batches on cuda:  99%|█████████▉| 580/584 [00:03<00:00, 133.62batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.00574, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:35,  3.95s/epoch, loss=0.00574, prev_loss=nan]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/584 [00:00<00:05, 109.18batch/s]

Training batches on cuda:   4%|▍         | 25/584 [00:00<00:04, 116.39batch/s]

Training batches on cuda:   7%|▋         | 38/584 [00:00<00:04, 118.56batch/s]

Training batches on cuda:   9%|▉         | 52/584 [00:00<00:04, 122.68batch/s]

Training batches on cuda:  12%|█▏        | 68/584 [00:00<00:03, 130.95batch/s]

Training batches on cuda:  15%|█▍        | 85/584 [00:00<00:03, 140.62batch/s]

Training batches on cuda:  18%|█▊        | 104/584 [00:00<00:03, 151.09batch/s]

Training batches on cuda:  21%|██        | 120/584 [00:00<00:03, 152.23batch/s]

Training batches on cuda:  23%|██▎       | 136/584 [00:00<00:02, 153.56batch/s]

Training batches on cuda:  26%|██▌       | 152/584 [00:01<00:03, 133.54batch/s]

Training batches on cuda:  28%|██▊       | 166/584 [00:01<00:03, 134.77batch/s]

Training batches on cuda:  31%|███       | 181/584 [00:01<00:02, 137.23batch/s]

Training batches on cuda:  33%|███▎      | 195/584 [00:01<00:02, 133.58batch/s]

Training batches on cuda:  36%|███▌      | 209/584 [00:01<00:02, 129.79batch/s]

Training batches on cuda:  38%|███▊      | 223/584 [00:01<00:02, 125.55batch/s]

Training batches on cuda:  40%|████      | 236/584 [00:01<00:02, 125.66batch/s]

Training batches on cuda:  43%|████▎     | 251/584 [00:01<00:02, 130.41batch/s]

Training batches on cuda:  45%|████▌     | 265/584 [00:01<00:02, 129.13batch/s]

Training batches on cuda:  48%|████▊     | 278/584 [00:02<00:02, 127.08batch/s]

Training batches on cuda:  50%|████▉     | 291/584 [00:02<00:02, 127.28batch/s]

Training batches on cuda:  52%|█████▏    | 304/584 [00:02<00:02, 126.90batch/s]

Training batches on cuda:  54%|█████▍    | 317/584 [00:02<00:02, 126.51batch/s]

Training batches on cuda:  57%|█████▋    | 330/584 [00:02<00:02, 122.70batch/s]

Training batches on cuda:  59%|█████▊    | 343/584 [00:02<00:01, 121.74batch/s]

Training batches on cuda:  61%|██████    | 356/584 [00:02<00:01, 123.49batch/s]

Training batches on cuda:  63%|██████▎   | 369/584 [00:02<00:01, 122.78batch/s]

Training batches on cuda:  66%|██████▌   | 385/584 [00:02<00:01, 130.48batch/s]

Training batches on cuda:  69%|██████▊   | 401/584 [00:03<00:01, 137.64batch/s]

Training batches on cuda:  71%|███████   | 416/584 [00:03<00:01, 139.38batch/s]

Training batches on cuda:  74%|███████▍  | 431/584 [00:03<00:01, 140.15batch/s]

Training batches on cuda:  76%|███████▋  | 446/584 [00:03<00:00, 141.44batch/s]

Training batches on cuda:  79%|███████▉  | 461/584 [00:03<00:00, 138.82batch/s]

Training batches on cuda:  81%|████████▏ | 475/584 [00:03<00:00, 138.95batch/s]

Training batches on cuda:  84%|████████▍ | 490/584 [00:03<00:00, 139.63batch/s]

Training batches on cuda:  86%|████████▋ | 504/584 [00:03<00:00, 138.99batch/s]

Training batches on cuda:  89%|████████▊ | 518/584 [00:03<00:00, 138.77batch/s]

Training batches on cuda:  91%|█████████▏| 533/584 [00:03<00:00, 139.90batch/s]

Training batches on cuda:  94%|█████████▍| 548/584 [00:04<00:00, 140.70batch/s]

Training batches on cuda:  96%|█████████▋| 563/584 [00:04<00:00, 143.29batch/s]

Training batches on cuda:  99%|█████████▉| 578/584 [00:04<00:00, 143.60batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:35,  3.95s/epoch, loss=0.00573, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:33,  4.14s/epoch, loss=0.00573, prev_loss=nan]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 10/584 [00:00<00:05, 98.45batch/s]

Training batches on cuda:   4%|▍         | 24/584 [00:00<00:05, 107.51batch/s]

Training batches on cuda:   7%|▋         | 38/584 [00:00<00:04, 114.64batch/s]

Training batches on cuda:   9%|▉         | 53/584 [00:00<00:04, 122.71batch/s]

Training batches on cuda:  12%|█▏        | 68/584 [00:00<00:04, 128.32batch/s]

Training batches on cuda:  14%|█▍        | 83/584 [00:00<00:03, 133.51batch/s]

Training batches on cuda:  17%|█▋        | 99/584 [00:00<00:03, 139.78batch/s]

Training batches on cuda:  20%|█▉        | 116/584 [00:00<00:03, 145.91batch/s]

Training batches on cuda:  23%|██▎       | 133/584 [00:00<00:02, 150.76batch/s]

Training batches on cuda:  26%|██▌       | 149/584 [00:01<00:02, 152.16batch/s]

Training batches on cuda:  28%|██▊       | 166/584 [00:01<00:02, 156.77batch/s]

Training batches on cuda:  31%|███       | 182/584 [00:01<00:02, 156.94batch/s]

Training batches on cuda:  34%|███▍      | 199/584 [00:01<00:02, 160.29batch/s]

Training batches on cuda:  37%|███▋      | 216/584 [00:01<00:02, 159.15batch/s]

Training batches on cuda:  40%|███▉      | 232/584 [00:01<00:02, 144.13batch/s]

Training batches on cuda:  42%|████▏     | 248/584 [00:01<00:02, 148.03batch/s]

Training batches on cuda:  45%|████▌     | 265/584 [00:01<00:02, 151.68batch/s]

Training batches on cuda:  48%|████▊     | 282/584 [00:01<00:01, 154.10batch/s]

Training batches on cuda:  51%|█████     | 298/584 [00:01<00:01, 155.76batch/s]

Training batches on cuda:  54%|█████▍    | 314/584 [00:02<00:01, 154.96batch/s]

Training batches on cuda:  57%|█████▋    | 331/584 [00:02<00:01, 156.19batch/s]

Training batches on cuda:  59%|█████▉    | 347/584 [00:02<00:01, 156.20batch/s]

Training batches on cuda:  62%|██████▏   | 363/584 [00:02<00:01, 150.08batch/s]

Training batches on cuda:  65%|██████▌   | 380/584 [00:02<00:01, 153.74batch/s]

Training batches on cuda:  68%|██████▊   | 396/584 [00:02<00:01, 153.94batch/s]

Training batches on cuda:  71%|███████   | 413/584 [00:02<00:01, 156.42batch/s]

Training batches on cuda:  74%|███████▎  | 430/584 [00:02<00:00, 157.72batch/s]

Training batches on cuda:  76%|███████▋  | 446/584 [00:02<00:00, 158.02batch/s]

Training batches on cuda:  79%|███████▉  | 462/584 [00:03<00:00, 156.57batch/s]

Training batches on cuda:  82%|████████▏ | 479/584 [00:03<00:00, 158.26batch/s]

Training batches on cuda:  85%|████████▍ | 496/584 [00:03<00:00, 159.99batch/s]

Training batches on cuda:  88%|████████▊ | 513/584 [00:03<00:00, 145.73batch/s]

Training batches on cuda:  90%|█████████ | 528/584 [00:03<00:00, 139.49batch/s]

Training batches on cuda:  93%|█████████▎| 543/584 [00:03<00:00, 140.30batch/s]

Training batches on cuda:  96%|█████████▌| 558/584 [00:03<00:00, 138.80batch/s]

Training batches on cuda:  98%|█████████▊| 573/584 [00:03<00:00, 140.23batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:33,  4.14s/epoch, loss=0.00573, prev_loss=0.00573]
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:28,  4.12s/epoch, loss=0.00573, prev_loss=0.00573]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 12/584 [00:00<00:05, 113.77batch/s]

Training batches on cuda:   5%|▍         | 29/584 [00:00<00:04, 125.08batch/s]

Training batches on cuda:   8%|▊         | 45/584 [00:00<00:04, 132.87batch/s]

Training batches on cuda:  10%|█         | 59/584 [00:00<00:03, 133.18batch/s]

Training batches on cuda:  13%|█▎        | 74/584 [00:00<00:03, 136.28batch/s]

Training batches on cuda:  15%|█▌        | 89/584 [00:00<00:03, 138.32batch/s]

Training batches on cuda:  18%|█▊        | 105/584 [00:00<00:03, 142.24batch/s]

Training batches on cuda:  21%|██        | 122/584 [00:00<00:03, 149.26batch/s]

Training batches on cuda:  24%|██▎       | 138/584 [00:00<00:02, 151.85batch/s]

Training batches on cuda:  26%|██▌       | 153/584 [00:01<00:02, 150.54batch/s]

Training batches on cuda:  29%|██▉       | 168/584 [00:01<00:02, 148.65batch/s]

Training batches on cuda:  31%|███▏      | 183/584 [00:01<00:02, 146.43batch/s]

Training batches on cuda:  34%|███▍      | 198/584 [00:01<00:02, 145.41batch/s]

Training batches on cuda:  36%|███▋      | 213/584 [00:01<00:02, 140.79batch/s]

Training batches on cuda:  39%|███▉      | 228/584 [00:01<00:02, 138.82batch/s]

Training batches on cuda:  41%|████▏     | 242/584 [00:01<00:02, 136.80batch/s]

Training batches on cuda:  44%|████▍     | 256/584 [00:01<00:02, 133.50batch/s]

Training batches on cuda:  46%|████▌     | 270/584 [00:01<00:02, 133.37batch/s]

Training batches on cuda:  49%|████▊     | 284/584 [00:02<00:02, 133.56batch/s]

Training batches on cuda:  51%|█████     | 298/584 [00:02<00:02, 131.10batch/s]

Training batches on cuda:  53%|█████▎    | 312/584 [00:02<00:02, 130.65batch/s]

Training batches on cuda:  56%|█████▌    | 326/584 [00:02<00:01, 131.38batch/s]

Training batches on cuda:  58%|█████▊    | 340/584 [00:02<00:01, 130.84batch/s]

Training batches on cuda:  61%|██████    | 354/584 [00:02<00:02, 114.06batch/s]

Training batches on cuda:  63%|██████▎   | 367/584 [00:02<00:01, 117.12batch/s]

Training batches on cuda:  65%|██████▌   | 380/584 [00:02<00:01, 120.29batch/s]

Training batches on cuda:  67%|██████▋   | 393/584 [00:02<00:01, 122.70batch/s]

Training batches on cuda:  70%|██████▉   | 406/584 [00:03<00:01, 121.76batch/s]

Training batches on cuda:  72%|███████▏  | 421/584 [00:03<00:01, 128.43batch/s]

Training batches on cuda:  75%|███████▍  | 436/584 [00:03<00:01, 133.84batch/s]

Training batches on cuda:  77%|███████▋  | 451/584 [00:03<00:00, 138.03batch/s]

Training batches on cuda:  80%|███████▉  | 467/584 [00:03<00:00, 141.99batch/s]

Training batches on cuda:  83%|████████▎ | 482/584 [00:03<00:00, 143.76batch/s]

Training batches on cuda:  85%|████████▌ | 498/584 [00:03<00:00, 146.48batch/s]

Training batches on cuda:  88%|████████▊ | 514/584 [00:03<00:00, 148.87batch/s]

Training batches on cuda:  91%|█████████ | 530/584 [00:03<00:00, 149.49batch/s]

Training batches on cuda:  93%|█████████▎| 545/584 [00:03<00:00, 147.43batch/s]

Training batches on cuda:  96%|█████████▌| 561/584 [00:04<00:00, 148.85batch/s]

Training batches on cuda:  99%|█████████▉| 577/584 [00:04<00:00, 149.35batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:28,  4.12s/epoch, loss=0.00572, prev_loss=0.00573]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:25,  4.21s/epoch, loss=0.00572, prev_loss=0.00573]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 8/584 [00:00<00:07, 75.61batch/s]

Training batches on cuda:   4%|▎         | 21/584 [00:00<00:06, 86.08batch/s]

Training batches on cuda:   6%|▌         | 35/584 [00:00<00:05, 96.47batch/s]

Training batches on cuda:   9%|▊         | 50/584 [00:00<00:05, 106.53batch/s]

Training batches on cuda:  11%|█         | 64/584 [00:00<00:04, 114.39batch/s]

Training batches on cuda:  13%|█▎        | 78/584 [00:00<00:04, 121.00batch/s]

Training batches on cuda:  16%|█▌        | 92/584 [00:00<00:03, 125.56batch/s]

Training batches on cuda:  18%|█▊        | 105/584 [00:00<00:03, 124.96batch/s]

Training batches on cuda:  21%|██        | 120/584 [00:00<00:03, 129.77batch/s]

Training batches on cuda:  23%|██▎       | 135/584 [00:01<00:03, 134.91batch/s]

Training batches on cuda:  26%|██▌       | 150/584 [00:01<00:03, 136.49batch/s]

Training batches on cuda:  28%|██▊       | 164/584 [00:01<00:03, 136.37batch/s]

Training batches on cuda:  30%|███       | 178/584 [00:01<00:03, 134.49batch/s]

Training batches on cuda:  33%|███▎      | 192/584 [00:01<00:02, 135.55batch/s]

Training batches on cuda:  35%|███▌      | 207/584 [00:01<00:02, 137.35batch/s]

Training batches on cuda:  38%|███▊      | 222/584 [00:01<00:02, 139.37batch/s]

Training batches on cuda:  41%|████      | 237/584 [00:01<00:02, 141.14batch/s]

Training batches on cuda:  43%|████▎     | 252/584 [00:01<00:02, 141.07batch/s]

Training batches on cuda:  46%|████▌     | 267/584 [00:01<00:02, 137.76batch/s]

Training batches on cuda:  48%|████▊     | 281/584 [00:02<00:02, 136.41batch/s]

Training batches on cuda:  51%|█████     | 295/584 [00:02<00:02, 136.87batch/s]

Training batches on cuda:  53%|█████▎    | 311/584 [00:02<00:01, 142.29batch/s]

Training batches on cuda:  56%|█████▌    | 328/584 [00:02<00:01, 147.72batch/s]

Training batches on cuda:  59%|█████▉    | 344/584 [00:02<00:01, 151.17batch/s]

Training batches on cuda:  62%|██████▏   | 360/584 [00:02<00:01, 150.89batch/s]

Training batches on cuda:  64%|██████▍   | 376/584 [00:02<00:01, 142.95batch/s]

Training batches on cuda:  67%|██████▋   | 391/584 [00:02<00:01, 139.34batch/s]

Training batches on cuda:  70%|██████▉   | 406/584 [00:02<00:01, 140.08batch/s]

Training batches on cuda:  72%|███████▏  | 421/584 [00:03<00:01, 142.72batch/s]

Training batches on cuda:  75%|███████▍  | 436/584 [00:03<00:01, 138.05batch/s]

Training batches on cuda:  77%|███████▋  | 452/584 [00:03<00:00, 143.55batch/s]

Training batches on cuda:  80%|████████  | 468/584 [00:03<00:00, 148.10batch/s]

Training batches on cuda:  83%|████████▎ | 484/584 [00:03<00:00, 151.36batch/s]

Training batches on cuda:  86%|████████▌ | 500/584 [00:03<00:00, 153.69batch/s]

Training batches on cuda:  88%|████████▊ | 516/584 [00:03<00:00, 154.45batch/s]

Training batches on cuda:  91%|█████████ | 532/584 [00:03<00:00, 155.98batch/s]

Training batches on cuda:  94%|█████████▍| 548/584 [00:03<00:00, 154.34batch/s]

Training batches on cuda:  97%|█████████▋| 564/584 [00:03<00:00, 152.17batch/s]

Training batches on cuda:  99%|█████████▉| 580/584 [00:04<00:00, 148.13batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:21<00:25,  4.21s/epoch, loss=0.00572, prev_loss=0.00572]
Training epochs on cuda:  50%|█████     | 5/10 [00:21<00:21,  4.25s/epoch, loss=0.00572, prev_loss=0.00572]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 10/584 [00:00<00:06, 95.36batch/s]

Training batches on cuda:   4%|▍         | 23/584 [00:00<00:05, 103.05batch/s]

Training batches on cuda:   6%|▌         | 36/584 [00:00<00:05, 109.25batch/s]

Training batches on cuda:   8%|▊         | 49/584 [00:00<00:04, 113.95batch/s]

Training batches on cuda:  11%|█         | 62/584 [00:00<00:04, 117.38batch/s]

Training batches on cuda:  13%|█▎        | 76/584 [00:00<00:04, 122.94batch/s]

Training batches on cuda:  16%|█▌        | 91/584 [00:00<00:03, 127.91batch/s]

Training batches on cuda:  18%|█▊        | 106/584 [00:00<00:03, 132.81batch/s]

Training batches on cuda:  21%|██        | 120/584 [00:00<00:03, 132.95batch/s]

Training batches on cuda:  23%|██▎       | 134/584 [00:01<00:03, 131.19batch/s]

Training batches on cuda:  25%|██▌       | 148/584 [00:01<00:03, 130.24batch/s]

Training batches on cuda:  28%|██▊       | 161/584 [00:01<00:03, 124.91batch/s]

Training batches on cuda:  30%|██▉       | 174/584 [00:01<00:03, 125.82batch/s]

Training batches on cuda:  32%|███▏      | 189/584 [00:01<00:03, 130.45batch/s]

Training batches on cuda:  35%|███▍      | 203/584 [00:01<00:02, 130.42batch/s]

Training batches on cuda:  37%|███▋      | 217/584 [00:01<00:02, 132.88batch/s]

Training batches on cuda:  40%|███▉      | 232/584 [00:01<00:02, 136.00batch/s]

Training batches on cuda:  42%|████▏     | 247/584 [00:01<00:02, 138.13batch/s]

Training batches on cuda:  45%|████▍     | 262/584 [00:01<00:02, 139.11batch/s]

Training batches on cuda:  47%|████▋     | 277/584 [00:02<00:02, 140.19batch/s]

Training batches on cuda:  50%|█████     | 292/584 [00:02<00:02, 140.87batch/s]

Training batches on cuda:  53%|█████▎    | 307/584 [00:02<00:01, 140.34batch/s]

Training batches on cuda:  55%|█████▌    | 322/584 [00:02<00:01, 135.64batch/s]

Training batches on cuda:  58%|█████▊    | 336/584 [00:02<00:01, 134.61batch/s]

Training batches on cuda:  60%|█████▉    | 350/584 [00:02<00:01, 134.16batch/s]

Training batches on cuda:  62%|██████▏   | 364/584 [00:02<00:01, 135.46batch/s]

Training batches on cuda:  65%|██████▍   | 379/584 [00:02<00:01, 138.07batch/s]

Training batches on cuda:  67%|██████▋   | 393/584 [00:02<00:01, 134.08batch/s]

Training batches on cuda:  70%|██████▉   | 407/584 [00:03<00:01, 129.86batch/s]

Training batches on cuda:  72%|███████▏  | 421/584 [00:03<00:01, 119.66batch/s]

Training batches on cuda:  74%|███████▍  | 434/584 [00:03<00:01, 120.40batch/s]

Training batches on cuda:  77%|███████▋  | 448/584 [00:03<00:01, 124.55batch/s]

Training batches on cuda:  79%|███████▉  | 461/584 [00:03<00:00, 124.54batch/s]

Training batches on cuda:  81%|████████▏ | 475/584 [00:03<00:00, 127.69batch/s]

Training batches on cuda:  84%|████████▎ | 489/584 [00:03<00:00, 129.62batch/s]

Training batches on cuda:  86%|████████▌ | 503/584 [00:03<00:00, 118.63batch/s]

Training batches on cuda:  89%|████████▊ | 517/584 [00:03<00:00, 123.94batch/s]

Training batches on cuda:  91%|█████████ | 532/584 [00:04<00:00, 128.49batch/s]

Training batches on cuda:  93%|█████████▎| 546/584 [00:04<00:00, 131.05batch/s]

Training batches on cuda:  96%|█████████▌| 560/584 [00:04<00:00, 132.91batch/s]

Training batches on cuda:  98%|█████████▊| 574/584 [00:04<00:00, 133.44batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:26<00:21,  4.25s/epoch, loss=0.0057, prev_loss=0.00572] 
Training epochs on cuda:  60%|██████    | 6/10 [00:26<00:17,  4.37s/epoch, loss=0.0057, prev_loss=0.00572]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 8/584 [00:00<00:07, 79.94batch/s]

Training batches on cuda:   4%|▍         | 23/584 [00:00<00:06, 92.90batch/s]

Training batches on cuda:   7%|▋         | 39/584 [00:00<00:05, 105.39batch/s]

Training batches on cuda:   9%|▉         | 55/584 [00:00<00:04, 116.38batch/s]

Training batches on cuda:  12%|█▏        | 72/584 [00:00<00:04, 126.83batch/s]

Training batches on cuda:  15%|█▌        | 88/584 [00:00<00:03, 133.60batch/s]

Training batches on cuda:  18%|█▊        | 104/584 [00:00<00:03, 140.30batch/s]

Training batches on cuda:  20%|██        | 119/584 [00:00<00:03, 141.68batch/s]

Training batches on cuda:  23%|██▎       | 135/584 [00:00<00:03, 145.13batch/s]

Training batches on cuda:  26%|██▌       | 150/584 [00:01<00:03, 143.70batch/s]

Training batches on cuda:  28%|██▊       | 165/584 [00:01<00:02, 144.02batch/s]

Training batches on cuda:  31%|███       | 180/584 [00:01<00:02, 141.78batch/s]

Training batches on cuda:  33%|███▎      | 195/584 [00:01<00:02, 132.43batch/s]

Training batches on cuda:  36%|███▌      | 209/584 [00:01<00:02, 128.97batch/s]

Training batches on cuda:  38%|███▊      | 223/584 [00:01<00:02, 131.81batch/s]

Training batches on cuda:  41%|████      | 237/584 [00:01<00:02, 133.00batch/s]

Training batches on cuda:  43%|████▎     | 251/584 [00:01<00:02, 133.10batch/s]

Training batches on cuda:  46%|████▌     | 266/584 [00:01<00:02, 135.23batch/s]

Training batches on cuda:  48%|████▊     | 280/584 [00:02<00:02, 135.84batch/s]

Training batches on cuda:  51%|█████     | 295/584 [00:02<00:02, 137.66batch/s]

Training batches on cuda:  53%|█████▎    | 309/584 [00:02<00:01, 137.88batch/s]

Training batches on cuda:  55%|█████▌    | 323/584 [00:02<00:01, 135.76batch/s]

Training batches on cuda:  58%|█████▊    | 337/584 [00:02<00:01, 133.46batch/s]

Training batches on cuda:  60%|██████    | 351/584 [00:02<00:01, 134.33batch/s]

Training batches on cuda:  62%|██████▎   | 365/584 [00:02<00:01, 133.77batch/s]

Training batches on cuda:  65%|██████▍   | 379/584 [00:02<00:01, 135.32batch/s]

Training batches on cuda:  67%|██████▋   | 394/584 [00:02<00:01, 137.41batch/s]

Training batches on cuda:  70%|██████▉   | 408/584 [00:02<00:01, 138.09batch/s]

Training batches on cuda:  73%|███████▎  | 424/584 [00:03<00:01, 143.37batch/s]

Training batches on cuda:  75%|███████▌  | 439/584 [00:03<00:01, 138.29batch/s]

Training batches on cuda:  78%|███████▊  | 454/584 [00:03<00:00, 139.40batch/s]

Training batches on cuda:  80%|████████  | 470/584 [00:03<00:00, 141.76batch/s]

Training batches on cuda:  83%|████████▎ | 485/584 [00:03<00:00, 138.29batch/s]

Training batches on cuda:  86%|████████▌ | 501/584 [00:03<00:00, 142.31batch/s]

Training batches on cuda:  88%|████████▊ | 516/584 [00:03<00:00, 143.72batch/s]

Training batches on cuda:  91%|█████████ | 531/584 [00:03<00:00, 143.57batch/s]

Training batches on cuda:  93%|█████████▎| 546/584 [00:03<00:00, 139.60batch/s]

Training batches on cuda:  96%|█████████▌| 561/584 [00:04<00:00, 136.23batch/s]

Training batches on cuda:  98%|█████████▊| 575/584 [00:04<00:00, 135.93batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:17,  4.37s/epoch, loss=0.00565, prev_loss=0.0057]
Training epochs on cuda:  70%|███████   | 7/10 [00:30<00:13,  4.43s/epoch, loss=0.00565, prev_loss=0.0057]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 12/584 [00:00<00:04, 116.04batch/s]

Training batches on cuda:   4%|▍         | 26/584 [00:00<00:04, 120.41batch/s]

Training batches on cuda:   7%|▋         | 40/584 [00:00<00:04, 124.40batch/s]

Training batches on cuda:   9%|▉         | 54/584 [00:00<00:04, 127.36batch/s]

Training batches on cuda:  12%|█▏        | 69/584 [00:00<00:03, 132.72batch/s]

Training batches on cuda:  15%|█▍        | 85/584 [00:00<00:03, 138.30batch/s]

Training batches on cuda:  17%|█▋        | 100/584 [00:00<00:03, 141.60batch/s]

Training batches on cuda:  20%|█▉        | 116/584 [00:00<00:03, 144.58batch/s]

Training batches on cuda:  23%|██▎       | 132/584 [00:00<00:03, 147.36batch/s]

Training batches on cuda:  25%|██▌       | 147/584 [00:01<00:03, 138.29batch/s]

Training batches on cuda:  28%|██▊       | 161/584 [00:01<00:03, 136.83batch/s]

Training batches on cuda:  30%|███       | 176/584 [00:01<00:02, 139.27batch/s]

Training batches on cuda:  33%|███▎      | 192/584 [00:01<00:02, 142.65batch/s]

Training batches on cuda:  36%|███▌      | 208/584 [00:01<00:02, 146.81batch/s]

Training batches on cuda:  38%|███▊      | 224/584 [00:01<00:02, 149.66batch/s]

Training batches on cuda:  41%|████▏     | 241/584 [00:01<00:02, 154.19batch/s]

Training batches on cuda:  44%|████▍     | 257/584 [00:01<00:02, 145.81batch/s]

Training batches on cuda:  47%|████▋     | 272/584 [00:01<00:02, 137.58batch/s]

Training batches on cuda:  49%|████▉     | 287/584 [00:02<00:02, 140.83batch/s]

Training batches on cuda:  52%|█████▏    | 302/584 [00:02<00:02, 139.55batch/s]

Training batches on cuda:  54%|█████▍    | 318/584 [00:02<00:01, 144.20batch/s]

Training batches on cuda:  57%|█████▋    | 333/584 [00:02<00:01, 143.77batch/s]

Training batches on cuda:  60%|█████▉    | 348/584 [00:02<00:01, 131.64batch/s]

Training batches on cuda:  62%|██████▏   | 363/584 [00:02<00:01, 135.97batch/s]

Training batches on cuda:  65%|██████▍   | 379/584 [00:02<00:01, 141.45batch/s]

Training batches on cuda:  68%|██████▊   | 395/584 [00:02<00:01, 144.29batch/s]

Training batches on cuda:  70%|███████   | 410/584 [00:02<00:01, 144.98batch/s]

Training batches on cuda:  73%|███████▎  | 425/584 [00:02<00:01, 142.48batch/s]

Training batches on cuda:  75%|███████▌  | 440/584 [00:03<00:01, 142.21batch/s]

Training batches on cuda:  78%|███████▊  | 455/584 [00:03<00:00, 142.54batch/s]

Training batches on cuda:  80%|████████  | 470/584 [00:03<00:00, 134.27batch/s]

Training batches on cuda:  83%|████████▎ | 484/584 [00:03<00:00, 130.70batch/s]

Training batches on cuda:  85%|████████▌ | 498/584 [00:03<00:00, 125.68batch/s]

Training batches on cuda:  88%|████████▊ | 512/584 [00:03<00:00, 127.70batch/s]

Training batches on cuda:  90%|█████████ | 527/584 [00:03<00:00, 131.81batch/s]

Training batches on cuda:  93%|█████████▎| 542/584 [00:03<00:00, 134.37batch/s]

Training batches on cuda:  95%|█████████▌| 557/584 [00:03<00:00, 136.69batch/s]

Training batches on cuda:  98%|█████████▊| 572/584 [00:04<00:00, 138.12batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:13,  4.43s/epoch, loss=0.00546, prev_loss=0.00565]
Training epochs on cuda:  80%|████████  | 8/10 [00:35<00:08,  4.43s/epoch, loss=0.00546, prev_loss=0.00565]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 13/584 [00:00<00:04, 123.52batch/s]

Training batches on cuda:   5%|▍         | 29/584 [00:00<00:04, 132.01batch/s]

Training batches on cuda:   7%|▋         | 39/584 [00:00<00:04, 120.23batch/s]

Training batches on cuda:   9%|▉         | 55/584 [00:00<00:04, 129.62batch/s]

Training batches on cuda:  12%|█▏        | 71/584 [00:00<00:03, 135.67batch/s]

Training batches on cuda:  15%|█▌        | 88/584 [00:00<00:03, 142.52batch/s]

Training batches on cuda:  18%|█▊        | 105/584 [00:00<00:03, 148.41batch/s]

Training batches on cuda:  21%|██        | 121/584 [00:00<00:03, 151.59batch/s]

Training batches on cuda:  24%|██▎       | 138/584 [00:00<00:02, 154.02batch/s]

Training batches on cuda:  27%|██▋       | 155/584 [00:01<00:02, 156.54batch/s]

Training batches on cuda:  29%|██▉       | 172/584 [00:01<00:02, 157.61batch/s]

Training batches on cuda:  33%|███▎      | 190/584 [00:01<00:02, 161.39batch/s]

Training batches on cuda:  35%|███▌      | 207/584 [00:01<00:02, 162.06batch/s]

Training batches on cuda:  38%|███▊      | 224/584 [00:01<00:02, 160.66batch/s]

Training batches on cuda:  41%|████▏     | 242/584 [00:01<00:02, 164.30batch/s]

Training batches on cuda:  44%|████▍     | 259/584 [00:01<00:01, 164.19batch/s]

Training batches on cuda:  47%|████▋     | 276/584 [00:01<00:01, 158.01batch/s]

Training batches on cuda:  50%|█████     | 292/584 [00:01<00:01, 151.68batch/s]

Training batches on cuda:  53%|█████▎    | 308/584 [00:02<00:01, 143.08batch/s]

Training batches on cuda:  55%|█████▌    | 323/584 [00:02<00:01, 140.74batch/s]

Training batches on cuda:  58%|█████▊    | 338/584 [00:02<00:01, 141.24batch/s]

Training batches on cuda:  60%|██████    | 353/584 [00:02<00:01, 137.66batch/s]

Training batches on cuda:  63%|██████▎   | 368/584 [00:02<00:01, 139.08batch/s]

Training batches on cuda:  66%|██████▌   | 384/584 [00:02<00:01, 144.43batch/s]

Training batches on cuda:  69%|██████▊   | 401/584 [00:02<00:01, 150.61batch/s]

Training batches on cuda:  72%|███████▏  | 418/584 [00:02<00:01, 154.81batch/s]

Training batches on cuda:  74%|███████▍  | 435/584 [00:02<00:00, 157.53batch/s]

Training batches on cuda:  77%|███████▋  | 452/584 [00:02<00:00, 160.09batch/s]

Training batches on cuda:  80%|████████  | 469/584 [00:03<00:00, 161.49batch/s]

Training batches on cuda:  83%|████████▎ | 486/584 [00:03<00:00, 161.62batch/s]

Training batches on cuda:  86%|████████▌ | 503/584 [00:03<00:00, 161.91batch/s]

Training batches on cuda:  89%|████████▉ | 520/584 [00:03<00:00, 161.54batch/s]

Training batches on cuda:  92%|█████████▏| 537/584 [00:03<00:00, 162.57batch/s]

Training batches on cuda:  95%|█████████▍| 554/584 [00:03<00:00, 162.12batch/s]

Training batches on cuda:  98%|█████████▊| 571/584 [00:03<00:00, 162.72batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:39<00:08,  4.43s/epoch, loss=0.00483, prev_loss=0.00546]
Training epochs on cuda:  90%|█████████ | 9/10 [00:39<00:04,  4.30s/epoch, loss=0.00483, prev_loss=0.00546]

Training batches on cuda:   0%|          | 0/584 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 13/584 [00:00<00:04, 129.74batch/s]

Training batches on cuda:   5%|▌         | 30/584 [00:00<00:04, 138.35batch/s]

Training batches on cuda:   8%|▊         | 47/584 [00:00<00:03, 145.50batch/s]

Training batches on cuda:  11%|█         | 64/584 [00:00<00:03, 151.36batch/s]

Training batches on cuda:  14%|█▍        | 81/584 [00:00<00:03, 155.57batch/s]

Training batches on cuda:  17%|█▋        | 98/584 [00:00<00:03, 157.86batch/s]

Training batches on cuda:  20%|█▉        | 114/584 [00:00<00:02, 157.97batch/s]

Training batches on cuda:  22%|██▏       | 129/584 [00:00<00:02, 153.98batch/s]

Training batches on cuda:  25%|██▍       | 144/584 [00:00<00:02, 148.71batch/s]

Training batches on cuda:  27%|██▋       | 159/584 [00:01<00:02, 142.54batch/s]

Training batches on cuda:  30%|██▉       | 174/584 [00:01<00:02, 142.80batch/s]

Training batches on cuda:  32%|███▏      | 189/584 [00:01<00:02, 143.94batch/s]

Training batches on cuda:  35%|███▍      | 204/584 [00:01<00:02, 132.88batch/s]

Training batches on cuda:  38%|███▊      | 220/584 [00:01<00:02, 139.45batch/s]

Training batches on cuda:  40%|████      | 235/584 [00:01<00:02, 140.19batch/s]

Training batches on cuda:  43%|████▎     | 250/584 [00:01<00:02, 142.68batch/s]

Training batches on cuda:  46%|████▌     | 266/584 [00:01<00:02, 145.69batch/s]

Training batches on cuda:  48%|████▊     | 282/584 [00:01<00:02, 147.33batch/s]

Training batches on cuda:  51%|█████     | 298/584 [00:02<00:01, 150.14batch/s]

Training batches on cuda:  54%|█████▍    | 314/584 [00:02<00:01, 150.31batch/s]

Training batches on cuda:  57%|█████▋    | 330/584 [00:02<00:01, 151.03batch/s]

Training batches on cuda:  59%|█████▉    | 346/584 [00:02<00:01, 150.87batch/s]

Training batches on cuda:  62%|██████▏   | 362/584 [00:02<00:01, 149.91batch/s]

Training batches on cuda:  65%|██████▍   | 378/584 [00:02<00:01, 139.20batch/s]

Training batches on cuda:  67%|██████▋   | 393/584 [00:02<00:01, 130.84batch/s]

Training batches on cuda:  70%|██████▉   | 407/584 [00:02<00:01, 128.36batch/s]

Training batches on cuda:  72%|███████▏  | 420/584 [00:02<00:01, 125.67batch/s]

Training batches on cuda:  74%|███████▍  | 434/584 [00:03<00:01, 127.90batch/s]

Training batches on cuda:  77%|███████▋  | 449/584 [00:03<00:01, 132.38batch/s]

Training batches on cuda:  79%|███████▉  | 463/584 [00:03<00:00, 129.78batch/s]

Training batches on cuda:  82%|████████▏ | 478/584 [00:03<00:00, 132.74batch/s]

Training batches on cuda:  84%|████████▍ | 493/584 [00:03<00:00, 136.65batch/s]

Training batches on cuda:  87%|████████▋ | 508/584 [00:03<00:00, 140.25batch/s]

Training batches on cuda:  90%|████████▉ | 523/584 [00:03<00:00, 141.52batch/s]

Training batches on cuda:  92%|█████████▏| 538/584 [00:03<00:00, 138.23batch/s]

Training batches on cuda:  95%|█████████▍| 552/584 [00:03<00:00, 136.01batch/s]

Training batches on cuda:  97%|█████████▋| 566/584 [00:03<00:00, 130.12batch/s]

Training batches on cuda:  99%|█████████▉| 580/584 [00:04<00:00, 124.93batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:43<00:04,  4.30s/epoch, loss=0.00371, prev_loss=0.00483]2021-09-16:17:46:02,549 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:46:03,589 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:46:18,509 INFO     [evaluator.py:657] Evaluation took 14.92s seconds
2021-09-16:17:46:18,663 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00,  9.15s/epoch, loss=0.00371, prev_loss=0.00483]
Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00,  5.95s/epoch, loss=0.00371, prev_loss=0.00483]
2021-09-16:17:46:18,663 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp2kmtguac'
2021-09-16:17:46:18,737 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2kmtguac' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.62ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.65ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:46:33,544 INFO     [evaluator.py:657] Evaluation took 14.79s seconds
[I 2021-09-16 17:46:33,550] Trial 13 finished with value: 0.8570932931764637 and parameters: {'loss.margin': 3, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.02444643564788449, 'optimizer.lr': 0.032226302028931335, 'negative_sampler.num_negs_per_pos': 8, 'training.batch_size': 532}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:46:33,572 WARNING  [api.py:823] No random seed is specified. Setting to 4104506862.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 7/374 [00:00<00:05, 64.58batch/s]

Training batches on cuda:   5%|▌         | 19/374 [00:00<00:04, 74.19batch/s]

Training batches on cuda:   8%|▊         | 31/374 [00:00<00:04, 83.59batch/s]

Training batches on cuda:  11%|█▏        | 43/374 [00:00<00:03, 91.81batch/s]

Training batches on cuda:  15%|█▍        | 55/374 [00:00<00:03, 98.36batch/s]

Training batches on cuda:  18%|█▊        | 67/374 [00:00<00:02, 102.44batch/s]

Training batches on cuda:  21%|██        | 78/374 [00:00<00:02, 103.98batch/s]

Training batches on cuda:  24%|██▍       | 90/374 [00:00<00:02, 106.87batch/s]

Training batches on cuda:  27%|██▋       | 101/374 [00:00<00:02, 98.70batch/s]

Training batches on cuda:  30%|███       | 113/374 [00:01<00:02, 104.05batch/s]

Training batches on cuda:  33%|███▎      | 124/374 [00:01<00:02, 104.78batch/s]

Training batches on cuda:  36%|███▌      | 135/374 [00:01<00:02, 103.83batch/s]

Training batches on cuda:  39%|███▉      | 146/374 [00:01<00:02, 100.17batch/s]

Training batches on cuda:  42%|████▏     | 157/374 [00:01<00:02, 100.65batch/s]

Training batches on cuda:  45%|████▍     | 168/374 [00:01<00:02, 97.88batch/s] 

Training batches on cuda:  48%|████▊     | 178/374 [00:01<00:02, 96.82batch/s]

Training batches on cuda:  51%|█████     | 190/374 [00:01<00:01, 102.55batch/s]

Training batches on cuda:  54%|█████▎    | 201/374 [00:01<00:01, 98.69batch/s] 

Training batches on cuda:  57%|█████▋    | 212/374 [00:02<00:01, 99.97batch/s]

Training batches on cuda:  60%|█████▉    | 224/374 [00:02<00:01, 105.13batch/s]

Training batches on cuda:  63%|██████▎   | 237/374 [00:02<00:01, 110.12batch/s]

Training batches on cuda:  67%|██████▋   | 249/374 [00:02<00:01, 111.58batch/s]

Training batches on cuda:  70%|███████   | 262/374 [00:02<00:00, 114.92batch/s]

Training batches on cuda:  74%|███████▎  | 275/374 [00:02<00:00, 117.17batch/s]

Training batches on cuda:  77%|███████▋  | 287/374 [00:02<00:00, 114.88batch/s]

Training batches on cuda:  80%|███████▉  | 299/374 [00:02<00:00, 110.62batch/s]

Training batches on cuda:  83%|████████▎ | 311/374 [00:02<00:00, 103.20batch/s]

Training batches on cuda:  86%|████████▌ | 322/374 [00:03<00:00, 95.36batch/s] 

Training batches on cuda:  89%|████████▉ | 333/374 [00:03<00:00, 98.07batch/s]

Training batches on cuda:  93%|█████████▎| 346/374 [00:03<00:00, 104.80batch/s]

Training batches on cuda:  95%|█████████▌| 357/374 [00:03<00:00, 105.10batch/s]

Training batches on cuda:  99%|█████████▊| 369/374 [00:03<00:00, 107.33batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.00369, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:34,  3.83s/epoch, loss=0.00369, prev_loss=nan]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 7/374 [00:00<00:05, 67.43batch/s]

Training batches on cuda:   5%|▌         | 20/374 [00:00<00:04, 78.13batch/s]

Training batches on cuda:   9%|▊         | 32/374 [00:00<00:03, 87.21batch/s]

Training batches on cuda:  12%|█▏        | 44/374 [00:00<00:03, 93.41batch/s]

Training batches on cuda:  14%|█▍        | 54/374 [00:00<00:03, 94.42batch/s]

Training batches on cuda:  18%|█▊        | 67/374 [00:00<00:03, 101.52batch/s]

Training batches on cuda:  21%|██        | 79/374 [00:00<00:02, 105.36batch/s]

Training batches on cuda:  24%|██▍       | 90/374 [00:00<00:02, 106.18batch/s]

Training batches on cuda:  27%|██▋       | 102/374 [00:00<00:02, 109.18batch/s]

Training batches on cuda:  30%|███       | 114/374 [00:01<00:02, 109.24batch/s]

Training batches on cuda:  33%|███▎      | 125/374 [00:01<00:02, 107.72batch/s]

Training batches on cuda:  37%|███▋      | 137/374 [00:01<00:02, 110.94batch/s]

Training batches on cuda:  40%|███▉      | 149/374 [00:01<00:02, 108.85batch/s]

Training batches on cuda:  43%|████▎     | 160/374 [00:01<00:02, 104.08batch/s]

Training batches on cuda:  46%|████▌     | 171/374 [00:01<00:01, 104.47batch/s]

Training batches on cuda:  49%|████▉     | 184/374 [00:01<00:01, 109.13batch/s]

Training batches on cuda:  52%|█████▏    | 195/374 [00:01<00:01, 103.86batch/s]

Training batches on cuda:  56%|█████▌    | 208/374 [00:01<00:01, 108.95batch/s]

Training batches on cuda:  59%|█████▉    | 220/374 [00:02<00:01, 109.65batch/s]

Training batches on cuda:  62%|██████▏   | 232/374 [00:02<00:01, 96.21batch/s] 

Training batches on cuda:  65%|██████▍   | 243/374 [00:02<00:01, 98.57batch/s]

Training batches on cuda:  68%|██████▊   | 254/374 [00:02<00:01, 101.04batch/s]

Training batches on cuda:  71%|███████   | 266/374 [00:02<00:01, 104.78batch/s]

Training batches on cuda:  74%|███████▍  | 278/374 [00:02<00:00, 108.57batch/s]

Training batches on cuda:  78%|███████▊  | 290/374 [00:02<00:00, 111.51batch/s]

Training batches on cuda:  81%|████████  | 302/374 [00:02<00:00, 112.23batch/s]

Training batches on cuda:  84%|████████▍ | 314/374 [00:02<00:00, 109.40batch/s]

Training batches on cuda:  87%|████████▋ | 327/374 [00:03<00:00, 113.21batch/s]

Training batches on cuda:  91%|█████████ | 339/374 [00:03<00:00, 115.11batch/s]

Training batches on cuda:  94%|█████████▍| 352/374 [00:03<00:00, 117.09batch/s]

Training batches on cuda:  97%|█████████▋| 364/374 [00:03<00:00, 116.31batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:07<00:34,  3.83s/epoch, loss=0.00368, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:30,  3.80s/epoch, loss=0.00368, prev_loss=nan]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/374 [00:00<00:04, 73.77batch/s]

Training batches on cuda:   5%|▌         | 20/374 [00:00<00:04, 82.17batch/s]

Training batches on cuda:   9%|▊         | 32/374 [00:00<00:03, 89.92batch/s]

Training batches on cuda:  11%|█         | 42/374 [00:00<00:03, 92.52batch/s]

Training batches on cuda:  14%|█▍        | 53/374 [00:00<00:03, 95.05batch/s]

Training batches on cuda:  17%|█▋        | 63/374 [00:00<00:03, 94.79batch/s]

Training batches on cuda:  20%|██        | 76/374 [00:00<00:02, 101.25batch/s]

Training batches on cuda:  24%|██▍       | 89/374 [00:00<00:02, 106.56batch/s]

Training batches on cuda:  27%|██▋       | 102/374 [00:00<00:02, 111.06batch/s]

Training batches on cuda:  30%|███       | 114/374 [00:01<00:02, 112.84batch/s]

Training batches on cuda:  34%|███▎      | 126/374 [00:01<00:02, 113.53batch/s]

Training batches on cuda:  37%|███▋      | 139/374 [00:01<00:02, 115.97batch/s]

Training batches on cuda:  41%|████      | 152/374 [00:01<00:01, 119.10batch/s]

Training batches on cuda:  44%|████▍     | 165/374 [00:01<00:01, 120.59batch/s]

Training batches on cuda:  48%|████▊     | 178/374 [00:01<00:01, 120.94batch/s]

Training batches on cuda:  51%|█████     | 191/374 [00:01<00:01, 120.17batch/s]

Training batches on cuda:  55%|█████▍    | 204/374 [00:01<00:01, 119.79batch/s]

Training batches on cuda:  58%|█████▊    | 216/374 [00:01<00:01, 118.74batch/s]

Training batches on cuda:  61%|██████    | 228/374 [00:02<00:01, 110.00batch/s]

Training batches on cuda:  64%|██████▍   | 241/374 [00:02<00:01, 114.10batch/s]

Training batches on cuda:  68%|██████▊   | 254/374 [00:02<00:01, 117.10batch/s]

Training batches on cuda:  71%|███████▏  | 267/374 [00:02<00:00, 118.56batch/s]

Training batches on cuda:  75%|███████▍  | 279/374 [00:02<00:00, 113.93batch/s]

Training batches on cuda:  78%|███████▊  | 291/374 [00:02<00:00, 115.37batch/s]

Training batches on cuda:  81%|████████  | 303/374 [00:02<00:00, 115.52batch/s]

Training batches on cuda:  84%|████████▍ | 315/374 [00:02<00:00, 115.49batch/s]

Training batches on cuda:  88%|████████▊ | 328/374 [00:02<00:00, 117.36batch/s]

Training batches on cuda:  91%|█████████ | 340/374 [00:02<00:00, 115.86batch/s]

Training batches on cuda:  94%|█████████▍| 352/374 [00:03<00:00, 110.42batch/s]

Training batches on cuda:  97%|█████████▋| 364/374 [00:03<00:00, 104.25batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:30,  3.80s/epoch, loss=0.00368, prev_loss=0.00368]
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:25,  3.71s/epoch, loss=0.00368, prev_loss=0.00368]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/374 [00:00<00:05, 72.54batch/s]

Training batches on cuda:   5%|▌         | 20/374 [00:00<00:04, 81.88batch/s]

Training batches on cuda:   9%|▊         | 32/374 [00:00<00:03, 89.01batch/s]

Training batches on cuda:  11%|█         | 42/374 [00:00<00:03, 91.89batch/s]

Training batches on cuda:  14%|█▎        | 51/374 [00:00<00:03, 90.21batch/s]

Training batches on cuda:  16%|█▌        | 60/374 [00:00<00:03, 89.68batch/s]

Training batches on cuda:  19%|█▊        | 70/374 [00:00<00:03, 90.16batch/s]

Training batches on cuda:  22%|██▏       | 82/374 [00:00<00:03, 95.60batch/s]

Training batches on cuda:  25%|██▌       | 95/374 [00:00<00:02, 101.97batch/s]

Training batches on cuda:  28%|██▊       | 106/374 [00:01<00:02, 103.45batch/s]

Training batches on cuda:  31%|███▏      | 117/374 [00:01<00:02, 104.16batch/s]

Training batches on cuda:  34%|███▍      | 128/374 [00:01<00:02, 103.21batch/s]

Training batches on cuda:  37%|███▋      | 139/374 [00:01<00:02, 104.46batch/s]

Training batches on cuda:  40%|████      | 150/374 [00:01<00:02, 101.26batch/s]

Training batches on cuda:  43%|████▎     | 161/374 [00:01<00:02, 99.89batch/s] 

Training batches on cuda:  46%|████▌     | 172/374 [00:01<00:02, 98.07batch/s]

Training batches on cuda:  49%|████▊     | 182/374 [00:01<00:02, 93.56batch/s]

Training batches on cuda:  51%|█████▏    | 192/374 [00:01<00:02, 89.12batch/s]

Training batches on cuda:  54%|█████▍    | 203/374 [00:02<00:01, 92.70batch/s]

Training batches on cuda:  57%|█████▋    | 213/374 [00:02<00:01, 94.04batch/s]

Training batches on cuda:  60%|██████    | 225/374 [00:02<00:01, 98.76batch/s]

Training batches on cuda:  63%|██████▎   | 237/374 [00:02<00:01, 104.20batch/s]

Training batches on cuda:  66%|██████▋   | 248/374 [00:02<00:01, 104.82batch/s]

Training batches on cuda:  70%|██████▉   | 260/374 [00:02<00:01, 108.42batch/s]

Training batches on cuda:  73%|███████▎  | 272/374 [00:02<00:00, 109.11batch/s]

Training batches on cuda:  76%|███████▌  | 283/374 [00:02<00:00, 107.97batch/s]

Training batches on cuda:  79%|███████▊  | 294/374 [00:02<00:00, 105.22batch/s]

Training batches on cuda:  82%|████████▏ | 306/374 [00:03<00:00, 108.23batch/s]

Training batches on cuda:  85%|████████▍ | 317/374 [00:03<00:00, 108.54batch/s]

Training batches on cuda:  88%|████████▊ | 329/374 [00:03<00:00, 109.00batch/s]

Training batches on cuda:  91%|█████████ | 340/374 [00:03<00:00, 105.01batch/s]

Training batches on cuda:  94%|█████████▍| 351/374 [00:03<00:00, 100.76batch/s]

Training batches on cuda:  97%|█████████▋| 362/374 [00:03<00:00, 101.80batch/s]

Training batches on cuda: 100%|█████████▉| 373/374 [00:03<00:00, 99.58batch/s] 

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:25,  3.71s/epoch, loss=0.00368, prev_loss=0.00368]
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:22,  3.79s/epoch, loss=0.00368, prev_loss=0.00368]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 9/374 [00:00<00:04, 83.57batch/s]

Training batches on cuda:   6%|▌         | 21/374 [00:00<00:03, 90.76batch/s]

Training batches on cuda:   8%|▊         | 31/374 [00:00<00:03, 92.61batch/s]

Training batches on cuda:  11%|█▏        | 43/374 [00:00<00:03, 98.96batch/s]

Training batches on cuda:  15%|█▍        | 55/374 [00:00<00:03, 101.97batch/s]

Training batches on cuda:  18%|█▊        | 67/374 [00:00<00:02, 104.16batch/s]

Training batches on cuda:  21%|██        | 77/374 [00:00<00:02, 100.93batch/s]

Training batches on cuda:  23%|██▎       | 87/374 [00:00<00:03, 92.41batch/s] 

Training batches on cuda:  26%|██▌       | 96/374 [00:00<00:03, 87.29batch/s]

Training batches on cuda:  28%|██▊       | 106/374 [00:01<00:03, 89.18batch/s]

Training batches on cuda:  31%|███       | 115/374 [00:01<00:02, 87.87batch/s]

Training batches on cuda:  34%|███▎      | 126/374 [00:01<00:02, 93.50batch/s]

Training batches on cuda:  37%|███▋      | 138/374 [00:01<00:02, 98.53batch/s]

Training batches on cuda:  40%|███▉      | 149/374 [00:01<00:02, 100.05batch/s]

Training batches on cuda:  43%|████▎     | 161/374 [00:01<00:02, 103.02batch/s]

Training batches on cuda:  46%|████▋     | 173/374 [00:01<00:01, 105.66batch/s]

Training batches on cuda:  49%|████▉     | 184/374 [00:01<00:01, 104.05batch/s]

Training batches on cuda:  52%|█████▏    | 196/374 [00:01<00:01, 107.80batch/s]

Training batches on cuda:  56%|█████▌    | 208/374 [00:02<00:01, 110.50batch/s]

Training batches on cuda:  59%|█████▉    | 220/374 [00:02<00:01, 105.62batch/s]

Training batches on cuda:  62%|██████▏   | 231/374 [00:02<00:01, 103.79batch/s]

Training batches on cuda:  65%|██████▍   | 242/374 [00:02<00:01, 100.44batch/s]

Training batches on cuda:  68%|██████▊   | 253/374 [00:02<00:01, 100.90batch/s]

Training batches on cuda:  71%|███████   | 264/374 [00:02<00:01, 102.17batch/s]

Training batches on cuda:  74%|███████▎  | 275/374 [00:02<00:00, 103.39batch/s]

Training batches on cuda:  76%|███████▋  | 286/374 [00:02<00:00, 93.76batch/s] 

Training batches on cuda:  80%|███████▉  | 298/374 [00:02<00:00, 99.03batch/s]

Training batches on cuda:  83%|████████▎ | 309/374 [00:03<00:00, 95.41batch/s]

Training batches on cuda:  86%|████████▌ | 320/374 [00:03<00:00, 96.56batch/s]

Training batches on cuda:  88%|████████▊ | 330/374 [00:03<00:00, 94.81batch/s]

Training batches on cuda:  91%|█████████ | 340/374 [00:03<00:00, 94.97batch/s]

Training batches on cuda:  94%|█████████▎| 350/374 [00:03<00:00, 95.32batch/s]

Training batches on cuda:  97%|█████████▋| 361/374 [00:03<00:00, 98.09batch/s]

Training batches on cuda:  99%|█████████▉| 372/374 [00:03<00:00, 100.20batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:18<00:22,  3.79s/epoch, loss=0.00368, prev_loss=0.00368]
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:19,  3.84s/epoch, loss=0.00368, prev_loss=0.00368]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 4/374 [00:00<00:09, 39.95batch/s]

Training batches on cuda:   3%|▎         | 12/374 [00:00<00:07, 46.78batch/s]

Training batches on cuda:   6%|▌         | 22/374 [00:00<00:06, 55.50batch/s]

Training batches on cuda:   8%|▊         | 30/374 [00:00<00:05, 60.88batch/s]

Training batches on cuda:  10%|█         | 39/374 [00:00<00:05, 66.95batch/s]

Training batches on cuda:  13%|█▎        | 50/374 [00:00<00:04, 75.04batch/s]

Training batches on cuda:  17%|█▋        | 62/374 [00:00<00:03, 83.66batch/s]

Training batches on cuda:  20%|█▉        | 73/374 [00:00<00:03, 89.78batch/s]

Training batches on cuda:  22%|██▏       | 83/374 [00:00<00:03, 87.94batch/s]

Training batches on cuda:  26%|██▌       | 96/374 [00:01<00:02, 97.15batch/s]

Training batches on cuda:  29%|██▉       | 110/374 [00:01<00:02, 105.24batch/s]

Training batches on cuda:  33%|███▎      | 124/374 [00:01<00:02, 113.18batch/s]

Training batches on cuda:  37%|███▋      | 138/374 [00:01<00:01, 119.34batch/s]

Training batches on cuda:  41%|████      | 152/374 [00:01<00:01, 123.82batch/s]

Training batches on cuda:  44%|████▍     | 165/374 [00:01<00:01, 125.14batch/s]

Training batches on cuda:  48%|████▊     | 178/374 [00:01<00:01, 118.69batch/s]

Training batches on cuda:  51%|█████     | 191/374 [00:01<00:01, 115.07batch/s]

Training batches on cuda:  54%|█████▍    | 203/374 [00:01<00:01, 107.21batch/s]

Training batches on cuda:  57%|█████▋    | 214/374 [00:02<00:01, 97.56batch/s] 

Training batches on cuda:  61%|██████    | 227/374 [00:02<00:01, 102.79batch/s]

Training batches on cuda:  64%|██████▍   | 240/374 [00:02<00:01, 107.33batch/s]

Training batches on cuda:  67%|██████▋   | 252/374 [00:02<00:01, 104.92batch/s]

Training batches on cuda:  71%|███████   | 264/374 [00:02<00:01, 108.24batch/s]

Training batches on cuda:  74%|███████▍  | 276/374 [00:02<00:00, 109.10batch/s]

Training batches on cuda:  77%|███████▋  | 288/374 [00:02<00:00, 103.09batch/s]

Training batches on cuda:  80%|███████▉  | 299/374 [00:02<00:00, 104.94batch/s]

Training batches on cuda:  83%|████████▎ | 310/374 [00:02<00:00, 105.02batch/s]

Training batches on cuda:  86%|████████▌ | 321/374 [00:03<00:00, 103.12batch/s]

Training batches on cuda:  89%|████████▉ | 332/374 [00:03<00:00, 98.63batch/s] 

Training batches on cuda:  92%|█████████▏| 344/374 [00:03<00:00, 103.65batch/s]

Training batches on cuda:  95%|█████████▌| 356/374 [00:03<00:00, 107.08batch/s]

Training batches on cuda:  98%|█████████▊| 368/374 [00:03<00:00, 110.36batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:19,  3.84s/epoch, loss=0.00368, prev_loss=0.00368]
Training epochs on cuda:  60%|██████    | 6/10 [00:22<00:15,  3.86s/epoch, loss=0.00368, prev_loss=0.00368]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 7/374 [00:00<00:05, 68.89batch/s]

Training batches on cuda:   5%|▍         | 18/374 [00:00<00:04, 76.28batch/s]

Training batches on cuda:   8%|▊         | 30/374 [00:00<00:04, 85.34batch/s]

Training batches on cuda:  11%|█         | 41/374 [00:00<00:03, 91.29batch/s]

Training batches on cuda:  14%|█▍        | 54/374 [00:00<00:03, 98.59batch/s]

Training batches on cuda:  18%|█▊        | 66/374 [00:00<00:02, 102.68batch/s]

Training batches on cuda:  21%|██        | 78/374 [00:00<00:02, 106.46batch/s]

Training batches on cuda:  24%|██▍       | 89/374 [00:00<00:02, 107.44batch/s]

Training batches on cuda:  27%|██▋       | 101/374 [00:00<00:02, 108.71batch/s]

Training batches on cuda:  30%|███       | 114/374 [00:01<00:02, 111.88batch/s]

Training batches on cuda:  34%|███▍      | 128/374 [00:01<00:02, 117.34batch/s]

Training batches on cuda:  37%|███▋      | 140/374 [00:01<00:02, 116.93batch/s]

Training batches on cuda:  41%|████      | 152/374 [00:01<00:02, 103.85batch/s]

Training batches on cuda:  44%|████▍     | 164/374 [00:01<00:01, 107.64batch/s]

Training batches on cuda:  47%|████▋     | 175/374 [00:01<00:02, 99.20batch/s] 

Training batches on cuda:  50%|████▉     | 186/374 [00:01<00:01, 100.87batch/s]

Training batches on cuda:  53%|█████▎    | 197/374 [00:01<00:01, 101.62batch/s]

Training batches on cuda:  56%|█████▌    | 209/374 [00:01<00:01, 105.52batch/s]

Training batches on cuda:  59%|█████▉    | 221/374 [00:02<00:01, 107.28batch/s]

Training batches on cuda:  62%|██████▏   | 233/374 [00:02<00:01, 110.37batch/s]

Training batches on cuda:  66%|██████▌   | 245/374 [00:02<00:01, 110.85batch/s]

Training batches on cuda:  69%|██████▊   | 257/374 [00:02<00:01, 111.29batch/s]

Training batches on cuda:  72%|███████▏  | 269/374 [00:02<00:00, 110.70batch/s]

Training batches on cuda:  75%|███████▌  | 281/374 [00:02<00:00, 104.00batch/s]

Training batches on cuda:  78%|███████▊  | 293/374 [00:02<00:00, 107.46batch/s]

Training batches on cuda:  82%|████████▏ | 305/374 [00:02<00:00, 110.76batch/s]

Training batches on cuda:  85%|████████▍ | 317/374 [00:02<00:00, 112.56batch/s]

Training batches on cuda:  88%|████████▊ | 329/374 [00:03<00:00, 113.64batch/s]

Training batches on cuda:  91%|█████████ | 341/374 [00:03<00:00, 111.88batch/s]

Training batches on cuda:  94%|█████████▍| 353/374 [00:03<00:00, 112.84batch/s]

Training batches on cuda:  98%|█████████▊| 365/374 [00:03<00:00, 112.50batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:26<00:15,  3.86s/epoch, loss=0.00368, prev_loss=0.00368]
Training epochs on cuda:  70%|███████   | 7/10 [00:26<00:11,  3.80s/epoch, loss=0.00368, prev_loss=0.00368]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 6/374 [00:00<00:06, 57.92batch/s]

Training batches on cuda:   4%|▍         | 16/374 [00:00<00:05, 65.83batch/s]

Training batches on cuda:   7%|▋         | 28/374 [00:00<00:04, 75.17batch/s]

Training batches on cuda:  11%|█         | 40/374 [00:00<00:03, 83.98batch/s]

Training batches on cuda:  14%|█▍        | 52/374 [00:00<00:03, 91.86batch/s]

Training batches on cuda:  17%|█▋        | 63/374 [00:00<00:03, 96.41batch/s]

Training batches on cuda:  20%|██        | 75/374 [00:00<00:02, 101.32batch/s]

Training batches on cuda:  23%|██▎       | 86/374 [00:00<00:02, 103.08batch/s]

Training batches on cuda:  26%|██▌       | 97/374 [00:00<00:02, 104.80batch/s]

Training batches on cuda:  29%|██▉       | 109/374 [00:01<00:02, 106.09batch/s]

Training batches on cuda:  32%|███▏      | 120/374 [00:01<00:02, 97.52batch/s] 

Training batches on cuda:  35%|███▌      | 132/374 [00:01<00:02, 100.48batch/s]

Training batches on cuda:  39%|███▊      | 144/374 [00:01<00:02, 104.33batch/s]

Training batches on cuda:  42%|████▏     | 156/374 [00:01<00:02, 107.00batch/s]

Training batches on cuda:  45%|████▍     | 168/374 [00:01<00:01, 109.34batch/s]

Training batches on cuda:  48%|████▊     | 180/374 [00:01<00:01, 111.74batch/s]

Training batches on cuda:  51%|█████▏    | 192/374 [00:01<00:01, 112.76batch/s]

Training batches on cuda:  55%|█████▍    | 204/374 [00:01<00:01, 114.22batch/s]

Training batches on cuda:  58%|█████▊    | 216/374 [00:02<00:01, 114.82batch/s]

Training batches on cuda:  61%|██████    | 228/374 [00:02<00:01, 114.69batch/s]

Training batches on cuda:  64%|██████▍   | 240/374 [00:02<00:01, 113.25batch/s]

Training batches on cuda:  67%|██████▋   | 252/374 [00:02<00:01, 112.02batch/s]

Training batches on cuda:  71%|███████   | 264/374 [00:02<00:00, 113.06batch/s]

Training batches on cuda:  74%|███████▍  | 276/374 [00:02<00:00, 106.61batch/s]

Training batches on cuda:  77%|███████▋  | 288/374 [00:02<00:00, 108.53batch/s]

Training batches on cuda:  80%|████████  | 300/374 [00:02<00:00, 110.93batch/s]

Training batches on cuda:  83%|████████▎ | 312/374 [00:02<00:00, 112.53batch/s]

Training batches on cuda:  87%|████████▋ | 324/374 [00:02<00:00, 113.51batch/s]

Training batches on cuda:  90%|█████████ | 337/374 [00:03<00:00, 117.86batch/s]

Training batches on cuda:  94%|█████████▍| 351/374 [00:03<00:00, 121.86batch/s]

Training batches on cuda:  97%|█████████▋| 364/374 [00:03<00:00, 122.69batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:30<00:11,  3.80s/epoch, loss=0.00368, prev_loss=0.00368]
Training epochs on cuda:  80%|████████  | 8/10 [00:30<00:07,  3.75s/epoch, loss=0.00368, prev_loss=0.00368]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 11/374 [00:00<00:03, 102.85batch/s]

Training batches on cuda:   7%|▋         | 25/374 [00:00<00:03, 110.75batch/s]

Training batches on cuda:  11%|█         | 40/374 [00:00<00:02, 118.42batch/s]

Training batches on cuda:  15%|█▍        | 55/374 [00:00<00:02, 124.56batch/s]

Training batches on cuda:  18%|█▊        | 69/374 [00:00<00:02, 127.91batch/s]

Training batches on cuda:  22%|██▏       | 83/374 [00:00<00:02, 130.79batch/s]

Training batches on cuda:  26%|██▌       | 96/374 [00:00<00:02, 121.75batch/s]

Training batches on cuda:  29%|██▉       | 108/374 [00:00<00:02, 111.91batch/s]

Training batches on cuda:  32%|███▏      | 119/374 [00:00<00:02, 105.48batch/s]

Training batches on cuda:  35%|███▌      | 131/374 [00:01<00:02, 107.23batch/s]

Training batches on cuda:  38%|███▊      | 143/374 [00:01<00:02, 110.35batch/s]

Training batches on cuda:  41%|████▏     | 155/374 [00:01<00:02, 107.84batch/s]

Training batches on cuda:  45%|████▍     | 167/374 [00:01<00:01, 110.58batch/s]

Training batches on cuda:  48%|████▊     | 179/374 [00:01<00:01, 109.98batch/s]

Training batches on cuda:  51%|█████     | 191/374 [00:01<00:01, 107.10batch/s]

Training batches on cuda:  54%|█████▍    | 203/374 [00:01<00:01, 106.89batch/s]

Training batches on cuda:  58%|█████▊    | 216/374 [00:01<00:01, 112.02batch/s]

Training batches on cuda:  61%|██████    | 228/374 [00:01<00:01, 113.86batch/s]

Training batches on cuda:  64%|██████▍   | 241/374 [00:02<00:01, 117.44batch/s]

Training batches on cuda:  68%|██████▊   | 253/374 [00:02<00:01, 114.24batch/s]

Training batches on cuda:  71%|███████   | 265/374 [00:02<00:01, 102.97batch/s]

Training batches on cuda:  74%|███████▍  | 276/374 [00:02<00:00, 100.15batch/s]

Training batches on cuda:  77%|███████▋  | 287/374 [00:02<00:00, 99.13batch/s] 

Training batches on cuda:  80%|███████▉  | 298/374 [00:02<00:00, 100.59batch/s]

Training batches on cuda:  83%|████████▎ | 309/374 [00:02<00:00, 95.75batch/s] 

Training batches on cuda:  85%|████████▌ | 319/374 [00:02<00:00, 95.63batch/s]

Training batches on cuda:  88%|████████▊ | 329/374 [00:02<00:00, 96.43batch/s]

Training batches on cuda:  91%|█████████ | 339/374 [00:03<00:00, 97.27batch/s]

Training batches on cuda:  94%|█████████▎| 350/374 [00:03<00:00, 98.26batch/s]

Training batches on cuda:  97%|█████████▋| 361/374 [00:03<00:00, 100.21batch/s]

Training batches on cuda:  99%|█████████▉| 372/374 [00:03<00:00, 101.21batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:33<00:07,  3.75s/epoch, loss=0.00368, prev_loss=0.00368]
Training epochs on cuda:  90%|█████████ | 9/10 [00:33<00:03,  3.74s/epoch, loss=0.00368, prev_loss=0.00368]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 6/374 [00:00<00:06, 55.83batch/s]

Training batches on cuda:   5%|▍         | 17/374 [00:00<00:05, 64.37batch/s]

Training batches on cuda:   7%|▋         | 28/374 [00:00<00:04, 73.30batch/s]

Training batches on cuda:  10%|█         | 38/374 [00:00<00:04, 78.06batch/s]

Training batches on cuda:  13%|█▎        | 50/374 [00:00<00:03, 86.76batch/s]

Training batches on cuda:  17%|█▋        | 62/374 [00:00<00:03, 94.27batch/s]

Training batches on cuda:  20%|█▉        | 74/374 [00:00<00:03, 99.50batch/s]

Training batches on cuda:  23%|██▎       | 87/374 [00:00<00:02, 106.46batch/s]

Training batches on cuda:  26%|██▋       | 99/374 [00:00<00:02, 109.75batch/s]

Training batches on cuda:  30%|██▉       | 111/374 [00:01<00:02, 111.41batch/s]

Training batches on cuda:  33%|███▎      | 123/374 [00:01<00:02, 108.75batch/s]

Training batches on cuda:  36%|███▌      | 134/374 [00:01<00:02, 106.02batch/s]

Training batches on cuda:  39%|███▉      | 147/374 [00:01<00:02, 110.72batch/s]

Training batches on cuda:  43%|████▎     | 159/374 [00:01<00:01, 109.89batch/s]

Training batches on cuda:  46%|████▌     | 171/374 [00:01<00:01, 106.98batch/s]

Training batches on cuda:  49%|████▊     | 182/374 [00:01<00:01, 107.70batch/s]

Training batches on cuda:  52%|█████▏    | 193/374 [00:01<00:01, 104.62batch/s]

Training batches on cuda:  55%|█████▍    | 204/374 [00:01<00:01, 101.43batch/s]

Training batches on cuda:  57%|█████▋    | 215/374 [00:02<00:01, 100.35batch/s]

Training batches on cuda:  60%|██████    | 226/374 [00:02<00:01, 100.01batch/s]

Training batches on cuda:  63%|██████▎   | 237/374 [00:02<00:01, 92.20batch/s] 

Training batches on cuda:  66%|██████▋   | 248/374 [00:02<00:01, 94.84batch/s]

Training batches on cuda:  70%|██████▉   | 260/374 [00:02<00:01, 99.08batch/s]

Training batches on cuda:  73%|███████▎  | 272/374 [00:02<00:00, 103.52batch/s]

Training batches on cuda:  76%|███████▌  | 283/374 [00:02<00:00, 100.62batch/s]

Training batches on cuda:  79%|███████▊  | 294/374 [00:02<00:00, 99.26batch/s] 

Training batches on cuda:  82%|████████▏ | 306/374 [00:02<00:00, 102.28batch/s]

Training batches on cuda:  85%|████████▌ | 318/374 [00:03<00:00, 105.21batch/s]

Training batches on cuda:  88%|████████▊ | 329/374 [00:03<00:00, 101.41batch/s]

Training batches on cuda:  91%|█████████ | 340/374 [00:03<00:00, 100.05batch/s]

Training batches on cuda:  94%|█████████▍| 351/374 [00:03<00:00, 98.07batch/s] 

Training batches on cuda:  97%|█████████▋| 361/374 [00:03<00:00, 97.72batch/s]

Training batches on cuda: 100%|█████████▉| 373/374 [00:03<00:00, 102.66batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:37<00:03,  3.74s/epoch, loss=0.00368, prev_loss=0.00368]2021-09-16:17:47:11,836 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:47:12,897 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:47:27,899 INFO     [evaluator.py:657] Evaluation took 15.00s seconds
2021-09-16:17:47:28,30 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00,  8.65s/epoch, loss=0.00368, prev_loss=0.00368]
Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00,  5.40s/epoch, loss=0.00368, prev_loss=0.00368]
2021-09-16:17:47:28,31 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpdazeuw3e'
2021-09-16:17:47:28,103 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdazeuw3e' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:17:47:42,980 INFO     [evaluator.py:657] Evaluation took 14.85s seconds
[I 2021-09-16 17:47:42,986] Trial 14 finished with value: -0.006964663866187104 and parameters: {'loss.margin': 3, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.1935191965473909, 'optimizer.lr': 0.0445058431805547, 'negative_sampler.num_negs_per_pos': 7, 'training.batch_size': 832}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:47:43,13 WARNING  [api.py:823] No random seed is specified. Setting to 2488369916.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:06, 12.51batch/s]

Training batches on cuda:   5%|▌         | 4/80 [00:00<00:05, 13.40batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:05, 14.10batch/s]

Training batches on cuda:  10%|█         | 8/80 [00:00<00:04, 15.09batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:04, 15.86batch/s]

Training batches on cuda:  15%|█▌        | 12/80 [00:00<00:04, 16.37batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:03, 16.75batch/s]

Training batches on cuda:  20%|██        | 16/80 [00:00<00:03, 17.15batch/s]

Training batches on cuda:  22%|██▎       | 18/80 [00:01<00:03, 17.39batch/s]

Training batches on cuda:  25%|██▌       | 20/80 [00:01<00:03, 17.46batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:01<00:03, 17.44batch/s]

Training batches on cuda:  30%|███       | 24/80 [00:01<00:03, 17.45batch/s]

Training batches on cuda:  32%|███▎      | 26/80 [00:01<00:03, 17.33batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:02, 17.34batch/s]

Training batches on cuda:  38%|███▊      | 30/80 [00:01<00:02, 16.99batch/s]

Training batches on cuda:  40%|████      | 32/80 [00:01<00:02, 16.68batch/s]

Training batches on cuda:  42%|████▎     | 34/80 [00:02<00:02, 16.89batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:02<00:02, 17.21batch/s]

Training batches on cuda:  48%|████▊     | 38/80 [00:02<00:02, 17.51batch/s]

Training batches on cuda:  50%|█████     | 40/80 [00:02<00:02, 17.62batch/s]

Training batches on cuda:  52%|█████▎    | 42/80 [00:02<00:02, 17.59batch/s]

Training batches on cuda:  55%|█████▌    | 44/80 [00:02<00:02, 17.17batch/s]

Training batches on cuda:  57%|█████▊    | 46/80 [00:02<00:01, 17.06batch/s]

Training batches on cuda:  60%|██████    | 48/80 [00:02<00:01, 17.21batch/s]

Training batches on cuda:  62%|██████▎   | 50/80 [00:02<00:01, 17.04batch/s]

Training batches on cuda:  65%|██████▌   | 52/80 [00:03<00:01, 16.58batch/s]

Training batches on cuda:  68%|██████▊   | 54/80 [00:03<00:01, 16.50batch/s]

Training batches on cuda:  70%|███████   | 56/80 [00:03<00:01, 16.54batch/s]

Training batches on cuda:  72%|███████▎  | 58/80 [00:03<00:01, 16.50batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:03<00:01, 16.58batch/s]

Training batches on cuda:  78%|███████▊  | 62/80 [00:03<00:01, 16.58batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:03<00:00, 16.61batch/s]

Training batches on cuda:  82%|████████▎ | 66/80 [00:03<00:00, 16.58batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:04<00:00, 16.60batch/s]

Training batches on cuda:  88%|████████▊ | 70/80 [00:04<00:00, 16.17batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:04<00:00, 16.38batch/s]

Training batches on cuda:  92%|█████████▎| 74/80 [00:04<00:00, 16.69batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:04<00:00, 17.05batch/s]

Training batches on cuda:  98%|█████████▊| 78/80 [00:04<00:00, 16.78batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000779, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:44,  4.98s/epoch, loss=0.000779, prev_loss=nan]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 1/80 [00:00<00:09,  8.23batch/s]

Training batches on cuda:   4%|▍         | 3/80 [00:00<00:07,  9.84batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:06, 11.41batch/s]

Training batches on cuda:   9%|▉         | 7/80 [00:00<00:05, 12.90batch/s]

Training batches on cuda:  11%|█▏        | 9/80 [00:00<00:05, 14.16batch/s]

Training batches on cuda:  14%|█▍        | 11/80 [00:00<00:04, 15.18batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:04, 15.89batch/s]

Training batches on cuda:  19%|█▉        | 15/80 [00:00<00:03, 16.30batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:01<00:03, 16.67batch/s]

Training batches on cuda:  24%|██▍       | 19/80 [00:01<00:03, 16.68batch/s]

Training batches on cuda:  26%|██▋       | 21/80 [00:01<00:03, 16.45batch/s]

Training batches on cuda:  29%|██▉       | 23/80 [00:01<00:03, 16.91batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:01<00:03, 17.03batch/s]

Training batches on cuda:  34%|███▍      | 27/80 [00:01<00:03, 17.24batch/s]

Training batches on cuda:  36%|███▋      | 29/80 [00:01<00:02, 17.14batch/s]

Training batches on cuda:  39%|███▉      | 31/80 [00:01<00:02, 17.42batch/s]

Training batches on cuda:  41%|████▏     | 33/80 [00:01<00:02, 16.41batch/s]

Training batches on cuda:  44%|████▍     | 35/80 [00:02<00:02, 17.04batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:02<00:02, 17.03batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:02<00:02, 17.33batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:02<00:02, 17.54batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:02<00:02, 17.95batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:02<00:01, 18.24batch/s]

Training batches on cuda:  59%|█████▉    | 47/80 [00:02<00:01, 18.30batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:02<00:01, 18.40batch/s]

Training batches on cuda:  64%|██████▍   | 51/80 [00:02<00:01, 17.77batch/s]

Training batches on cuda:  66%|██████▋   | 53/80 [00:03<00:01, 17.55batch/s]

Training batches on cuda:  69%|██████▉   | 55/80 [00:03<00:01, 17.46batch/s]

Training batches on cuda:  71%|███████▏  | 57/80 [00:03<00:01, 17.47batch/s]

Training batches on cuda:  74%|███████▍  | 59/80 [00:03<00:01, 17.25batch/s]

Training batches on cuda:  76%|███████▋  | 61/80 [00:03<00:01, 16.98batch/s]

Training batches on cuda:  79%|███████▉  | 63/80 [00:03<00:00, 17.13batch/s]

Training batches on cuda:  81%|████████▏ | 65/80 [00:03<00:00, 17.31batch/s]

Training batches on cuda:  84%|████████▍ | 67/80 [00:03<00:00, 17.16batch/s]

Training batches on cuda:  86%|████████▋ | 69/80 [00:04<00:00, 17.13batch/s]

Training batches on cuda:  89%|████████▉ | 71/80 [00:04<00:00, 17.44batch/s]

Training batches on cuda:  91%|█████████▏| 73/80 [00:04<00:00, 17.50batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:04<00:00, 17.41batch/s]

Training batches on cuda:  96%|█████████▋| 77/80 [00:04<00:00, 17.53batch/s]

Training batches on cuda:  99%|█████████▉| 79/80 [00:04<00:00, 17.15batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:09<00:44,  4.98s/epoch, loss=0.000774, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:39,  4.96s/epoch, loss=0.000774, prev_loss=nan]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 1/80 [00:00<00:08,  9.52batch/s]

Training batches on cuda:   4%|▍         | 3/80 [00:00<00:06, 11.02batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:06, 12.38batch/s]

Training batches on cuda:   9%|▉         | 7/80 [00:00<00:05, 13.67batch/s]

Training batches on cuda:  11%|█▏        | 9/80 [00:00<00:04, 14.63batch/s]

Training batches on cuda:  14%|█▍        | 11/80 [00:00<00:04, 15.44batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:04, 16.00batch/s]

Training batches on cuda:  19%|█▉        | 15/80 [00:00<00:03, 16.50batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:01<00:03, 16.73batch/s]

Training batches on cuda:  24%|██▍       | 19/80 [00:01<00:03, 16.94batch/s]

Training batches on cuda:  26%|██▋       | 21/80 [00:01<00:03, 17.21batch/s]

Training batches on cuda:  29%|██▉       | 23/80 [00:01<00:03, 17.37batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:01<00:03, 17.60batch/s]

Training batches on cuda:  34%|███▍      | 27/80 [00:01<00:03, 17.62batch/s]

Training batches on cuda:  36%|███▋      | 29/80 [00:01<00:02, 17.60batch/s]

Training batches on cuda:  39%|███▉      | 31/80 [00:01<00:02, 17.57batch/s]

Training batches on cuda:  41%|████▏     | 33/80 [00:01<00:02, 17.36batch/s]

Training batches on cuda:  44%|████▍     | 35/80 [00:02<00:02, 15.78batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:02<00:02, 16.21batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:02<00:02, 16.21batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:02<00:02, 16.43batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:02<00:02, 16.75batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:02<00:02, 16.86batch/s]

Training batches on cuda:  59%|█████▉    | 47/80 [00:02<00:01, 16.85batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:02<00:01, 17.07batch/s]

Training batches on cuda:  64%|██████▍   | 51/80 [00:03<00:01, 17.39batch/s]

Training batches on cuda:  66%|██████▋   | 53/80 [00:03<00:01, 17.56batch/s]

Training batches on cuda:  69%|██████▉   | 55/80 [00:03<00:01, 17.59batch/s]

Training batches on cuda:  71%|███████▏  | 57/80 [00:03<00:01, 17.65batch/s]

Training batches on cuda:  74%|███████▍  | 59/80 [00:03<00:01, 17.46batch/s]

Training batches on cuda:  76%|███████▋  | 61/80 [00:03<00:01, 17.43batch/s]

Training batches on cuda:  79%|███████▉  | 63/80 [00:03<00:00, 17.56batch/s]

Training batches on cuda:  81%|████████▏ | 65/80 [00:03<00:00, 17.34batch/s]

Training batches on cuda:  84%|████████▍ | 67/80 [00:03<00:00, 17.05batch/s]

Training batches on cuda:  86%|████████▋ | 69/80 [00:04<00:00, 16.57batch/s]

Training batches on cuda:  89%|████████▉ | 71/80 [00:04<00:00, 16.81batch/s]

Training batches on cuda:  91%|█████████▏| 73/80 [00:04<00:00, 17.23batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:04<00:00, 17.37batch/s]

Training batches on cuda:  96%|█████████▋| 77/80 [00:04<00:00, 17.54batch/s]

Training batches on cuda:  99%|█████████▉| 79/80 [00:04<00:00, 17.38batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:14<00:39,  4.96s/epoch, loss=0.000774, prev_loss=0.000774]
Training epochs on cuda:  30%|███       | 3/10 [00:14<00:34,  4.93s/epoch, loss=0.000774, prev_loss=0.000774]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 1/80 [00:00<00:08,  8.84batch/s]

Training batches on cuda:   4%|▍         | 3/80 [00:00<00:07, 10.12batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:06, 11.38batch/s]

Training batches on cuda:   9%|▉         | 7/80 [00:00<00:05, 12.73batch/s]

Training batches on cuda:  11%|█▏        | 9/80 [00:00<00:05, 13.13batch/s]

Training batches on cuda:  14%|█▍        | 11/80 [00:00<00:04, 13.84batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:04, 14.79batch/s]

Training batches on cuda:  19%|█▉        | 15/80 [00:00<00:04, 15.37batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:01<00:03, 15.95batch/s]

Training batches on cuda:  24%|██▍       | 19/80 [00:01<00:03, 16.11batch/s]

Training batches on cuda:  26%|██▋       | 21/80 [00:01<00:03, 16.25batch/s]

Training batches on cuda:  29%|██▉       | 23/80 [00:01<00:03, 16.18batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:01<00:03, 16.83batch/s]

Training batches on cuda:  34%|███▍      | 27/80 [00:01<00:03, 17.24batch/s]

Training batches on cuda:  36%|███▋      | 29/80 [00:01<00:02, 17.53batch/s]

Training batches on cuda:  39%|███▉      | 31/80 [00:01<00:02, 17.85batch/s]

Training batches on cuda:  41%|████▏     | 33/80 [00:02<00:02, 18.02batch/s]

Training batches on cuda:  44%|████▍     | 35/80 [00:02<00:02, 18.07batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:02<00:02, 16.29batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:02<00:02, 16.86batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:02<00:02, 17.41batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:02<00:02, 17.83batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:02<00:01, 18.07batch/s]

Training batches on cuda:  59%|█████▉    | 47/80 [00:02<00:01, 18.24batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:02<00:01, 18.16batch/s]

Training batches on cuda:  64%|██████▍   | 51/80 [00:03<00:01, 18.28batch/s]

Training batches on cuda:  66%|██████▋   | 53/80 [00:03<00:01, 18.43batch/s]

Training batches on cuda:  69%|██████▉   | 55/80 [00:03<00:01, 18.37batch/s]

Training batches on cuda:  71%|███████▏  | 57/80 [00:03<00:01, 18.48batch/s]

Training batches on cuda:  74%|███████▍  | 59/80 [00:03<00:01, 18.44batch/s]

Training batches on cuda:  76%|███████▋  | 61/80 [00:03<00:01, 18.16batch/s]

Training batches on cuda:  79%|███████▉  | 63/80 [00:03<00:00, 18.19batch/s]

Training batches on cuda:  81%|████████▏ | 65/80 [00:03<00:00, 18.26batch/s]

Training batches on cuda:  84%|████████▍ | 67/80 [00:03<00:00, 18.25batch/s]

Training batches on cuda:  86%|████████▋ | 69/80 [00:04<00:00, 18.35batch/s]

Training batches on cuda:  89%|████████▉ | 71/80 [00:04<00:00, 18.44batch/s]

Training batches on cuda:  91%|█████████▏| 73/80 [00:04<00:00, 18.35batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:04<00:00, 18.32batch/s]

Training batches on cuda:  96%|█████████▋| 77/80 [00:04<00:00, 18.50batch/s]

Training batches on cuda:  99%|█████████▉| 79/80 [00:04<00:00, 18.49batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:19<00:34,  4.93s/epoch, loss=0.000774, prev_loss=0.000774]
Training epochs on cuda:  40%|████      | 4/10 [00:19<00:29,  4.89s/epoch, loss=0.000774, prev_loss=0.000774]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:05, 13.81batch/s]

Training batches on cuda:   5%|▌         | 4/80 [00:00<00:05, 14.93batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:04, 15.79batch/s]

Training batches on cuda:  10%|█         | 8/80 [00:00<00:04, 16.56batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:04, 16.72batch/s]

Training batches on cuda:  15%|█▌        | 12/80 [00:00<00:03, 17.28batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:03, 17.60batch/s]

Training batches on cuda:  20%|██        | 16/80 [00:00<00:03, 17.90batch/s]

Training batches on cuda:  22%|██▎       | 18/80 [00:01<00:03, 18.05batch/s]

Training batches on cuda:  25%|██▌       | 20/80 [00:01<00:03, 18.22batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:01<00:03, 18.41batch/s]

Training batches on cuda:  30%|███       | 24/80 [00:01<00:03, 18.45batch/s]

Training batches on cuda:  32%|███▎      | 26/80 [00:01<00:02, 18.36batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:02, 18.35batch/s]

Training batches on cuda:  38%|███▊      | 30/80 [00:01<00:02, 18.27batch/s]

Training batches on cuda:  40%|████      | 32/80 [00:01<00:02, 17.07batch/s]

Training batches on cuda:  42%|████▎     | 34/80 [00:01<00:02, 17.33batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:02<00:02, 17.40batch/s]

Training batches on cuda:  48%|████▊     | 38/80 [00:02<00:02, 17.32batch/s]

Training batches on cuda:  50%|█████     | 40/80 [00:02<00:02, 17.39batch/s]

Training batches on cuda:  52%|█████▎    | 42/80 [00:02<00:02, 17.53batch/s]

Training batches on cuda:  55%|█████▌    | 44/80 [00:02<00:02, 15.38batch/s]

Training batches on cuda:  57%|█████▊    | 46/80 [00:02<00:02, 15.93batch/s]

Training batches on cuda:  60%|██████    | 48/80 [00:02<00:01, 16.48batch/s]

Training batches on cuda:  62%|██████▎   | 50/80 [00:02<00:01, 16.91batch/s]

Training batches on cuda:  65%|██████▌   | 52/80 [00:02<00:01, 17.23batch/s]

Training batches on cuda:  68%|██████▊   | 54/80 [00:03<00:01, 16.87batch/s]

Training batches on cuda:  70%|███████   | 56/80 [00:03<00:01, 17.25batch/s]

Training batches on cuda:  72%|███████▎  | 58/80 [00:03<00:01, 17.48batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:03<00:01, 17.23batch/s]

Training batches on cuda:  78%|███████▊  | 62/80 [00:03<00:01, 17.37batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:03<00:00, 17.53batch/s]

Training batches on cuda:  82%|████████▎ | 66/80 [00:03<00:00, 17.57batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:03<00:00, 17.27batch/s]

Training batches on cuda:  88%|████████▊ | 70/80 [00:04<00:00, 17.25batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:04<00:00, 17.50batch/s]

Training batches on cuda:  92%|█████████▎| 74/80 [00:04<00:00, 17.57batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:04<00:00, 17.11batch/s]

Training batches on cuda:  98%|█████████▊| 78/80 [00:04<00:00, 17.31batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:24<00:29,  4.89s/epoch, loss=0.000774, prev_loss=0.000774]
Training epochs on cuda:  50%|█████     | 5/10 [00:24<00:24,  4.88s/epoch, loss=0.000774, prev_loss=0.000774]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:05, 13.71batch/s]

Training batches on cuda:   5%|▌         | 4/80 [00:00<00:05, 14.64batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:04, 15.49batch/s]

Training batches on cuda:  10%|█         | 8/80 [00:00<00:04, 16.17batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:04, 16.67batch/s]

Training batches on cuda:  15%|█▌        | 12/80 [00:00<00:04, 17.00batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:03, 17.25batch/s]

Training batches on cuda:  20%|██        | 16/80 [00:00<00:03, 17.36batch/s]

Training batches on cuda:  22%|██▎       | 18/80 [00:01<00:03, 17.55batch/s]

Training batches on cuda:  25%|██▌       | 20/80 [00:01<00:03, 17.66batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:01<00:03, 17.72batch/s]

Training batches on cuda:  30%|███       | 24/80 [00:01<00:03, 17.72batch/s]

Training batches on cuda:  32%|███▎      | 26/80 [00:01<00:03, 17.86batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:02, 17.82batch/s]

Training batches on cuda:  38%|███▊      | 30/80 [00:01<00:02, 17.93batch/s]

Training batches on cuda:  40%|████      | 32/80 [00:01<00:02, 17.97batch/s]

Training batches on cuda:  42%|████▎     | 34/80 [00:01<00:02, 17.92batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:02<00:02, 17.91batch/s]

Training batches on cuda:  48%|████▊     | 38/80 [00:02<00:02, 17.91batch/s]

Training batches on cuda:  50%|█████     | 40/80 [00:02<00:02, 17.94batch/s]

Training batches on cuda:  52%|█████▎    | 42/80 [00:02<00:02, 17.93batch/s]

Training batches on cuda:  55%|█████▌    | 44/80 [00:02<00:02, 18.00batch/s]

Training batches on cuda:  57%|█████▊    | 46/80 [00:02<00:01, 18.06batch/s]

Training batches on cuda:  60%|██████    | 48/80 [00:02<00:01, 16.68batch/s]

Training batches on cuda:  62%|██████▎   | 50/80 [00:02<00:01, 17.00batch/s]

Training batches on cuda:  65%|██████▌   | 52/80 [00:02<00:01, 16.96batch/s]

Training batches on cuda:  68%|██████▊   | 54/80 [00:03<00:01, 16.72batch/s]

Training batches on cuda:  70%|███████   | 56/80 [00:03<00:01, 16.11batch/s]

Training batches on cuda:  72%|███████▎  | 58/80 [00:03<00:01, 16.61batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:03<00:01, 16.39batch/s]

Training batches on cuda:  78%|███████▊  | 62/80 [00:03<00:01, 16.72batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:03<00:00, 16.91batch/s]

Training batches on cuda:  82%|████████▎ | 66/80 [00:03<00:00, 17.28batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:03<00:00, 17.44batch/s]

Training batches on cuda:  88%|████████▊ | 70/80 [00:04<00:00, 17.62batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:04<00:00, 17.75batch/s]

Training batches on cuda:  92%|█████████▎| 74/80 [00:04<00:00, 17.90batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:04<00:00, 17.99batch/s]

Training batches on cuda:  98%|█████████▊| 78/80 [00:04<00:00, 17.83batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:29<00:24,  4.88s/epoch, loss=0.000774, prev_loss=0.000774]
Training epochs on cuda:  60%|██████    | 6/10 [00:29<00:19,  4.85s/epoch, loss=0.000774, prev_loss=0.000774]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 1/80 [00:00<00:09,  8.29batch/s]

Training batches on cuda:   4%|▍         | 3/80 [00:00<00:07,  9.82batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:06, 11.36batch/s]

Training batches on cuda:   9%|▉         | 7/80 [00:00<00:05, 12.76batch/s]

Training batches on cuda:  11%|█▏        | 9/80 [00:00<00:05, 13.93batch/s]

Training batches on cuda:  14%|█▍        | 11/80 [00:00<00:04, 14.89batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:04, 15.67batch/s]

Training batches on cuda:  19%|█▉        | 15/80 [00:00<00:04, 16.23batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:01<00:03, 16.60batch/s]

Training batches on cuda:  24%|██▍       | 19/80 [00:01<00:03, 16.92batch/s]

Training batches on cuda:  26%|██▋       | 21/80 [00:01<00:03, 17.22batch/s]

Training batches on cuda:  29%|██▉       | 23/80 [00:01<00:03, 17.25batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:01<00:03, 17.17batch/s]

Training batches on cuda:  34%|███▍      | 27/80 [00:01<00:03, 17.27batch/s]

Training batches on cuda:  36%|███▋      | 29/80 [00:01<00:02, 17.31batch/s]

Training batches on cuda:  39%|███▉      | 31/80 [00:01<00:02, 17.35batch/s]

Training batches on cuda:  41%|████▏     | 33/80 [00:01<00:02, 17.42batch/s]

Training batches on cuda:  44%|████▍     | 35/80 [00:02<00:02, 17.57batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:02<00:02, 17.72batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:02<00:02, 17.83batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:02<00:02, 17.90batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:02<00:02, 17.98batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:02<00:01, 18.04batch/s]

Training batches on cuda:  59%|█████▉    | 47/80 [00:02<00:01, 17.98batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:02<00:01, 17.99batch/s]

Training batches on cuda:  64%|██████▍   | 51/80 [00:02<00:01, 16.82batch/s]

Training batches on cuda:  66%|██████▋   | 53/80 [00:03<00:01, 17.14batch/s]

Training batches on cuda:  69%|██████▉   | 55/80 [00:03<00:01, 17.22batch/s]

Training batches on cuda:  71%|███████▏  | 57/80 [00:03<00:01, 17.49batch/s]

Training batches on cuda:  74%|███████▍  | 59/80 [00:03<00:01, 17.69batch/s]

Training batches on cuda:  76%|███████▋  | 61/80 [00:03<00:01, 17.85batch/s]

Training batches on cuda:  79%|███████▉  | 63/80 [00:03<00:00, 17.92batch/s]

Training batches on cuda:  81%|████████▏ | 65/80 [00:03<00:00, 17.92batch/s]

Training batches on cuda:  84%|████████▍ | 67/80 [00:03<00:00, 17.28batch/s]

Training batches on cuda:  86%|████████▋ | 69/80 [00:03<00:00, 17.43batch/s]

Training batches on cuda:  89%|████████▉ | 71/80 [00:04<00:00, 17.51batch/s]

Training batches on cuda:  91%|█████████▏| 73/80 [00:04<00:00, 17.31batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:04<00:00, 17.59batch/s]

Training batches on cuda:  96%|█████████▋| 77/80 [00:04<00:00, 17.60batch/s]

Training batches on cuda:  99%|█████████▉| 79/80 [00:04<00:00, 17.12batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [00:34<00:19,  4.85s/epoch, loss=0.000773, prev_loss=0.000774]
Training epochs on cuda:  70%|███████   | 7/10 [00:34<00:14,  4.85s/epoch, loss=0.000773, prev_loss=0.000774]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:05, 13.33batch/s]

Training batches on cuda:   5%|▌         | 4/80 [00:00<00:05, 14.39batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:04, 15.29batch/s]

Training batches on cuda:  10%|█         | 8/80 [00:00<00:04, 16.06batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:04, 16.61batch/s]

Training batches on cuda:  15%|█▌        | 12/80 [00:00<00:03, 17.05batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:03, 17.35batch/s]

Training batches on cuda:  20%|██        | 16/80 [00:00<00:03, 17.57batch/s]

Training batches on cuda:  22%|██▎       | 18/80 [00:01<00:03, 17.70batch/s]

Training batches on cuda:  25%|██▌       | 20/80 [00:01<00:03, 17.84batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:01<00:03, 18.08batch/s]

Training batches on cuda:  30%|███       | 24/80 [00:01<00:03, 18.13batch/s]

Training batches on cuda:  32%|███▎      | 26/80 [00:01<00:02, 18.14batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:02, 18.02batch/s]

Training batches on cuda:  38%|███▊      | 30/80 [00:01<00:02, 18.08batch/s]

Training batches on cuda:  40%|████      | 32/80 [00:01<00:02, 18.09batch/s]

Training batches on cuda:  42%|████▎     | 34/80 [00:01<00:02, 17.84batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:02<00:02, 17.74batch/s]

Training batches on cuda:  48%|████▊     | 38/80 [00:02<00:02, 17.87batch/s]

Training batches on cuda:  50%|█████     | 40/80 [00:02<00:02, 17.93batch/s]

Training batches on cuda:  52%|█████▎    | 42/80 [00:02<00:02, 17.94batch/s]

Training batches on cuda:  55%|█████▌    | 44/80 [00:02<00:02, 17.97batch/s]

Training batches on cuda:  57%|█████▊    | 46/80 [00:02<00:01, 18.00batch/s]

Training batches on cuda:  60%|██████    | 48/80 [00:02<00:01, 17.52batch/s]

Training batches on cuda:  62%|██████▎   | 50/80 [00:02<00:01, 17.70batch/s]

Training batches on cuda:  65%|██████▌   | 52/80 [00:02<00:01, 17.87batch/s]

Training batches on cuda:  68%|██████▊   | 54/80 [00:03<00:01, 18.04batch/s]

Training batches on cuda:  70%|███████   | 56/80 [00:03<00:01, 16.88batch/s]

Training batches on cuda:  72%|███████▎  | 58/80 [00:03<00:01, 17.19batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:03<00:01, 17.39batch/s]

Training batches on cuda:  78%|███████▊  | 62/80 [00:03<00:01, 17.74batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:03<00:00, 17.98batch/s]

Training batches on cuda:  82%|████████▎ | 66/80 [00:03<00:00, 16.93batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:03<00:00, 17.24batch/s]

Training batches on cuda:  88%|████████▊ | 70/80 [00:03<00:00, 17.42batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:04<00:00, 16.88batch/s]

Training batches on cuda:  92%|█████████▎| 74/80 [00:04<00:00, 16.96batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:04<00:00, 17.11batch/s]

Training batches on cuda:  98%|█████████▊| 78/80 [00:04<00:00, 17.40batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [00:38<00:14,  4.85s/epoch, loss=0.000773, prev_loss=0.000773]
Training epochs on cuda:  80%|████████  | 8/10 [00:38<00:09,  4.81s/epoch, loss=0.000773, prev_loss=0.000773]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 1/80 [00:00<00:09,  7.92batch/s]

Training batches on cuda:   4%|▍         | 3/80 [00:00<00:08,  9.48batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:06, 11.04batch/s]

Training batches on cuda:   9%|▉         | 7/80 [00:00<00:06, 12.15batch/s]

Training batches on cuda:  11%|█▏        | 9/80 [00:00<00:05, 13.26batch/s]

Training batches on cuda:  14%|█▍        | 11/80 [00:00<00:05, 13.65batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:04, 14.01batch/s]

Training batches on cuda:  19%|█▉        | 15/80 [00:00<00:04, 14.35batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:01<00:04, 15.06batch/s]

Training batches on cuda:  24%|██▍       | 19/80 [00:01<00:03, 16.20batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:01<00:03, 17.23batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:01<00:03, 18.05batch/s]

Training batches on cuda:  34%|███▍      | 27/80 [00:01<00:02, 18.27batch/s]

Training batches on cuda:  36%|███▋      | 29/80 [00:01<00:02, 18.54batch/s]

Training batches on cuda:  39%|███▉      | 31/80 [00:01<00:02, 18.74batch/s]

Training batches on cuda:  41%|████▏     | 33/80 [00:01<00:02, 18.90batch/s]

Training batches on cuda:  44%|████▍     | 35/80 [00:02<00:02, 19.00batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:02<00:02, 18.88batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:02<00:02, 18.99batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:02<00:02, 19.04batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:02<00:01, 19.14batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:02<00:01, 19.10batch/s]

Training batches on cuda:  59%|█████▉    | 47/80 [00:02<00:01, 19.15batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:02<00:01, 19.18batch/s]

Training batches on cuda:  64%|██████▍   | 51/80 [00:02<00:01, 19.18batch/s]

Training batches on cuda:  66%|██████▋   | 53/80 [00:02<00:01, 19.20batch/s]

Training batches on cuda:  69%|██████▉   | 55/80 [00:03<00:01, 19.20batch/s]

Training batches on cuda:  71%|███████▏  | 57/80 [00:03<00:01, 19.20batch/s]

Training batches on cuda:  74%|███████▍  | 59/80 [00:03<00:01, 19.20batch/s]

Training batches on cuda:  76%|███████▋  | 61/80 [00:03<00:00, 19.20batch/s]

Training batches on cuda:  79%|███████▉  | 63/80 [00:03<00:00, 17.63batch/s]

Training batches on cuda:  81%|████████▏ | 65/80 [00:03<00:00, 18.08batch/s]

Training batches on cuda:  84%|████████▍ | 67/80 [00:03<00:00, 18.40batch/s]

Training batches on cuda:  86%|████████▋ | 69/80 [00:03<00:00, 18.66batch/s]

Training batches on cuda:  89%|████████▉ | 71/80 [00:03<00:00, 18.81batch/s]

Training batches on cuda:  91%|█████████▏| 73/80 [00:04<00:00, 18.68batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:04<00:00, 18.83batch/s]

Training batches on cuda:  96%|█████████▋| 77/80 [00:04<00:00, 18.90batch/s]

Training batches on cuda:  99%|█████████▉| 79/80 [00:04<00:00, 19.01batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [00:43<00:09,  4.81s/epoch, loss=0.000759, prev_loss=0.000773]
Training epochs on cuda:  90%|█████████ | 9/10 [00:43<00:04,  4.75s/epoch, loss=0.000759, prev_loss=0.000773]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:05, 13.16batch/s]

Training batches on cuda:   5%|▌         | 4/80 [00:00<00:05, 14.13batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:04, 15.27batch/s]

Training batches on cuda:  10%|█         | 8/80 [00:00<00:04, 16.22batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:04, 16.95batch/s]

Training batches on cuda:  15%|█▌        | 12/80 [00:00<00:03, 17.46batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:03, 17.83batch/s]

Training batches on cuda:  20%|██        | 16/80 [00:00<00:03, 18.13batch/s]

Training batches on cuda:  22%|██▎       | 18/80 [00:01<00:03, 17.86batch/s]

Training batches on cuda:  25%|██▌       | 20/80 [00:01<00:03, 17.85batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:01<00:03, 17.61batch/s]

Training batches on cuda:  30%|███       | 24/80 [00:01<00:03, 17.48batch/s]

Training batches on cuda:  32%|███▎      | 26/80 [00:01<00:03, 17.57batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:02, 17.47batch/s]

Training batches on cuda:  38%|███▊      | 30/80 [00:01<00:02, 17.53batch/s]

Training batches on cuda:  40%|████      | 32/80 [00:01<00:02, 17.65batch/s]

Training batches on cuda:  42%|████▎     | 34/80 [00:01<00:02, 17.56batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:02<00:02, 17.45batch/s]

Training batches on cuda:  48%|████▊     | 38/80 [00:02<00:02, 17.37batch/s]

Training batches on cuda:  50%|█████     | 40/80 [00:02<00:02, 17.44batch/s]

Training batches on cuda:  52%|█████▎    | 42/80 [00:02<00:02, 17.61batch/s]

Training batches on cuda:  55%|█████▌    | 44/80 [00:02<00:02, 17.73batch/s]

Training batches on cuda:  57%|█████▊    | 46/80 [00:02<00:01, 17.59batch/s]

Training batches on cuda:  60%|██████    | 48/80 [00:02<00:01, 17.62batch/s]

Training batches on cuda:  62%|██████▎   | 50/80 [00:02<00:01, 17.69batch/s]

Training batches on cuda:  65%|██████▌   | 52/80 [00:02<00:01, 17.72batch/s]

Training batches on cuda:  68%|██████▊   | 54/80 [00:03<00:01, 17.83batch/s]

Training batches on cuda:  70%|███████   | 56/80 [00:03<00:01, 17.89batch/s]

Training batches on cuda:  72%|███████▎  | 58/80 [00:03<00:01, 17.67batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:03<00:01, 17.80batch/s]

Training batches on cuda:  78%|███████▊  | 62/80 [00:03<00:01, 17.76batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:03<00:00, 17.65batch/s]

Training batches on cuda:  82%|████████▎ | 66/80 [00:03<00:00, 17.78batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:03<00:00, 17.86batch/s]

Training batches on cuda:  88%|████████▊ | 70/80 [00:03<00:00, 16.97batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:04<00:00, 17.29batch/s]

Training batches on cuda:  92%|█████████▎| 74/80 [00:04<00:00, 17.53batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:04<00:00, 17.61batch/s]

Training batches on cuda:  98%|█████████▊| 78/80 [00:04<00:00, 17.46batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [00:48<00:04,  4.75s/epoch, loss=0.000741, prev_loss=0.000759]2021-09-16:17:48:31,767 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:48:32,872 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:48:47,890 INFO     [evaluator.py:657] Evaluation took 15.02s seconds
2021-09-16:17:48:48,26 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00,  9.64s/epoch, loss=0.000741, prev_loss=0.000759]
Training epochs on cuda: 100%|██████████| 10/10 [01:04<00:00,  6.44s/epoch, loss=0.000741, prev_loss=0.000759]
2021-09-16:17:48:48,27 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpoi3e_u99'
2021-09-16:17:48:48,104 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpoi3e_u99' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.51ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.62ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:17:49:02,933 INFO     [evaluator.py:657] Evaluation took 14.81s seconds
[I 2021-09-16 17:49:02,938] Trial 15 finished with value: 0.17212141669971792 and parameters: {'loss.margin': 3, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.1187181904805383, 'optimizer.lr': 0.03431612508387206, 'negative_sampler.num_negs_per_pos': 18, 'training.batch_size': 3932}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:49:02,958 WARNING  [api.py:823] No random seed is specified. Setting to 933910477.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 11/2354 [00:00<00:22, 106.07batch/s]

Training batches on cuda:   1%|▏         | 30/2354 [00:00<00:19, 122.05batch/s]

Training batches on cuda:   2%|▏         | 49/2354 [00:00<00:17, 135.39batch/s]

Training batches on cuda:   3%|▎         | 70/2354 [00:00<00:15, 151.15batch/s]

Training batches on cuda:   4%|▍         | 92/2354 [00:00<00:13, 165.28batch/s]

Training batches on cuda:   5%|▍         | 113/2354 [00:00<00:12, 176.03batch/s]

Training batches on cuda:   6%|▌         | 131/2354 [00:00<00:12, 176.62batch/s]

Training batches on cuda:   7%|▋         | 155/2354 [00:00<00:11, 189.91batch/s]

Training batches on cuda:   8%|▊         | 180/2354 [00:00<00:10, 203.46batch/s]

Training batches on cuda:   9%|▊         | 204/2354 [00:01<00:10, 212.01batch/s]

Training batches on cuda:  10%|▉         | 228/2354 [00:01<00:09, 219.06batch/s]

Training batches on cuda:  11%|█         | 251/2354 [00:01<00:09, 216.89batch/s]

Training batches on cuda:  12%|█▏        | 273/2354 [00:01<00:10, 200.89batch/s]

Training batches on cuda:  13%|█▎        | 297/2354 [00:01<00:09, 209.59batch/s]

Training batches on cuda:  14%|█▎        | 319/2354 [00:01<00:09, 208.32batch/s]

Training batches on cuda:  14%|█▍        | 341/2354 [00:01<00:09, 207.06batch/s]

Training batches on cuda:  15%|█▌        | 364/2354 [00:01<00:09, 211.84batch/s]

Training batches on cuda:  16%|█▋        | 387/2354 [00:01<00:09, 214.73batch/s]

Training batches on cuda:  17%|█▋        | 410/2354 [00:01<00:08, 217.29batch/s]

Training batches on cuda:  18%|█▊        | 432/2354 [00:02<00:09, 212.33batch/s]

Training batches on cuda:  19%|█▉        | 454/2354 [00:02<00:09, 206.87batch/s]

Training batches on cuda:  20%|██        | 475/2354 [00:02<00:09, 202.44batch/s]

Training batches on cuda:  21%|██        | 498/2354 [00:02<00:08, 208.22batch/s]

Training batches on cuda:  22%|██▏       | 519/2354 [00:02<00:08, 206.65batch/s]

Training batches on cuda:  23%|██▎       | 540/2354 [00:02<00:08, 206.65batch/s]

Training batches on cuda:  24%|██▍       | 561/2354 [00:02<00:09, 197.65batch/s]

Training batches on cuda:  25%|██▍       | 581/2354 [00:02<00:09, 183.58batch/s]

Training batches on cuda:  25%|██▌       | 600/2354 [00:02<00:09, 180.72batch/s]

Training batches on cuda:  26%|██▋       | 619/2354 [00:03<00:09, 182.76batch/s]

Training batches on cuda:  27%|██▋       | 642/2354 [00:03<00:08, 192.49batch/s]

Training batches on cuda:  28%|██▊       | 662/2354 [00:03<00:08, 190.79batch/s]

Training batches on cuda:  29%|██▉       | 683/2354 [00:03<00:08, 196.06batch/s]

Training batches on cuda:  30%|██▉       | 704/2354 [00:03<00:08, 199.37batch/s]

Training batches on cuda:  31%|███       | 725/2354 [00:03<00:08, 199.82batch/s]

Training batches on cuda:  32%|███▏      | 746/2354 [00:03<00:08, 195.73batch/s]

Training batches on cuda:  33%|███▎      | 766/2354 [00:03<00:08, 194.70batch/s]

Training batches on cuda:  33%|███▎      | 786/2354 [00:03<00:08, 186.78batch/s]

Training batches on cuda:  34%|███▍      | 808/2354 [00:04<00:07, 194.96batch/s]

Training batches on cuda:  35%|███▌      | 829/2354 [00:04<00:07, 197.30batch/s]

Training batches on cuda:  36%|███▌      | 849/2354 [00:04<00:08, 187.54batch/s]

Training batches on cuda:  37%|███▋      | 872/2354 [00:04<00:07, 196.58batch/s]

Training batches on cuda:  38%|███▊      | 895/2354 [00:04<00:07, 203.60batch/s]

Training batches on cuda:  39%|███▉      | 916/2354 [00:04<00:07, 200.27batch/s]

Training batches on cuda:  40%|███▉      | 937/2354 [00:04<00:07, 201.01batch/s]

Training batches on cuda:  41%|████      | 958/2354 [00:04<00:06, 200.97batch/s]

Training batches on cuda:  42%|████▏     | 979/2354 [00:04<00:06, 199.44batch/s]

Training batches on cuda:  43%|████▎     | 1001/2354 [00:04<00:06, 204.78batch/s]

Training batches on cuda:  43%|████▎     | 1023/2354 [00:05<00:06, 208.89batch/s]

Training batches on cuda:  44%|████▍     | 1044/2354 [00:05<00:06, 203.84batch/s]

Training batches on cuda:  45%|████▌     | 1066/2354 [00:05<00:06, 207.64batch/s]

Training batches on cuda:  46%|████▌     | 1087/2354 [00:05<00:06, 193.47batch/s]

Training batches on cuda:  47%|████▋     | 1107/2354 [00:05<00:06, 184.25batch/s]

Training batches on cuda:  48%|████▊     | 1127/2354 [00:05<00:06, 188.24batch/s]

Training batches on cuda:  49%|████▉     | 1149/2354 [00:05<00:06, 193.95batch/s]

Training batches on cuda:  50%|████▉     | 1172/2354 [00:05<00:05, 201.76batch/s]

Training batches on cuda:  51%|█████     | 1193/2354 [00:05<00:05, 202.78batch/s]

Training batches on cuda:  52%|█████▏    | 1214/2354 [00:06<00:05, 202.69batch/s]

Training batches on cuda:  53%|█████▎    | 1236/2354 [00:06<00:05, 205.94batch/s]

Training batches on cuda:  53%|█████▎    | 1258/2354 [00:06<00:05, 207.46batch/s]

Training batches on cuda:  54%|█████▍    | 1279/2354 [00:06<00:05, 205.74batch/s]

Training batches on cuda:  55%|█████▌    | 1300/2354 [00:06<00:05, 200.28batch/s]

Training batches on cuda:  56%|█████▌    | 1321/2354 [00:06<00:05, 198.95batch/s]

Training batches on cuda:  57%|█████▋    | 1341/2354 [00:06<00:05, 194.60batch/s]

Training batches on cuda:  58%|█████▊    | 1361/2354 [00:06<00:05, 194.73batch/s]

Training batches on cuda:  59%|█████▊    | 1381/2354 [00:06<00:05, 194.55batch/s]

Training batches on cuda:  60%|█████▉    | 1402/2354 [00:07<00:04, 197.14batch/s]

Training batches on cuda:  60%|██████    | 1422/2354 [00:07<00:04, 190.41batch/s]

Training batches on cuda:  61%|██████▏   | 1443/2354 [00:07<00:04, 195.14batch/s]

Training batches on cuda:  62%|██████▏   | 1468/2354 [00:07<00:04, 207.59batch/s]

Training batches on cuda:  63%|██████▎   | 1491/2354 [00:07<00:04, 213.37batch/s]

Training batches on cuda:  64%|██████▍   | 1516/2354 [00:07<00:03, 221.92batch/s]

Training batches on cuda:  65%|██████▌   | 1540/2354 [00:07<00:03, 224.94batch/s]

Training batches on cuda:  66%|██████▋   | 1564/2354 [00:07<00:03, 228.95batch/s]

Training batches on cuda:  67%|██████▋   | 1588/2354 [00:07<00:03, 227.62batch/s]

Training batches on cuda:  68%|██████▊   | 1612/2354 [00:07<00:03, 228.68batch/s]

Training batches on cuda:  70%|██████▉   | 1637/2354 [00:08<00:03, 233.97batch/s]

Training batches on cuda:  71%|███████   | 1662/2354 [00:08<00:02, 235.22batch/s]

Training batches on cuda:  72%|███████▏  | 1687/2354 [00:08<00:02, 238.99batch/s]

Training batches on cuda:  73%|███████▎  | 1713/2354 [00:08<00:02, 242.77batch/s]

Training batches on cuda:  74%|███████▍  | 1738/2354 [00:08<00:02, 242.48batch/s]

Training batches on cuda:  75%|███████▍  | 1764/2354 [00:08<00:02, 246.49batch/s]

Training batches on cuda:  76%|███████▌  | 1789/2354 [00:08<00:02, 241.83batch/s]

Training batches on cuda:  77%|███████▋  | 1814/2354 [00:08<00:02, 237.99batch/s]

Training batches on cuda:  78%|███████▊  | 1838/2354 [00:08<00:02, 235.22batch/s]

Training batches on cuda:  79%|███████▉  | 1862/2354 [00:08<00:02, 231.59batch/s]

Training batches on cuda:  80%|████████  | 1886/2354 [00:09<00:02, 231.76batch/s]

Training batches on cuda:  81%|████████  | 1910/2354 [00:09<00:01, 229.71batch/s]

Training batches on cuda:  82%|████████▏ | 1934/2354 [00:09<00:01, 230.13batch/s]

Training batches on cuda:  83%|████████▎ | 1958/2354 [00:09<00:01, 231.02batch/s]

Training batches on cuda:  84%|████████▍ | 1982/2354 [00:09<00:01, 231.02batch/s]

Training batches on cuda:  85%|████████▌ | 2006/2354 [00:09<00:01, 229.43batch/s]

Training batches on cuda:  86%|████████▌ | 2029/2354 [00:09<00:01, 228.44batch/s]

Training batches on cuda:  87%|████████▋ | 2052/2354 [00:09<00:01, 224.95batch/s]

Training batches on cuda:  88%|████████▊ | 2075/2354 [00:09<00:01, 224.90batch/s]

Training batches on cuda:  89%|████████▉ | 2098/2354 [00:10<00:01, 225.43batch/s]

Training batches on cuda:  90%|█████████ | 2121/2354 [00:10<00:01, 212.11batch/s]

Training batches on cuda:  91%|█████████ | 2143/2354 [00:10<00:01, 194.21batch/s]

Training batches on cuda:  92%|█████████▏| 2163/2354 [00:10<00:01, 185.03batch/s]

Training batches on cuda:  93%|█████████▎| 2183/2354 [00:10<00:00, 187.37batch/s]

Training batches on cuda:  94%|█████████▎| 2204/2354 [00:10<00:00, 193.01batch/s]

Training batches on cuda:  94%|█████████▍| 2224/2354 [00:10<00:00, 193.70batch/s]

Training batches on cuda:  95%|█████████▌| 2244/2354 [00:10<00:00, 195.36batch/s]

Training batches on cuda:  96%|█████████▌| 2265/2354 [00:10<00:00, 199.51batch/s]

Training batches on cuda:  97%|█████████▋| 2286/2354 [00:11<00:00, 200.26batch/s]

Training batches on cuda:  98%|█████████▊| 2307/2354 [00:11<00:00, 198.20batch/s]

Training batches on cuda:  99%|█████████▉| 2327/2354 [00:11<00:00, 196.75batch/s]

Training batches on cuda: 100%|█████████▉| 2347/2354 [00:11<00:00, 194.74batch/s]

                                                                                 
Training epochs on cuda:   0%|          | 0/10 [00:11<?, ?epoch/s, loss=0.0162, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:11<01:44, 11.64s/epoch, loss=0.0162, prev_loss=nan]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 13/2354 [00:00<00:18, 127.76batch/s]

Training batches on cuda:   1%|▏         | 32/2354 [00:00<00:16, 141.68batch/s]

Training batches on cuda:   2%|▏         | 52/2354 [00:00<00:15, 153.45batch/s]

Training batches on cuda:   3%|▎         | 72/2354 [00:00<00:13, 163.07batch/s]

Training batches on cuda:   4%|▍         | 91/2354 [00:00<00:13, 170.06batch/s]

Training batches on cuda:   5%|▍         | 111/2354 [00:00<00:12, 176.72batch/s]

Training batches on cuda:   6%|▌         | 131/2354 [00:00<00:12, 181.52batch/s]

Training batches on cuda:   6%|▋         | 153/2354 [00:00<00:11, 189.44batch/s]

Training batches on cuda:   7%|▋         | 176/2354 [00:00<00:10, 199.90batch/s]

Training batches on cuda:   8%|▊         | 199/2354 [00:01<00:10, 207.89batch/s]

Training batches on cuda:   9%|▉         | 222/2354 [00:01<00:09, 213.78batch/s]

Training batches on cuda:  10%|█         | 244/2354 [00:01<00:10, 210.78batch/s]

Training batches on cuda:  11%|█▏        | 266/2354 [00:01<00:10, 204.56batch/s]

Training batches on cuda:  12%|█▏        | 287/2354 [00:01<00:10, 203.13batch/s]

Training batches on cuda:  13%|█▎        | 308/2354 [00:01<00:10, 203.17batch/s]

Training batches on cuda:  14%|█▍        | 329/2354 [00:01<00:10, 201.35batch/s]

Training batches on cuda:  15%|█▍        | 350/2354 [00:01<00:09, 202.09batch/s]

Training batches on cuda:  16%|█▌        | 371/2354 [00:01<00:09, 202.44batch/s]

Training batches on cuda:  17%|█▋        | 392/2354 [00:01<00:09, 199.45batch/s]

Training batches on cuda:  18%|█▊        | 412/2354 [00:02<00:10, 183.35batch/s]

Training batches on cuda:  18%|█▊        | 431/2354 [00:02<00:10, 176.82batch/s]

Training batches on cuda:  19%|█▉        | 452/2354 [00:02<00:10, 183.58batch/s]

Training batches on cuda:  20%|██        | 473/2354 [00:02<00:09, 188.92batch/s]

Training batches on cuda:  21%|██        | 495/2354 [00:02<00:09, 195.16batch/s]

Training batches on cuda:  22%|██▏       | 516/2354 [00:02<00:09, 197.29batch/s]

Training batches on cuda:  23%|██▎       | 537/2354 [00:02<00:09, 200.65batch/s]

Training batches on cuda:  24%|██▎       | 558/2354 [00:02<00:08, 202.86batch/s]

Training batches on cuda:  25%|██▍       | 579/2354 [00:02<00:08, 203.51batch/s]

Training batches on cuda:  25%|██▌       | 600/2354 [00:03<00:08, 204.81batch/s]

Training batches on cuda:  26%|██▋       | 621/2354 [00:03<00:08, 204.88batch/s]

Training batches on cuda:  27%|██▋       | 643/2354 [00:03<00:08, 209.11batch/s]

Training batches on cuda:  28%|██▊       | 666/2354 [00:03<00:07, 213.11batch/s]

Training batches on cuda:  29%|██▉       | 689/2354 [00:03<00:07, 217.66batch/s]

Training batches on cuda:  30%|███       | 712/2354 [00:03<00:07, 218.95batch/s]

Training batches on cuda:  31%|███       | 735/2354 [00:03<00:07, 221.91batch/s]

Training batches on cuda:  32%|███▏      | 758/2354 [00:03<00:07, 219.64batch/s]

Training batches on cuda:  33%|███▎      | 780/2354 [00:03<00:07, 218.07batch/s]

Training batches on cuda:  34%|███▍      | 803/2354 [00:03<00:07, 219.83batch/s]

Training batches on cuda:  35%|███▌      | 826/2354 [00:04<00:07, 214.62batch/s]

Training batches on cuda:  36%|███▌      | 848/2354 [00:04<00:07, 192.54batch/s]

Training batches on cuda:  37%|███▋      | 868/2354 [00:04<00:08, 185.46batch/s]

Training batches on cuda:  38%|███▊      | 887/2354 [00:04<00:08, 181.63batch/s]

Training batches on cuda:  38%|███▊      | 906/2354 [00:04<00:08, 180.46batch/s]

Training batches on cuda:  39%|███▉      | 925/2354 [00:04<00:07, 182.53batch/s]

Training batches on cuda:  40%|████      | 948/2354 [00:04<00:07, 192.70batch/s]

Training batches on cuda:  41%|████      | 971/2354 [00:04<00:06, 201.17batch/s]

Training batches on cuda:  42%|████▏     | 992/2354 [00:04<00:07, 188.65batch/s]

Training batches on cuda:  43%|████▎     | 1012/2354 [00:05<00:07, 180.24batch/s]

Training batches on cuda:  44%|████▍     | 1031/2354 [00:05<00:07, 174.64batch/s]

Training batches on cuda:  45%|████▍     | 1049/2354 [00:05<00:07, 167.11batch/s]

Training batches on cuda:  45%|████▌     | 1067/2354 [00:05<00:07, 168.30batch/s]

Training batches on cuda:  46%|████▌     | 1085/2354 [00:05<00:07, 168.41batch/s]

Training batches on cuda:  47%|████▋     | 1103/2354 [00:05<00:07, 170.84batch/s]

Training batches on cuda:  48%|████▊     | 1123/2354 [00:05<00:06, 176.44batch/s]

Training batches on cuda:  49%|████▊     | 1142/2354 [00:05<00:06, 179.95batch/s]

Training batches on cuda:  49%|████▉     | 1162/2354 [00:05<00:06, 183.00batch/s]

Training batches on cuda:  50%|█████     | 1182/2354 [00:06<00:06, 185.72batch/s]

Training batches on cuda:  51%|█████     | 1201/2354 [00:06<00:06, 182.89batch/s]

Training batches on cuda:  52%|█████▏    | 1221/2354 [00:06<00:06, 185.96batch/s]

Training batches on cuda:  53%|█████▎    | 1240/2354 [00:06<00:06, 181.25batch/s]

Training batches on cuda:  54%|█████▎    | 1260/2354 [00:06<00:05, 184.01batch/s]

Training batches on cuda:  54%|█████▍    | 1279/2354 [00:06<00:05, 185.63batch/s]

Training batches on cuda:  55%|█████▌    | 1299/2354 [00:06<00:05, 188.85batch/s]

Training batches on cuda:  56%|█████▌    | 1319/2354 [00:06<00:05, 189.52batch/s]

Training batches on cuda:  57%|█████▋    | 1339/2354 [00:06<00:05, 190.12batch/s]

Training batches on cuda:  58%|█████▊    | 1359/2354 [00:07<00:05, 190.98batch/s]

Training batches on cuda:  59%|█████▊    | 1379/2354 [00:07<00:05, 190.93batch/s]

Training batches on cuda:  59%|█████▉    | 1399/2354 [00:07<00:04, 191.02batch/s]

Training batches on cuda:  60%|██████    | 1419/2354 [00:07<00:04, 190.43batch/s]

Training batches on cuda:  61%|██████    | 1439/2354 [00:07<00:04, 188.08batch/s]

Training batches on cuda:  62%|██████▏   | 1458/2354 [00:07<00:04, 187.30batch/s]

Training batches on cuda:  63%|██████▎   | 1477/2354 [00:07<00:04, 187.69batch/s]

Training batches on cuda:  64%|██████▎   | 1499/2354 [00:07<00:04, 194.13batch/s]

Training batches on cuda:  65%|██████▍   | 1522/2354 [00:07<00:04, 201.57batch/s]

Training batches on cuda:  66%|██████▌   | 1545/2354 [00:07<00:03, 207.05batch/s]

Training batches on cuda:  67%|██████▋   | 1566/2354 [00:08<00:03, 207.88batch/s]

Training batches on cuda:  67%|██████▋   | 1588/2354 [00:08<00:03, 209.49batch/s]

Training batches on cuda:  68%|██████▊   | 1611/2354 [00:08<00:03, 213.11batch/s]

Training batches on cuda:  69%|██████▉   | 1633/2354 [00:08<00:03, 213.66batch/s]

Training batches on cuda:  70%|███████   | 1655/2354 [00:08<00:03, 214.99batch/s]

Training batches on cuda:  71%|███████   | 1677/2354 [00:08<00:03, 214.86batch/s]

Training batches on cuda:  72%|███████▏  | 1699/2354 [00:08<00:03, 206.25batch/s]

Training batches on cuda:  73%|███████▎  | 1720/2354 [00:08<00:03, 201.50batch/s]

Training batches on cuda:  74%|███████▍  | 1741/2354 [00:08<00:03, 198.47batch/s]

Training batches on cuda:  75%|███████▍  | 1761/2354 [00:09<00:03, 195.83batch/s]

Training batches on cuda:  76%|███████▌  | 1781/2354 [00:09<00:02, 194.22batch/s]

Training batches on cuda:  77%|███████▋  | 1801/2354 [00:09<00:02, 193.48batch/s]

Training batches on cuda:  77%|███████▋  | 1821/2354 [00:09<00:02, 192.10batch/s]

Training batches on cuda:  78%|███████▊  | 1841/2354 [00:09<00:02, 188.09batch/s]

Training batches on cuda:  79%|███████▉  | 1860/2354 [00:09<00:02, 186.68batch/s]

Training batches on cuda:  80%|███████▉  | 1879/2354 [00:09<00:02, 183.79batch/s]

Training batches on cuda:  81%|████████  | 1899/2354 [00:09<00:02, 185.90batch/s]

Training batches on cuda:  82%|████████▏ | 1919/2354 [00:09<00:02, 187.75batch/s]

Training batches on cuda:  82%|████████▏ | 1939/2354 [00:09<00:02, 188.38batch/s]

Training batches on cuda:  83%|████████▎ | 1959/2354 [00:10<00:02, 188.92batch/s]

Training batches on cuda:  84%|████████▍ | 1979/2354 [00:10<00:01, 190.58batch/s]

Training batches on cuda:  85%|████████▍ | 1999/2354 [00:10<00:01, 190.44batch/s]

Training batches on cuda:  86%|████████▌ | 2019/2354 [00:10<00:01, 190.59batch/s]

Training batches on cuda:  87%|████████▋ | 2039/2354 [00:10<00:01, 188.13batch/s]

Training batches on cuda:  87%|████████▋ | 2058/2354 [00:10<00:01, 183.76batch/s]

Training batches on cuda:  88%|████████▊ | 2077/2354 [00:10<00:01, 185.29batch/s]

Training batches on cuda:  89%|████████▉ | 2098/2354 [00:10<00:01, 191.73batch/s]

Training batches on cuda:  90%|████████▉ | 2118/2354 [00:10<00:01, 190.73batch/s]

Training batches on cuda:  91%|█████████ | 2138/2354 [00:11<00:01, 185.63batch/s]

Training batches on cuda:  92%|█████████▏| 2157/2354 [00:11<00:01, 185.60batch/s]

Training batches on cuda:  92%|█████████▏| 2177/2354 [00:11<00:00, 187.05batch/s]

Training batches on cuda:  93%|█████████▎| 2196/2354 [00:11<00:00, 187.50batch/s]

Training batches on cuda:  94%|█████████▍| 2217/2354 [00:11<00:00, 191.64batch/s]

Training batches on cuda:  95%|█████████▌| 2237/2354 [00:11<00:00, 192.16batch/s]

Training batches on cuda:  96%|█████████▌| 2257/2354 [00:11<00:00, 191.52batch/s]

Training batches on cuda:  97%|█████████▋| 2277/2354 [00:11<00:00, 179.34batch/s]

Training batches on cuda:  98%|█████████▊| 2297/2354 [00:11<00:00, 183.43batch/s]

Training batches on cuda:  98%|█████████▊| 2318/2354 [00:11<00:00, 190.01batch/s]

Training batches on cuda:  99%|█████████▉| 2341/2354 [00:12<00:00, 199.28batch/s]

                                                                                 
Training epochs on cuda:  10%|█         | 1/10 [00:24<01:44, 11.64s/epoch, loss=0.0162, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:24<01:34, 11.87s/epoch, loss=0.0162, prev_loss=nan]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 12/2354 [00:00<00:20, 114.75batch/s]

Training batches on cuda:   1%|▏         | 31/2354 [00:00<00:17, 129.59batch/s]

Training batches on cuda:   2%|▏         | 47/2354 [00:00<00:16, 136.71batch/s]

Training batches on cuda:   3%|▎         | 66/2354 [00:00<00:15, 148.10batch/s]

Training batches on cuda:   4%|▎         | 85/2354 [00:00<00:14, 157.83batch/s]

Training batches on cuda:   4%|▍         | 101/2354 [00:00<00:14, 157.79batch/s]

Training batches on cuda:   5%|▌         | 123/2354 [00:00<00:13, 171.30batch/s]

Training batches on cuda:   6%|▌         | 147/2354 [00:00<00:11, 186.26batch/s]

Training batches on cuda:   7%|▋         | 167/2354 [00:00<00:11, 187.13batch/s]

Training batches on cuda:   8%|▊         | 187/2354 [00:01<00:11, 188.96batch/s]

Training batches on cuda:   9%|▉         | 207/2354 [00:01<00:11, 191.23batch/s]

Training batches on cuda:  10%|▉         | 227/2354 [00:01<00:11, 192.91batch/s]

Training batches on cuda:  10%|█         | 247/2354 [00:01<00:10, 194.29batch/s]

Training batches on cuda:  11%|█▏        | 269/2354 [00:01<00:10, 201.25batch/s]

Training batches on cuda:  12%|█▏        | 292/2354 [00:01<00:09, 207.42batch/s]

Training batches on cuda:  13%|█▎        | 314/2354 [00:01<00:09, 208.92batch/s]

Training batches on cuda:  14%|█▍        | 336/2354 [00:01<00:09, 209.27batch/s]

Training batches on cuda:  15%|█▌        | 359/2354 [00:01<00:09, 213.69batch/s]

Training batches on cuda:  16%|█▌        | 381/2354 [00:01<00:09, 215.39batch/s]

Training batches on cuda:  17%|█▋        | 403/2354 [00:02<00:10, 186.03batch/s]

Training batches on cuda:  18%|█▊        | 423/2354 [00:02<00:11, 172.63batch/s]

Training batches on cuda:  19%|█▊        | 441/2354 [00:02<00:11, 172.45batch/s]

Training batches on cuda:  19%|█▉        | 459/2354 [00:02<00:11, 171.17batch/s]

Training batches on cuda:  20%|██        | 478/2354 [00:02<00:10, 174.97batch/s]

Training batches on cuda:  21%|██        | 497/2354 [00:02<00:10, 176.80batch/s]

Training batches on cuda:  22%|██▏       | 517/2354 [00:02<00:10, 182.46batch/s]

Training batches on cuda:  23%|██▎       | 538/2354 [00:02<00:09, 189.13batch/s]

Training batches on cuda:  24%|██▎       | 559/2354 [00:02<00:09, 193.93batch/s]

Training batches on cuda:  25%|██▍       | 580/2354 [00:03<00:08, 198.29batch/s]

Training batches on cuda:  26%|██▌       | 601/2354 [00:03<00:08, 200.55batch/s]

Training batches on cuda:  26%|██▋       | 622/2354 [00:03<00:08, 194.97batch/s]

Training batches on cuda:  27%|██▋       | 642/2354 [00:03<00:08, 190.55batch/s]

Training batches on cuda:  28%|██▊       | 662/2354 [00:03<00:08, 191.95batch/s]

Training batches on cuda:  29%|██▉       | 684/2354 [00:03<00:08, 198.01batch/s]

Training batches on cuda:  30%|███       | 708/2354 [00:03<00:07, 207.34batch/s]

Training batches on cuda:  31%|███       | 729/2354 [00:03<00:08, 201.46batch/s]

Training batches on cuda:  32%|███▏      | 750/2354 [00:03<00:07, 201.12batch/s]

Training batches on cuda:  33%|███▎      | 771/2354 [00:04<00:07, 198.18batch/s]

Training batches on cuda:  34%|███▎      | 794/2354 [00:04<00:07, 204.39batch/s]

Training batches on cuda:  35%|███▍      | 815/2354 [00:04<00:08, 190.78batch/s]

Training batches on cuda:  35%|███▌      | 835/2354 [00:04<00:08, 173.22batch/s]

Training batches on cuda:  36%|███▌      | 853/2354 [00:04<00:08, 168.93batch/s]

Training batches on cuda:  37%|███▋      | 874/2354 [00:04<00:08, 178.59batch/s]

Training batches on cuda:  38%|███▊      | 896/2354 [00:04<00:07, 188.78batch/s]

Training batches on cuda:  39%|███▉      | 916/2354 [00:04<00:07, 184.54batch/s]

Training batches on cuda:  40%|███▉      | 938/2354 [00:04<00:07, 191.28batch/s]

Training batches on cuda:  41%|████      | 958/2354 [00:05<00:07, 188.23batch/s]

Training batches on cuda:  42%|████▏     | 978/2354 [00:05<00:07, 179.54batch/s]

Training batches on cuda:  42%|████▏     | 997/2354 [00:05<00:07, 180.52batch/s]

Training batches on cuda:  43%|████▎     | 1017/2354 [00:05<00:07, 182.95batch/s]

Training batches on cuda:  44%|████▍     | 1036/2354 [00:05<00:07, 184.23batch/s]

Training batches on cuda:  45%|████▍     | 1055/2354 [00:05<00:06, 185.59batch/s]

Training batches on cuda:  46%|████▌     | 1075/2354 [00:05<00:06, 189.58batch/s]

Training batches on cuda:  47%|████▋     | 1095/2354 [00:05<00:06, 190.06batch/s]

Training batches on cuda:  47%|████▋     | 1115/2354 [00:05<00:07, 175.36batch/s]

Training batches on cuda:  48%|████▊     | 1133/2354 [00:06<00:07, 171.90batch/s]

Training batches on cuda:  49%|████▉     | 1151/2354 [00:06<00:07, 169.92batch/s]

Training batches on cuda:  50%|████▉     | 1169/2354 [00:06<00:06, 172.35batch/s]

Training batches on cuda:  50%|█████     | 1187/2354 [00:06<00:06, 174.12batch/s]

Training batches on cuda:  51%|█████     | 1206/2354 [00:06<00:06, 177.59batch/s]

Training batches on cuda:  52%|█████▏    | 1224/2354 [00:06<00:06, 177.61batch/s]

Training batches on cuda:  53%|█████▎    | 1243/2354 [00:06<00:06, 179.21batch/s]

Training batches on cuda:  54%|█████▎    | 1262/2354 [00:06<00:06, 181.37batch/s]

Training batches on cuda:  54%|█████▍    | 1282/2354 [00:06<00:05, 185.60batch/s]

Training batches on cuda:  55%|█████▌    | 1302/2354 [00:06<00:05, 189.25batch/s]

Training batches on cuda:  56%|█████▌    | 1321/2354 [00:07<00:05, 187.50batch/s]

Training batches on cuda:  57%|█████▋    | 1341/2354 [00:07<00:05, 190.13batch/s]

Training batches on cuda:  58%|█████▊    | 1362/2354 [00:07<00:05, 193.32batch/s]

Training batches on cuda:  59%|█████▊    | 1382/2354 [00:07<00:05, 190.60batch/s]

Training batches on cuda:  60%|█████▉    | 1402/2354 [00:07<00:05, 190.24batch/s]

Training batches on cuda:  60%|██████    | 1422/2354 [00:07<00:04, 188.27batch/s]

Training batches on cuda:  61%|██████    | 1441/2354 [00:07<00:04, 186.50batch/s]

Training batches on cuda:  62%|██████▏   | 1461/2354 [00:07<00:04, 187.85batch/s]

Training batches on cuda:  63%|██████▎   | 1480/2354 [00:07<00:04, 187.09batch/s]

Training batches on cuda:  64%|██████▎   | 1499/2354 [00:07<00:04, 187.51batch/s]

Training batches on cuda:  65%|██████▍   | 1519/2354 [00:08<00:04, 189.01batch/s]

Training batches on cuda:  65%|██████▌   | 1540/2354 [00:08<00:04, 193.29batch/s]

Training batches on cuda:  66%|██████▋   | 1561/2354 [00:08<00:04, 196.25batch/s]

Training batches on cuda:  67%|██████▋   | 1582/2354 [00:08<00:03, 198.84batch/s]

Training batches on cuda:  68%|██████▊   | 1602/2354 [00:08<00:03, 199.04batch/s]

Training batches on cuda:  69%|██████▉   | 1622/2354 [00:08<00:03, 191.38batch/s]

Training batches on cuda:  70%|██████▉   | 1642/2354 [00:08<00:03, 191.56batch/s]

Training batches on cuda:  71%|███████   | 1662/2354 [00:08<00:03, 191.97batch/s]

Training batches on cuda:  71%|███████▏  | 1682/2354 [00:08<00:03, 188.28batch/s]

Training batches on cuda:  72%|███████▏  | 1701/2354 [00:09<00:03, 188.63batch/s]

Training batches on cuda:  73%|███████▎  | 1723/2354 [00:09<00:03, 195.93batch/s]

Training batches on cuda:  74%|███████▍  | 1743/2354 [00:09<00:03, 197.00batch/s]

Training batches on cuda:  75%|███████▍  | 1763/2354 [00:09<00:03, 194.67batch/s]

Training batches on cuda:  76%|███████▌  | 1783/2354 [00:09<00:03, 176.95batch/s]

Training batches on cuda:  77%|███████▋  | 1802/2354 [00:09<00:03, 173.96batch/s]

Training batches on cuda:  77%|███████▋  | 1820/2354 [00:09<00:03, 168.52batch/s]

Training batches on cuda:  78%|███████▊  | 1838/2354 [00:09<00:03, 169.14batch/s]

Training batches on cuda:  79%|███████▉  | 1856/2354 [00:09<00:02, 170.50batch/s]

Training batches on cuda:  80%|███████▉  | 1875/2354 [00:10<00:02, 174.42batch/s]

Training batches on cuda:  80%|████████  | 1894/2354 [00:10<00:02, 177.99batch/s]

Training batches on cuda:  81%|████████  | 1912/2354 [00:10<00:02, 176.12batch/s]

Training batches on cuda:  82%|████████▏ | 1930/2354 [00:10<00:02, 176.03batch/s]

Training batches on cuda:  83%|████████▎ | 1948/2354 [00:10<00:02, 172.09batch/s]

Training batches on cuda:  84%|████████▎ | 1967/2354 [00:10<00:02, 176.79batch/s]

Training batches on cuda:  84%|████████▍ | 1988/2354 [00:10<00:01, 183.05batch/s]

Training batches on cuda:  85%|████████▌ | 2007/2354 [00:10<00:01, 184.95batch/s]

Training batches on cuda:  86%|████████▌ | 2027/2354 [00:10<00:01, 186.98batch/s]

Training batches on cuda:  87%|████████▋ | 2046/2354 [00:10<00:01, 187.86batch/s]

Training batches on cuda:  88%|████████▊ | 2066/2354 [00:11<00:01, 191.04batch/s]

Training batches on cuda:  89%|████████▊ | 2086/2354 [00:11<00:01, 191.87batch/s]

Training batches on cuda:  89%|████████▉ | 2106/2354 [00:11<00:01, 180.78batch/s]

Training batches on cuda:  90%|█████████ | 2126/2354 [00:11<00:01, 184.00batch/s]

Training batches on cuda:  91%|█████████ | 2146/2354 [00:11<00:01, 187.18batch/s]

Training batches on cuda:  92%|█████████▏| 2165/2354 [00:11<00:01, 187.86batch/s]

Training batches on cuda:  93%|█████████▎| 2185/2354 [00:11<00:00, 190.26batch/s]

Training batches on cuda:  94%|█████████▎| 2205/2354 [00:11<00:00, 192.83batch/s]

Training batches on cuda:  95%|█████████▍| 2227/2354 [00:11<00:00, 198.18batch/s]

Training batches on cuda:  95%|█████████▌| 2248/2354 [00:12<00:00, 198.99batch/s]

Training batches on cuda:  96%|█████████▋| 2270/2354 [00:12<00:00, 203.38batch/s]

Training batches on cuda:  97%|█████████▋| 2291/2354 [00:12<00:00, 200.33batch/s]

Training batches on cuda:  98%|█████████▊| 2312/2354 [00:12<00:00, 194.48batch/s]

Training batches on cuda:  99%|█████████▉| 2332/2354 [00:12<00:00, 193.09batch/s]

Training batches on cuda: 100%|█████████▉| 2352/2354 [00:12<00:00, 192.22batch/s]

                                                                                 
Training epochs on cuda:  20%|██        | 2/10 [00:36<01:34, 11.87s/epoch, loss=0.0162, prev_loss=0.0162]
Training epochs on cuda:  30%|███       | 3/10 [00:36<01:25, 12.15s/epoch, loss=0.0162, prev_loss=0.0162]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 11/2354 [00:00<00:21, 107.88batch/s]

Training batches on cuda:   1%|▏         | 32/2354 [00:00<00:18, 125.32batch/s]

Training batches on cuda:   2%|▏         | 48/2354 [00:00<00:17, 133.48batch/s]

Training batches on cuda:   3%|▎         | 66/2354 [00:00<00:15, 144.21batch/s]

Training batches on cuda:   4%|▎         | 85/2354 [00:00<00:14, 155.19batch/s]

Training batches on cuda:   4%|▍         | 103/2354 [00:00<00:14, 159.65batch/s]

Training batches on cuda:   5%|▌         | 119/2354 [00:00<00:14, 159.19batch/s]

Training batches on cuda:   6%|▌         | 136/2354 [00:00<00:13, 160.42batch/s]

Training batches on cuda:   6%|▋         | 152/2354 [00:00<00:13, 159.83batch/s]

Training batches on cuda:   7%|▋         | 171/2354 [00:01<00:13, 166.61batch/s]

Training batches on cuda:   8%|▊         | 190/2354 [00:01<00:12, 171.66batch/s]

Training batches on cuda:   9%|▉         | 208/2354 [00:01<00:12, 173.66batch/s]

Training batches on cuda:  10%|▉         | 229/2354 [00:01<00:11, 183.09batch/s]

Training batches on cuda:  11%|█         | 251/2354 [00:01<00:10, 191.28batch/s]

Training batches on cuda:  12%|█▏        | 272/2354 [00:01<00:10, 195.60batch/s]

Training batches on cuda:  12%|█▏        | 292/2354 [00:01<00:12, 169.88batch/s]

Training batches on cuda:  13%|█▎        | 310/2354 [00:01<00:12, 166.41batch/s]

Training batches on cuda:  14%|█▍        | 329/2354 [00:01<00:11, 171.12batch/s]

Training batches on cuda:  15%|█▍        | 351/2354 [00:02<00:10, 182.83batch/s]

Training batches on cuda:  16%|█▌        | 373/2354 [00:02<00:10, 190.62batch/s]

Training batches on cuda:  17%|█▋        | 394/2354 [00:02<00:09, 196.01batch/s]

Training batches on cuda:  18%|█▊        | 415/2354 [00:02<00:09, 197.04batch/s]

Training batches on cuda:  18%|█▊        | 435/2354 [00:02<00:10, 190.26batch/s]

Training batches on cuda:  19%|█▉        | 455/2354 [00:02<00:10, 188.25batch/s]

Training batches on cuda:  20%|██        | 477/2354 [00:02<00:09, 195.25batch/s]

Training batches on cuda:  21%|██        | 499/2354 [00:02<00:09, 200.32batch/s]

Training batches on cuda:  22%|██▏       | 522/2354 [00:02<00:08, 205.85batch/s]

Training batches on cuda:  23%|██▎       | 544/2354 [00:02<00:08, 209.23batch/s]

Training batches on cuda:  24%|██▍       | 566/2354 [00:03<00:08, 211.07batch/s]

Training batches on cuda:  25%|██▌       | 590/2354 [00:03<00:08, 216.62batch/s]

Training batches on cuda:  26%|██▌       | 614/2354 [00:03<00:07, 222.38batch/s]

Training batches on cuda:  27%|██▋       | 637/2354 [00:03<00:07, 219.07batch/s]

Training batches on cuda:  28%|██▊       | 659/2354 [00:03<00:07, 218.43batch/s]

Training batches on cuda:  29%|██▉       | 682/2354 [00:03<00:07, 219.88batch/s]

Training batches on cuda:  30%|██▉       | 705/2354 [00:03<00:07, 219.96batch/s]

Training batches on cuda:  31%|███       | 728/2354 [00:03<00:07, 222.39batch/s]

Training batches on cuda:  32%|███▏      | 751/2354 [00:03<00:07, 224.01batch/s]

Training batches on cuda:  33%|███▎      | 774/2354 [00:03<00:07, 214.75batch/s]

Training batches on cuda:  34%|███▍      | 796/2354 [00:04<00:07, 210.43batch/s]

Training batches on cuda:  35%|███▍      | 818/2354 [00:04<00:07, 209.40batch/s]

Training batches on cuda:  36%|███▌      | 840/2354 [00:04<00:07, 206.80batch/s]

Training batches on cuda:  37%|███▋      | 861/2354 [00:04<00:07, 196.96batch/s]

Training batches on cuda:  37%|███▋      | 881/2354 [00:04<00:07, 192.53batch/s]

Training batches on cuda:  38%|███▊      | 901/2354 [00:04<00:07, 182.40batch/s]

Training batches on cuda:  39%|███▉      | 922/2354 [00:04<00:07, 189.10batch/s]

Training batches on cuda:  40%|████      | 942/2354 [00:04<00:07, 188.87batch/s]

Training batches on cuda:  41%|████      | 962/2354 [00:04<00:07, 189.32batch/s]

Training batches on cuda:  42%|████▏     | 982/2354 [00:05<00:07, 190.12batch/s]

Training batches on cuda:  43%|████▎     | 1003/2354 [00:05<00:06, 194.21batch/s]

Training batches on cuda:  44%|████▎     | 1025/2354 [00:05<00:06, 200.57batch/s]

Training batches on cuda:  45%|████▍     | 1048/2354 [00:05<00:06, 207.65batch/s]

Training batches on cuda:  45%|████▌     | 1071/2354 [00:05<00:06, 212.16batch/s]

Training batches on cuda:  46%|████▋     | 1093/2354 [00:05<00:06, 207.28batch/s]

Training batches on cuda:  47%|████▋     | 1115/2354 [00:05<00:05, 210.66batch/s]

Training batches on cuda:  48%|████▊     | 1138/2354 [00:05<00:05, 214.08batch/s]

Training batches on cuda:  49%|████▉     | 1160/2354 [00:05<00:05, 212.74batch/s]

Training batches on cuda:  50%|█████     | 1183/2354 [00:06<00:05, 215.27batch/s]

Training batches on cuda:  51%|█████     | 1206/2354 [00:06<00:05, 218.34batch/s]

Training batches on cuda:  52%|█████▏    | 1228/2354 [00:06<00:05, 205.46batch/s]

Training batches on cuda:  53%|█████▎    | 1249/2354 [00:06<00:05, 190.24batch/s]

Training batches on cuda:  54%|█████▍    | 1271/2354 [00:06<00:05, 197.60batch/s]

Training batches on cuda:  55%|█████▍    | 1294/2354 [00:06<00:05, 204.44batch/s]

Training batches on cuda:  56%|█████▌    | 1315/2354 [00:06<00:05, 188.91batch/s]

Training batches on cuda:  57%|█████▋    | 1337/2354 [00:06<00:05, 196.47batch/s]

Training batches on cuda:  58%|█████▊    | 1360/2354 [00:06<00:04, 203.62batch/s]

Training batches on cuda:  59%|█████▉    | 1383/2354 [00:07<00:04, 208.09batch/s]

Training batches on cuda:  60%|█████▉    | 1405/2354 [00:07<00:04, 210.67batch/s]

Training batches on cuda:  61%|██████    | 1427/2354 [00:07<00:04, 212.21batch/s]

Training batches on cuda:  62%|██████▏   | 1449/2354 [00:07<00:04, 211.89batch/s]

Training batches on cuda:  63%|██████▎   | 1472/2354 [00:07<00:04, 214.61batch/s]

Training batches on cuda:  64%|██████▎   | 1495/2354 [00:07<00:03, 218.26batch/s]

Training batches on cuda:  64%|██████▍   | 1518/2354 [00:07<00:03, 219.04batch/s]

Training batches on cuda:  65%|██████▌   | 1540/2354 [00:07<00:03, 217.99batch/s]

Training batches on cuda:  66%|██████▋   | 1562/2354 [00:07<00:03, 218.05batch/s]

Training batches on cuda:  67%|██████▋   | 1585/2354 [00:07<00:03, 219.76batch/s]

Training batches on cuda:  68%|██████▊   | 1608/2354 [00:08<00:03, 221.68batch/s]

Training batches on cuda:  69%|██████▉   | 1631/2354 [00:08<00:03, 218.94batch/s]

Training batches on cuda:  70%|███████   | 1653/2354 [00:08<00:03, 213.78batch/s]

Training batches on cuda:  71%|███████   | 1675/2354 [00:08<00:03, 204.42batch/s]

Training batches on cuda:  72%|███████▏  | 1696/2354 [00:08<00:03, 195.05batch/s]

Training batches on cuda:  73%|███████▎  | 1716/2354 [00:08<00:03, 191.83batch/s]

Training batches on cuda:  74%|███████▎  | 1736/2354 [00:08<00:03, 186.81batch/s]

Training batches on cuda:  75%|███████▍  | 1755/2354 [00:08<00:03, 183.69batch/s]

Training batches on cuda:  75%|███████▌  | 1774/2354 [00:08<00:03, 182.37batch/s]

Training batches on cuda:  76%|███████▌  | 1793/2354 [00:09<00:03, 182.54batch/s]

Training batches on cuda:  77%|███████▋  | 1812/2354 [00:09<00:02, 182.35batch/s]

Training batches on cuda:  78%|███████▊  | 1831/2354 [00:09<00:02, 184.58batch/s]

Training batches on cuda:  79%|███████▊  | 1852/2354 [00:09<00:02, 189.03batch/s]

Training batches on cuda:  79%|███████▉  | 1871/2354 [00:09<00:02, 186.42batch/s]

Training batches on cuda:  80%|████████  | 1890/2354 [00:09<00:02, 184.34batch/s]

Training batches on cuda:  81%|████████  | 1909/2354 [00:09<00:02, 183.56batch/s]

Training batches on cuda:  82%|████████▏ | 1928/2354 [00:09<00:02, 179.86batch/s]

Training batches on cuda:  83%|████████▎ | 1947/2354 [00:09<00:02, 181.07batch/s]

Training batches on cuda:  84%|████████▎ | 1966/2354 [00:09<00:02, 183.08batch/s]

Training batches on cuda:  84%|████████▍ | 1986/2354 [00:10<00:01, 187.73batch/s]

Training batches on cuda:  85%|████████▌ | 2008/2354 [00:10<00:01, 194.38batch/s]

Training batches on cuda:  86%|████████▌ | 2030/2354 [00:10<00:01, 199.49batch/s]

Training batches on cuda:  87%|████████▋ | 2051/2354 [00:10<00:01, 199.53batch/s]

Training batches on cuda:  88%|████████▊ | 2072/2354 [00:10<00:01, 199.69batch/s]

Training batches on cuda:  89%|████████▉ | 2094/2354 [00:10<00:01, 203.35batch/s]

Training batches on cuda:  90%|████████▉ | 2116/2354 [00:10<00:01, 206.90batch/s]

Training batches on cuda:  91%|█████████ | 2138/2354 [00:10<00:01, 209.32batch/s]

Training batches on cuda:  92%|█████████▏| 2160/2354 [00:10<00:00, 211.49batch/s]

Training batches on cuda:  93%|█████████▎| 2182/2354 [00:11<00:00, 202.76batch/s]

Training batches on cuda:  94%|█████████▎| 2205/2354 [00:11<00:00, 209.54batch/s]

Training batches on cuda:  95%|█████████▍| 2228/2354 [00:11<00:00, 213.24batch/s]

Training batches on cuda:  96%|█████████▌| 2250/2354 [00:11<00:00, 215.19batch/s]

Training batches on cuda:  97%|█████████▋| 2272/2354 [00:11<00:00, 210.61batch/s]

Training batches on cuda:  97%|█████████▋| 2294/2354 [00:11<00:00, 196.47batch/s]

Training batches on cuda:  98%|█████████▊| 2314/2354 [00:11<00:00, 188.05batch/s]

Training batches on cuda:  99%|█████████▉| 2334/2354 [00:11<00:00, 187.07batch/s]

Training batches on cuda: 100%|██████████| 2354/2354 [00:11<00:00, 185.98batch/s]

                                                                                 
Training epochs on cuda:  30%|███       | 3/10 [00:49<01:25, 12.15s/epoch, loss=0.0162, prev_loss=0.0162]
Training epochs on cuda:  40%|████      | 4/10 [00:49<01:12, 12.15s/epoch, loss=0.0162, prev_loss=0.0162]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 14/2354 [00:00<00:17, 135.17batch/s]

Training batches on cuda:   1%|▏         | 32/2354 [00:00<00:15, 145.24batch/s]

Training batches on cuda:   2%|▏         | 51/2354 [00:00<00:14, 156.15batch/s]

Training batches on cuda:   3%|▎         | 72/2354 [00:00<00:13, 167.85batch/s]

Training batches on cuda:   4%|▍         | 92/2354 [00:00<00:12, 175.11batch/s]

Training batches on cuda:   5%|▍         | 115/2354 [00:00<00:11, 186.95batch/s]

Training batches on cuda:   6%|▌         | 134/2354 [00:00<00:11, 186.19batch/s]

Training batches on cuda:   6%|▋         | 152/2354 [00:00<00:12, 177.09batch/s]

Training batches on cuda:   7%|▋         | 173/2354 [00:00<00:11, 184.03batch/s]

Training batches on cuda:   8%|▊         | 193/2354 [00:01<00:11, 186.77batch/s]

Training batches on cuda:   9%|▉         | 212/2354 [00:01<00:11, 182.35batch/s]

Training batches on cuda:  10%|▉         | 231/2354 [00:01<00:11, 183.70batch/s]

Training batches on cuda:  11%|█         | 251/2354 [00:01<00:11, 186.24batch/s]

Training batches on cuda:  12%|█▏        | 271/2354 [00:01<00:10, 190.08batch/s]

Training batches on cuda:  12%|█▏        | 291/2354 [00:01<00:11, 185.82batch/s]

Training batches on cuda:  13%|█▎        | 311/2354 [00:01<00:10, 189.67batch/s]

Training batches on cuda:  14%|█▍        | 334/2354 [00:01<00:10, 197.43batch/s]

Training batches on cuda:  15%|█▌        | 355/2354 [00:01<00:10, 199.17batch/s]

Training batches on cuda:  16%|█▌        | 376/2354 [00:01<00:09, 201.14batch/s]

Training batches on cuda:  17%|█▋        | 397/2354 [00:02<00:09, 202.86batch/s]

Training batches on cuda:  18%|█▊        | 419/2354 [00:02<00:09, 206.58batch/s]

Training batches on cuda:  19%|█▊        | 440/2354 [00:02<00:09, 206.67batch/s]

Training batches on cuda:  20%|█▉        | 461/2354 [00:02<00:09, 204.75batch/s]

Training batches on cuda:  20%|██        | 482/2354 [00:02<00:09, 203.02batch/s]

Training batches on cuda:  21%|██▏       | 503/2354 [00:02<00:09, 200.61batch/s]

Training batches on cuda:  22%|██▏       | 524/2354 [00:02<00:09, 198.47batch/s]

Training batches on cuda:  23%|██▎       | 544/2354 [00:02<00:09, 197.70batch/s]

Training batches on cuda:  24%|██▍       | 564/2354 [00:02<00:09, 196.13batch/s]

Training batches on cuda:  25%|██▍       | 584/2354 [00:03<00:09, 196.23batch/s]

Training batches on cuda:  26%|██▌       | 606/2354 [00:03<00:08, 201.39batch/s]

Training batches on cuda:  27%|██▋       | 628/2354 [00:03<00:08, 206.39batch/s]

Training batches on cuda:  28%|██▊       | 651/2354 [00:03<00:08, 211.74batch/s]

Training batches on cuda:  29%|██▊       | 674/2354 [00:03<00:07, 214.77batch/s]

Training batches on cuda:  30%|██▉       | 697/2354 [00:03<00:07, 217.22batch/s]

Training batches on cuda:  31%|███       | 719/2354 [00:03<00:07, 217.81batch/s]

Training batches on cuda:  31%|███▏      | 741/2354 [00:03<00:07, 209.64batch/s]

Training batches on cuda:  32%|███▏      | 763/2354 [00:03<00:07, 208.63batch/s]

Training batches on cuda:  33%|███▎      | 784/2354 [00:03<00:07, 206.64batch/s]

Training batches on cuda:  34%|███▍      | 805/2354 [00:04<00:07, 205.29batch/s]

Training batches on cuda:  35%|███▌      | 826/2354 [00:04<00:07, 201.97batch/s]

Training batches on cuda:  36%|███▌      | 847/2354 [00:04<00:07, 198.62batch/s]

Training batches on cuda:  37%|███▋      | 867/2354 [00:04<00:07, 197.18batch/s]

Training batches on cuda:  38%|███▊      | 887/2354 [00:04<00:07, 196.90batch/s]

Training batches on cuda:  39%|███▊      | 907/2354 [00:04<00:08, 169.16batch/s]

Training batches on cuda:  39%|███▉      | 927/2354 [00:04<00:08, 175.99batch/s]

Training batches on cuda:  40%|████      | 947/2354 [00:04<00:07, 181.59batch/s]

Training batches on cuda:  41%|████      | 966/2354 [00:04<00:07, 183.62batch/s]

Training batches on cuda:  42%|████▏     | 989/2354 [00:05<00:06, 195.01batch/s]

Training batches on cuda:  43%|████▎     | 1015/2354 [00:05<00:06, 209.53batch/s]

Training batches on cuda:  44%|████▍     | 1041/2354 [00:05<00:05, 220.47batch/s]

Training batches on cuda:  45%|████▌     | 1067/2354 [00:05<00:05, 230.41batch/s]

Training batches on cuda:  46%|████▋     | 1093/2354 [00:05<00:05, 238.07batch/s]

Training batches on cuda:  48%|████▊     | 1119/2354 [00:05<00:05, 242.92batch/s]

Training batches on cuda:  49%|████▊     | 1145/2354 [00:05<00:04, 246.01batch/s]

Training batches on cuda:  50%|████▉     | 1171/2354 [00:05<00:04, 248.80batch/s]

Training batches on cuda:  51%|█████     | 1197/2354 [00:05<00:04, 251.02batch/s]

Training batches on cuda:  52%|█████▏    | 1223/2354 [00:05<00:04, 251.48batch/s]

Training batches on cuda:  53%|█████▎    | 1249/2354 [00:06<00:04, 253.17batch/s]

Training batches on cuda:  54%|█████▍    | 1275/2354 [00:06<00:04, 250.84batch/s]

Training batches on cuda:  55%|█████▌    | 1301/2354 [00:06<00:04, 252.51batch/s]

Training batches on cuda:  56%|█████▋    | 1327/2354 [00:06<00:04, 253.49batch/s]

Training batches on cuda:  57%|█████▋    | 1353/2354 [00:06<00:03, 254.73batch/s]

Training batches on cuda:  59%|█████▊    | 1379/2354 [00:06<00:03, 250.39batch/s]

Training batches on cuda:  60%|█████▉    | 1405/2354 [00:06<00:04, 227.59batch/s]

Training batches on cuda:  61%|██████    | 1429/2354 [00:06<00:04, 213.49batch/s]

Training batches on cuda:  62%|██████▏   | 1451/2354 [00:06<00:04, 212.06batch/s]

Training batches on cuda:  63%|██████▎   | 1473/2354 [00:07<00:04, 211.83batch/s]

Training batches on cuda:  64%|██████▎   | 1495/2354 [00:07<00:04, 207.70batch/s]

Training batches on cuda:  64%|██████▍   | 1516/2354 [00:07<00:04, 204.46batch/s]

Training batches on cuda:  65%|██████▌   | 1537/2354 [00:07<00:03, 205.46batch/s]

Training batches on cuda:  66%|██████▌   | 1558/2354 [00:07<00:03, 205.99batch/s]

Training batches on cuda:  67%|██████▋   | 1579/2354 [00:07<00:03, 206.53batch/s]

Training batches on cuda:  68%|██████▊   | 1600/2354 [00:07<00:03, 205.32batch/s]

Training batches on cuda:  69%|██████▉   | 1621/2354 [00:07<00:03, 201.59batch/s]

Training batches on cuda:  70%|██████▉   | 1642/2354 [00:07<00:03, 199.68batch/s]

Training batches on cuda:  71%|███████   | 1664/2354 [00:08<00:03, 203.11batch/s]

Training batches on cuda:  72%|███████▏  | 1685/2354 [00:08<00:03, 203.59batch/s]

Training batches on cuda:  73%|███████▎  | 1707/2354 [00:08<00:03, 205.61batch/s]

Training batches on cuda:  73%|███████▎  | 1729/2354 [00:08<00:02, 209.45batch/s]

Training batches on cuda:  74%|███████▍  | 1752/2354 [00:08<00:02, 211.84batch/s]

Training batches on cuda:  75%|███████▌  | 1775/2354 [00:08<00:02, 214.87batch/s]

Training batches on cuda:  76%|███████▋  | 1798/2354 [00:08<00:02, 216.35batch/s]

Training batches on cuda:  77%|███████▋  | 1820/2354 [00:08<00:02, 213.95batch/s]

Training batches on cuda:  78%|███████▊  | 1842/2354 [00:08<00:02, 212.31batch/s]

Training batches on cuda:  79%|███████▉  | 1864/2354 [00:08<00:02, 213.08batch/s]

Training batches on cuda:  80%|████████  | 1886/2354 [00:09<00:02, 214.03batch/s]

Training batches on cuda:  81%|████████  | 1909/2354 [00:09<00:02, 217.07batch/s]

Training batches on cuda:  82%|████████▏ | 1931/2354 [00:09<00:01, 216.49batch/s]

Training batches on cuda:  83%|████████▎ | 1953/2354 [00:09<00:01, 206.42batch/s]

Training batches on cuda:  84%|████████▍ | 1974/2354 [00:09<00:01, 205.66batch/s]

Training batches on cuda:  85%|████████▍ | 1995/2354 [00:09<00:01, 204.68batch/s]

Training batches on cuda:  86%|████████▌ | 2016/2354 [00:09<00:01, 178.76batch/s]

Training batches on cuda:  87%|████████▋ | 2037/2354 [00:09<00:01, 186.31batch/s]

Training batches on cuda:  87%|████████▋ | 2057/2354 [00:09<00:01, 187.43batch/s]

Training batches on cuda:  88%|████████▊ | 2080/2354 [00:10<00:01, 196.39batch/s]

Training batches on cuda:  89%|████████▉ | 2102/2354 [00:10<00:01, 201.03batch/s]

Training batches on cuda:  90%|█████████ | 2123/2354 [00:10<00:01, 198.38batch/s]

Training batches on cuda:  91%|█████████ | 2144/2354 [00:10<00:01, 195.37batch/s]

Training batches on cuda:  92%|█████████▏| 2165/2354 [00:10<00:00, 197.73batch/s]

Training batches on cuda:  93%|█████████▎| 2185/2354 [00:10<00:00, 191.98batch/s]

Training batches on cuda:  94%|█████████▎| 2206/2354 [00:10<00:00, 195.73batch/s]

Training batches on cuda:  95%|█████████▍| 2228/2354 [00:10<00:00, 200.53batch/s]

Training batches on cuda:  96%|█████████▌| 2252/2354 [00:10<00:00, 209.09batch/s]

Training batches on cuda:  97%|█████████▋| 2274/2354 [00:10<00:00, 208.69batch/s]

Training batches on cuda:  98%|█████████▊| 2297/2354 [00:11<00:00, 212.20batch/s]

Training batches on cuda:  99%|█████████▊| 2319/2354 [00:11<00:00, 213.21batch/s]

Training batches on cuda:  99%|█████████▉| 2341/2354 [00:11<00:00, 209.80batch/s]

                                                                                 
Training epochs on cuda:  40%|████      | 4/10 [01:00<01:12, 12.15s/epoch, loss=0.0162, prev_loss=0.0162]
Training epochs on cuda:  50%|█████     | 5/10 [01:00<00:59, 11.98s/epoch, loss=0.0162, prev_loss=0.0162]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 9/2354 [00:00<00:26, 88.42batch/s]

Training batches on cuda:   1%|          | 28/2354 [00:00<00:22, 105.04batch/s]

Training batches on cuda:   2%|▏         | 49/2354 [00:00<00:18, 123.48batch/s]

Training batches on cuda:   3%|▎         | 71/2354 [00:00<00:16, 141.81batch/s]

Training batches on cuda:   4%|▍         | 89/2354 [00:00<00:14, 151.22batch/s]

Training batches on cuda:   5%|▍         | 110/2354 [00:00<00:13, 163.42batch/s]

Training batches on cuda:   6%|▌         | 133/2354 [00:00<00:12, 178.46batch/s]

Training batches on cuda:   7%|▋         | 155/2354 [00:00<00:11, 187.78batch/s]

Training batches on cuda:   8%|▊         | 178/2354 [00:00<00:11, 196.48batch/s]

Training batches on cuda:   8%|▊         | 200/2354 [00:01<00:10, 201.07batch/s]

Training batches on cuda:   9%|▉         | 221/2354 [00:01<00:11, 193.54batch/s]

Training batches on cuda:  10%|█         | 244/2354 [00:01<00:10, 201.29batch/s]

Training batches on cuda:  11%|█▏        | 265/2354 [00:01<00:10, 190.46batch/s]

Training batches on cuda:  12%|█▏        | 285/2354 [00:01<00:11, 185.22batch/s]

Training batches on cuda:  13%|█▎        | 304/2354 [00:01<00:11, 182.65batch/s]

Training batches on cuda:  14%|█▎        | 323/2354 [00:01<00:11, 179.10batch/s]

Training batches on cuda:  15%|█▍        | 343/2354 [00:01<00:10, 183.11batch/s]

Training batches on cuda:  15%|█▌        | 364/2354 [00:01<00:10, 189.33batch/s]

Training batches on cuda:  16%|█▋        | 384/2354 [00:02<00:10, 188.73batch/s]

Training batches on cuda:  17%|█▋        | 405/2354 [00:02<00:10, 192.46batch/s]

Training batches on cuda:  18%|█▊        | 425/2354 [00:02<00:10, 190.61batch/s]

Training batches on cuda:  19%|█▉        | 445/2354 [00:02<00:10, 188.63batch/s]

Training batches on cuda:  20%|█▉        | 464/2354 [00:02<00:10, 182.92batch/s]

Training batches on cuda:  21%|██        | 483/2354 [00:02<00:10, 180.86batch/s]

Training batches on cuda:  21%|██▏       | 504/2354 [00:02<00:09, 186.18batch/s]

Training batches on cuda:  22%|██▏       | 523/2354 [00:02<00:10, 181.94batch/s]

Training batches on cuda:  23%|██▎       | 542/2354 [00:02<00:10, 175.66batch/s]

Training batches on cuda:  24%|██▍       | 562/2354 [00:02<00:09, 180.52batch/s]

Training batches on cuda:  25%|██▍       | 583/2354 [00:03<00:09, 187.41batch/s]

Training batches on cuda:  26%|██▌       | 602/2354 [00:03<00:09, 176.18batch/s]

Training batches on cuda:  26%|██▋       | 620/2354 [00:03<00:10, 171.21batch/s]

Training batches on cuda:  27%|██▋       | 638/2354 [00:03<00:10, 170.64batch/s]

Training batches on cuda:  28%|██▊       | 656/2354 [00:03<00:10, 168.72batch/s]

Training batches on cuda:  29%|██▉       | 678/2354 [00:03<00:09, 180.54batch/s]

Training batches on cuda:  30%|██▉       | 701/2354 [00:03<00:08, 192.35batch/s]

Training batches on cuda:  31%|███       | 727/2354 [00:03<00:07, 207.62batch/s]

Training batches on cuda:  32%|███▏      | 754/2354 [00:03<00:07, 221.31batch/s]

Training batches on cuda:  33%|███▎      | 780/2354 [00:04<00:06, 231.22batch/s]

Training batches on cuda:  34%|███▍      | 807/2354 [00:04<00:06, 239.79batch/s]

Training batches on cuda:  35%|███▌      | 832/2354 [00:04<00:06, 240.00batch/s]

Training batches on cuda:  36%|███▋      | 857/2354 [00:04<00:06, 234.45batch/s]

Training batches on cuda:  37%|███▋      | 881/2354 [00:04<00:06, 228.63batch/s]

Training batches on cuda:  38%|███▊      | 905/2354 [00:04<00:06, 213.52batch/s]

Training batches on cuda:  39%|███▉      | 927/2354 [00:04<00:06, 204.96batch/s]

Training batches on cuda:  40%|████      | 948/2354 [00:04<00:07, 190.03batch/s]

Training batches on cuda:  41%|████      | 968/2354 [00:04<00:07, 175.78batch/s]

Training batches on cuda:  42%|████▏     | 988/2354 [00:05<00:07, 180.64batch/s]

Training batches on cuda:  43%|████▎     | 1008/2354 [00:05<00:07, 184.99batch/s]

Training batches on cuda:  44%|████▍     | 1030/2354 [00:05<00:06, 193.21batch/s]

Training batches on cuda:  45%|████▍     | 1054/2354 [00:05<00:06, 203.51batch/s]

Training batches on cuda:  46%|████▌     | 1076/2354 [00:05<00:06, 207.06batch/s]

Training batches on cuda:  47%|████▋     | 1099/2354 [00:05<00:05, 212.86batch/s]

Training batches on cuda:  48%|████▊     | 1121/2354 [00:05<00:05, 212.91batch/s]

Training batches on cuda:  49%|████▊     | 1143/2354 [00:05<00:05, 204.33batch/s]

Training batches on cuda:  49%|████▉     | 1164/2354 [00:05<00:05, 199.60batch/s]

Training batches on cuda:  50%|█████     | 1185/2354 [00:06<00:05, 199.68batch/s]

Training batches on cuda:  51%|█████     | 1206/2354 [00:06<00:05, 195.92batch/s]

Training batches on cuda:  52%|█████▏    | 1227/2354 [00:06<00:05, 198.95batch/s]

Training batches on cuda:  53%|█████▎    | 1250/2354 [00:06<00:05, 206.92batch/s]

Training batches on cuda:  54%|█████▍    | 1271/2354 [00:06<00:05, 206.12batch/s]

Training batches on cuda:  55%|█████▍    | 1294/2354 [00:06<00:05, 211.39batch/s]

Training batches on cuda:  56%|█████▌    | 1317/2354 [00:06<00:04, 214.79batch/s]

Training batches on cuda:  57%|█████▋    | 1339/2354 [00:06<00:04, 214.56batch/s]

Training batches on cuda:  58%|█████▊    | 1361/2354 [00:06<00:04, 211.49batch/s]

Training batches on cuda:  59%|█████▉    | 1383/2354 [00:06<00:04, 210.21batch/s]

Training batches on cuda:  60%|█████▉    | 1405/2354 [00:07<00:04, 210.63batch/s]

Training batches on cuda:  61%|██████    | 1427/2354 [00:07<00:04, 205.52batch/s]

Training batches on cuda:  62%|██████▏   | 1448/2354 [00:07<00:04, 203.19batch/s]

Training batches on cuda:  62%|██████▏   | 1469/2354 [00:07<00:04, 204.65batch/s]

Training batches on cuda:  63%|██████▎   | 1490/2354 [00:07<00:04, 193.70batch/s]

Training batches on cuda:  64%|██████▍   | 1510/2354 [00:07<00:04, 187.60batch/s]

Training batches on cuda:  65%|██████▍   | 1529/2354 [00:07<00:04, 186.89batch/s]

Training batches on cuda:  66%|██████▌   | 1548/2354 [00:07<00:04, 180.40batch/s]

Training batches on cuda:  67%|██████▋   | 1568/2354 [00:07<00:04, 185.54batch/s]

Training batches on cuda:  68%|██████▊   | 1589/2354 [00:08<00:03, 191.40batch/s]

Training batches on cuda:  68%|██████▊   | 1609/2354 [00:08<00:04, 168.96batch/s]

Training batches on cuda:  69%|██████▉   | 1631/2354 [00:08<00:04, 179.61batch/s]

Training batches on cuda:  70%|███████   | 1651/2354 [00:08<00:03, 183.64batch/s]

Training batches on cuda:  71%|███████   | 1670/2354 [00:08<00:03, 171.71batch/s]

Training batches on cuda:  72%|███████▏  | 1688/2354 [00:08<00:03, 170.85batch/s]

Training batches on cuda:  72%|███████▏  | 1706/2354 [00:08<00:03, 165.26batch/s]

Training batches on cuda:  73%|███████▎  | 1724/2354 [00:08<00:03, 168.93batch/s]

Training batches on cuda:  74%|███████▍  | 1742/2354 [00:08<00:03, 167.23batch/s]

Training batches on cuda:  75%|███████▍  | 1759/2354 [00:09<00:03, 161.72batch/s]

Training batches on cuda:  76%|███████▌  | 1778/2354 [00:09<00:03, 168.37batch/s]

Training batches on cuda:  76%|███████▋  | 1797/2354 [00:09<00:03, 172.40batch/s]

Training batches on cuda:  77%|███████▋  | 1816/2354 [00:09<00:03, 177.01batch/s]

Training batches on cuda:  78%|███████▊  | 1834/2354 [00:09<00:02, 173.34batch/s]

Training batches on cuda:  79%|███████▊  | 1853/2354 [00:09<00:02, 175.62batch/s]

Training batches on cuda:  79%|███████▉  | 1871/2354 [00:09<00:02, 174.94batch/s]

Training batches on cuda:  80%|████████  | 1891/2354 [00:09<00:02, 179.74batch/s]

Training batches on cuda:  81%|████████  | 1910/2354 [00:09<00:02, 182.31batch/s]

Training batches on cuda:  82%|████████▏ | 1929/2354 [00:10<00:02, 181.94batch/s]

Training batches on cuda:  83%|████████▎ | 1950/2354 [00:10<00:02, 188.36batch/s]

Training batches on cuda:  84%|████████▎ | 1969/2354 [00:10<00:02, 182.97batch/s]

Training batches on cuda:  84%|████████▍ | 1988/2354 [00:10<00:01, 183.51batch/s]

Training batches on cuda:  85%|████████▌ | 2009/2354 [00:10<00:01, 188.65batch/s]

Training batches on cuda:  86%|████████▌ | 2030/2354 [00:10<00:01, 194.07batch/s]

Training batches on cuda:  87%|████████▋ | 2050/2354 [00:10<00:01, 191.37batch/s]

Training batches on cuda:  88%|████████▊ | 2071/2354 [00:10<00:01, 193.86batch/s]

Training batches on cuda:  89%|████████▉ | 2091/2354 [00:10<00:01, 193.84batch/s]

Training batches on cuda:  90%|████████▉ | 2111/2354 [00:10<00:01, 195.07batch/s]

Training batches on cuda:  91%|█████████ | 2131/2354 [00:11<00:01, 194.95batch/s]

Training batches on cuda:  91%|█████████▏| 2151/2354 [00:11<00:01, 194.87batch/s]

Training batches on cuda:  92%|█████████▏| 2171/2354 [00:11<00:00, 188.25batch/s]

Training batches on cuda:  93%|█████████▎| 2193/2354 [00:11<00:00, 196.24batch/s]

Training batches on cuda:  94%|█████████▍| 2216/2354 [00:11<00:00, 203.72batch/s]

Training batches on cuda:  95%|█████████▌| 2239/2354 [00:11<00:00, 209.26batch/s]

Training batches on cuda:  96%|█████████▌| 2262/2354 [00:11<00:00, 213.27batch/s]

Training batches on cuda:  97%|█████████▋| 2286/2354 [00:11<00:00, 219.77batch/s]

Training batches on cuda:  98%|█████████▊| 2311/2354 [00:11<00:00, 227.00batch/s]

Training batches on cuda:  99%|█████████▉| 2336/2354 [00:11<00:00, 232.86batch/s]

                                                                                 
Training epochs on cuda:  50%|█████     | 5/10 [01:12<00:59, 11.98s/epoch, loss=0.0162, prev_loss=0.0162]
Training epochs on cuda:  60%|██████    | 6/10 [01:12<00:48, 12.08s/epoch, loss=0.0162, prev_loss=0.0162]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 14/2354 [00:00<00:16, 138.55batch/s]

Training batches on cuda:   2%|▏         | 36/2354 [00:00<00:14, 155.78batch/s]

Training batches on cuda:   3%|▎         | 59/2354 [00:00<00:13, 170.80batch/s]

Training batches on cuda:   3%|▎         | 82/2354 [00:00<00:12, 183.17batch/s]

Training batches on cuda:   4%|▍         | 105/2354 [00:00<00:11, 193.32batch/s]

Training batches on cuda:   5%|▌         | 128/2354 [00:00<00:11, 201.56batch/s]

Training batches on cuda:   6%|▋         | 150/2354 [00:00<00:10, 204.89batch/s]

Training batches on cuda:   7%|▋         | 170/2354 [00:00<00:10, 201.22batch/s]

Training batches on cuda:   8%|▊         | 190/2354 [00:00<00:10, 200.27batch/s]

Training batches on cuda:   9%|▉         | 211/2354 [00:01<00:10, 202.40batch/s]

Training batches on cuda:  10%|▉         | 232/2354 [00:01<00:10, 204.34batch/s]

Training batches on cuda:  11%|█         | 254/2354 [00:01<00:10, 205.98batch/s]

Training batches on cuda:  12%|█▏        | 275/2354 [00:01<00:10, 205.06batch/s]

Training batches on cuda:  13%|█▎        | 297/2354 [00:01<00:09, 207.22batch/s]

Training batches on cuda:  14%|█▎        | 319/2354 [00:01<00:09, 210.75batch/s]

Training batches on cuda:  14%|█▍        | 341/2354 [00:01<00:09, 204.65batch/s]

Training batches on cuda:  15%|█▌        | 362/2354 [00:01<00:10, 197.99batch/s]

Training batches on cuda:  16%|█▌        | 382/2354 [00:01<00:10, 185.42batch/s]

Training batches on cuda:  17%|█▋        | 401/2354 [00:01<00:10, 185.67batch/s]

Training batches on cuda:  18%|█▊        | 420/2354 [00:02<00:10, 179.67batch/s]

Training batches on cuda:  19%|█▊        | 439/2354 [00:02<00:10, 181.47batch/s]

Training batches on cuda:  20%|█▉        | 462/2354 [00:02<00:09, 193.63batch/s]

Training batches on cuda:  21%|██        | 483/2354 [00:02<00:09, 195.51batch/s]

Training batches on cuda:  21%|██▏       | 503/2354 [00:02<00:09, 192.38batch/s]

Training batches on cuda:  22%|██▏       | 523/2354 [00:02<00:09, 185.67batch/s]

Training batches on cuda:  23%|██▎       | 542/2354 [00:02<00:09, 185.75batch/s]

Training batches on cuda:  24%|██▍       | 561/2354 [00:02<00:09, 185.67batch/s]

Training batches on cuda:  25%|██▍       | 581/2354 [00:02<00:09, 187.69batch/s]

Training batches on cuda:  25%|██▌       | 600/2354 [00:03<00:09, 186.86batch/s]

Training batches on cuda:  26%|██▋       | 622/2354 [00:03<00:08, 194.50batch/s]

Training batches on cuda:  27%|██▋       | 642/2354 [00:03<00:08, 192.63batch/s]

Training batches on cuda:  28%|██▊       | 662/2354 [00:03<00:08, 191.69batch/s]

Training batches on cuda:  29%|██▉       | 682/2354 [00:03<00:08, 192.01batch/s]

Training batches on cuda:  30%|██▉       | 702/2354 [00:03<00:08, 192.45batch/s]

Training batches on cuda:  31%|███       | 722/2354 [00:03<00:09, 180.79batch/s]

Training batches on cuda:  31%|███▏      | 741/2354 [00:03<00:08, 182.34batch/s]

Training batches on cuda:  32%|███▏      | 760/2354 [00:03<00:09, 172.73batch/s]

Training batches on cuda:  33%|███▎      | 781/2354 [00:04<00:08, 182.24batch/s]

Training batches on cuda:  34%|███▍      | 803/2354 [00:04<00:08, 191.12batch/s]

Training batches on cuda:  35%|███▌      | 826/2354 [00:04<00:07, 199.54batch/s]

Training batches on cuda:  36%|███▌      | 849/2354 [00:04<00:07, 205.71batch/s]

Training batches on cuda:  37%|███▋      | 871/2354 [00:04<00:07, 208.26batch/s]

Training batches on cuda:  38%|███▊      | 893/2354 [00:04<00:07, 203.90batch/s]

Training batches on cuda:  39%|███▉      | 916/2354 [00:04<00:06, 209.02batch/s]

Training batches on cuda:  40%|███▉      | 938/2354 [00:04<00:06, 203.34batch/s]

Training batches on cuda:  41%|████      | 959/2354 [00:04<00:07, 198.14batch/s]

Training batches on cuda:  42%|████▏     | 981/2354 [00:04<00:06, 201.64batch/s]

Training batches on cuda:  43%|████▎     | 1002/2354 [00:05<00:06, 201.50batch/s]

Training batches on cuda:  44%|████▎     | 1024/2354 [00:05<00:06, 205.37batch/s]

Training batches on cuda:  44%|████▍     | 1046/2354 [00:05<00:06, 209.41batch/s]

Training batches on cuda:  45%|████▌     | 1068/2354 [00:05<00:06, 212.03batch/s]

Training batches on cuda:  46%|████▋     | 1090/2354 [00:05<00:05, 211.11batch/s]

Training batches on cuda:  47%|████▋     | 1113/2354 [00:05<00:05, 213.38batch/s]

Training batches on cuda:  48%|████▊     | 1136/2354 [00:05<00:05, 216.87batch/s]

Training batches on cuda:  49%|████▉     | 1159/2354 [00:05<00:05, 219.07batch/s]

Training batches on cuda:  50%|█████     | 1181/2354 [00:05<00:05, 211.28batch/s]

Training batches on cuda:  51%|█████     | 1204/2354 [00:06<00:05, 215.32batch/s]

Training batches on cuda:  52%|█████▏    | 1226/2354 [00:06<00:05, 216.02batch/s]

Training batches on cuda:  53%|█████▎    | 1249/2354 [00:06<00:05, 218.04batch/s]

Training batches on cuda:  54%|█████▍    | 1272/2354 [00:06<00:04, 220.57batch/s]

Training batches on cuda:  55%|█████▌    | 1295/2354 [00:06<00:04, 220.87batch/s]

Training batches on cuda:  56%|█████▌    | 1318/2354 [00:06<00:04, 222.75batch/s]

Training batches on cuda:  57%|█████▋    | 1341/2354 [00:06<00:04, 224.45batch/s]

Training batches on cuda:  58%|█████▊    | 1364/2354 [00:06<00:04, 225.56batch/s]

Training batches on cuda:  59%|█████▉    | 1387/2354 [00:06<00:04, 226.36batch/s]

Training batches on cuda:  60%|█████▉    | 1410/2354 [00:06<00:04, 226.34batch/s]

Training batches on cuda:  61%|██████    | 1433/2354 [00:07<00:04, 226.98batch/s]

Training batches on cuda:  62%|██████▏   | 1456/2354 [00:07<00:03, 227.63batch/s]

Training batches on cuda:  63%|██████▎   | 1479/2354 [00:07<00:03, 224.48batch/s]

Training batches on cuda:  64%|██████▍   | 1502/2354 [00:07<00:03, 220.81batch/s]

Training batches on cuda:  65%|██████▍   | 1525/2354 [00:07<00:03, 214.10batch/s]

Training batches on cuda:  66%|██████▌   | 1547/2354 [00:07<00:03, 205.94batch/s]

Training batches on cuda:  67%|██████▋   | 1568/2354 [00:07<00:03, 201.39batch/s]

Training batches on cuda:  68%|██████▊   | 1589/2354 [00:07<00:03, 196.73batch/s]

Training batches on cuda:  68%|██████▊   | 1609/2354 [00:07<00:03, 194.87batch/s]

Training batches on cuda:  69%|██████▉   | 1631/2354 [00:08<00:03, 200.28batch/s]

Training batches on cuda:  70%|███████   | 1652/2354 [00:08<00:03, 200.05batch/s]

Training batches on cuda:  71%|███████   | 1673/2354 [00:08<00:03, 200.59batch/s]

Training batches on cuda:  72%|███████▏  | 1694/2354 [00:08<00:03, 192.43batch/s]

Training batches on cuda:  73%|███████▎  | 1714/2354 [00:08<00:03, 192.34batch/s]

Training batches on cuda:  74%|███████▎  | 1734/2354 [00:08<00:03, 192.98batch/s]

Training batches on cuda:  75%|███████▍  | 1756/2354 [00:08<00:03, 198.87batch/s]

Training batches on cuda:  76%|███████▌  | 1779/2354 [00:08<00:02, 204.29batch/s]

Training batches on cuda:  76%|███████▋  | 1800/2354 [00:08<00:02, 204.41batch/s]

Training batches on cuda:  77%|███████▋  | 1821/2354 [00:08<00:02, 205.57batch/s]

Training batches on cuda:  78%|███████▊  | 1843/2354 [00:09<00:02, 208.01batch/s]

Training batches on cuda:  79%|███████▉  | 1865/2354 [00:09<00:02, 210.43batch/s]

Training batches on cuda:  80%|████████  | 1887/2354 [00:09<00:02, 211.11batch/s]

Training batches on cuda:  81%|████████  | 1909/2354 [00:09<00:02, 211.63batch/s]

Training batches on cuda:  82%|████████▏ | 1931/2354 [00:09<00:02, 211.49batch/s]

Training batches on cuda:  83%|████████▎ | 1953/2354 [00:09<00:01, 203.35batch/s]

Training batches on cuda:  84%|████████▍ | 1974/2354 [00:09<00:01, 194.45batch/s]

Training batches on cuda:  85%|████████▍ | 1994/2354 [00:09<00:01, 190.10batch/s]

Training batches on cuda:  86%|████████▌ | 2014/2354 [00:09<00:01, 188.18batch/s]

Training batches on cuda:  86%|████████▋ | 2034/2354 [00:10<00:01, 190.45batch/s]

Training batches on cuda:  87%|████████▋ | 2054/2354 [00:10<00:01, 191.91batch/s]

Training batches on cuda:  88%|████████▊ | 2075/2354 [00:10<00:01, 194.89batch/s]

Training batches on cuda:  89%|████████▉ | 2098/2354 [00:10<00:01, 202.42batch/s]

Training batches on cuda:  90%|█████████ | 2119/2354 [00:10<00:01, 198.14batch/s]

Training batches on cuda:  91%|█████████ | 2140/2354 [00:10<00:01, 199.19batch/s]

Training batches on cuda:  92%|█████████▏| 2163/2354 [00:10<00:00, 205.70batch/s]

Training batches on cuda:  93%|█████████▎| 2185/2354 [00:10<00:00, 209.14batch/s]

Training batches on cuda:  94%|█████████▍| 2207/2354 [00:10<00:00, 211.22batch/s]

Training batches on cuda:  95%|█████████▍| 2229/2354 [00:10<00:00, 208.76batch/s]

Training batches on cuda:  96%|█████████▌| 2250/2354 [00:11<00:00, 208.92batch/s]

Training batches on cuda:  97%|█████████▋| 2272/2354 [00:11<00:00, 211.63batch/s]

Training batches on cuda:  97%|█████████▋| 2294/2354 [00:11<00:00, 206.53batch/s]

Training batches on cuda:  98%|█████████▊| 2315/2354 [00:11<00:00, 203.97batch/s]

Training batches on cuda:  99%|█████████▉| 2338/2354 [00:11<00:00, 207.53batch/s]

                                                                                 
Training epochs on cuda:  60%|██████    | 6/10 [01:24<00:48, 12.08s/epoch, loss=0.0162, prev_loss=0.0162]
Training epochs on cuda:  70%|███████   | 7/10 [01:24<00:35, 11.99s/epoch, loss=0.0162, prev_loss=0.0162]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 13/2354 [00:00<00:19, 123.15batch/s]

Training batches on cuda:   1%|▏         | 30/2354 [00:00<00:17, 133.36batch/s]

Training batches on cuda:   2%|▏         | 48/2354 [00:00<00:15, 144.15batch/s]

Training batches on cuda:   3%|▎         | 67/2354 [00:00<00:14, 154.60batch/s]

Training batches on cuda:   4%|▎         | 86/2354 [00:00<00:13, 162.51batch/s]

Training batches on cuda:   5%|▍         | 106/2354 [00:00<00:13, 170.91batch/s]

Training batches on cuda:   5%|▌         | 125/2354 [00:00<00:12, 174.40batch/s]

Training batches on cuda:   6%|▌         | 143/2354 [00:00<00:12, 174.13batch/s]

Training batches on cuda:   7%|▋         | 162/2354 [00:00<00:12, 176.24batch/s]

Training batches on cuda:   8%|▊         | 181/2354 [00:01<00:12, 179.97batch/s]

Training batches on cuda:   9%|▊         | 202/2354 [00:01<00:11, 186.25batch/s]

Training batches on cuda:  10%|▉         | 225/2354 [00:01<00:10, 196.71batch/s]

Training batches on cuda:  11%|█         | 248/2354 [00:01<00:10, 204.61batch/s]

Training batches on cuda:  11%|█▏        | 270/2354 [00:01<00:09, 208.79batch/s]

Training batches on cuda:  12%|█▏        | 292/2354 [00:01<00:09, 207.21batch/s]

Training batches on cuda:  13%|█▎        | 315/2354 [00:01<00:09, 212.58batch/s]

Training batches on cuda:  14%|█▍        | 338/2354 [00:01<00:09, 215.31batch/s]

Training batches on cuda:  15%|█▌        | 361/2354 [00:01<00:09, 217.82batch/s]

Training batches on cuda:  16%|█▋        | 384/2354 [00:01<00:08, 219.55batch/s]

Training batches on cuda:  17%|█▋        | 407/2354 [00:02<00:08, 219.92batch/s]

Training batches on cuda:  18%|█▊        | 430/2354 [00:02<00:08, 220.54batch/s]

Training batches on cuda:  19%|█▉        | 453/2354 [00:02<00:08, 216.18batch/s]

Training batches on cuda:  20%|██        | 475/2354 [00:02<00:08, 210.47batch/s]

Training batches on cuda:  21%|██        | 497/2354 [00:02<00:09, 198.37batch/s]

Training batches on cuda:  22%|██▏       | 519/2354 [00:02<00:09, 203.68batch/s]

Training batches on cuda:  23%|██▎       | 540/2354 [00:02<00:08, 204.97batch/s]

Training batches on cuda:  24%|██▍       | 562/2354 [00:02<00:08, 207.75batch/s]

Training batches on cuda:  25%|██▍       | 583/2354 [00:02<00:08, 202.38batch/s]

Training batches on cuda:  26%|██▌       | 605/2354 [00:03<00:08, 206.44batch/s]

Training batches on cuda:  27%|██▋       | 628/2354 [00:03<00:08, 211.99batch/s]

Training batches on cuda:  28%|██▊       | 650/2354 [00:03<00:08, 202.78batch/s]

Training batches on cuda:  29%|██▊       | 671/2354 [00:03<00:08, 193.78batch/s]

Training batches on cuda:  29%|██▉       | 691/2354 [00:03<00:09, 179.19batch/s]

Training batches on cuda:  30%|███       | 710/2354 [00:03<00:09, 178.97batch/s]

Training batches on cuda:  31%|███       | 731/2354 [00:03<00:08, 186.22batch/s]

Training batches on cuda:  32%|███▏      | 754/2354 [00:03<00:08, 196.25batch/s]

Training batches on cuda:  33%|███▎      | 777/2354 [00:03<00:07, 204.91batch/s]

Training batches on cuda:  34%|███▍      | 800/2354 [00:04<00:07, 209.64batch/s]

Training batches on cuda:  35%|███▍      | 823/2354 [00:04<00:07, 213.82batch/s]

Training batches on cuda:  36%|███▌      | 845/2354 [00:04<00:08, 188.56batch/s]

Training batches on cuda:  37%|███▋      | 866/2354 [00:04<00:07, 191.72batch/s]

Training batches on cuda:  38%|███▊      | 886/2354 [00:04<00:07, 193.13batch/s]

Training batches on cuda:  39%|███▊      | 908/2354 [00:04<00:07, 200.19batch/s]

Training batches on cuda:  39%|███▉      | 929/2354 [00:04<00:07, 199.56batch/s]

Training batches on cuda:  40%|████      | 950/2354 [00:04<00:07, 199.27batch/s]

Training batches on cuda:  41%|████      | 971/2354 [00:04<00:06, 201.39batch/s]

Training batches on cuda:  42%|████▏     | 992/2354 [00:04<00:06, 201.82batch/s]

Training batches on cuda:  43%|████▎     | 1013/2354 [00:05<00:06, 200.88batch/s]

Training batches on cuda:  44%|████▍     | 1034/2354 [00:05<00:06, 199.23batch/s]

Training batches on cuda:  45%|████▍     | 1056/2354 [00:05<00:06, 203.17batch/s]

Training batches on cuda:  46%|████▌     | 1078/2354 [00:05<00:06, 205.09batch/s]

Training batches on cuda:  47%|████▋     | 1099/2354 [00:05<00:06, 202.02batch/s]

Training batches on cuda:  48%|████▊     | 1120/2354 [00:05<00:06, 198.27batch/s]

Training batches on cuda:  48%|████▊     | 1141/2354 [00:05<00:06, 199.79batch/s]

Training batches on cuda:  49%|████▉     | 1162/2354 [00:05<00:05, 201.59batch/s]

Training batches on cuda:  50%|█████     | 1183/2354 [00:05<00:05, 195.99batch/s]

Training batches on cuda:  51%|█████     | 1203/2354 [00:06<00:05, 195.93batch/s]

Training batches on cuda:  52%|█████▏    | 1224/2354 [00:06<00:05, 199.26batch/s]

Training batches on cuda:  53%|█████▎    | 1246/2354 [00:06<00:05, 203.08batch/s]

Training batches on cuda:  54%|█████▍    | 1267/2354 [00:06<00:05, 201.43batch/s]

Training batches on cuda:  55%|█████▍    | 1290/2354 [00:06<00:05, 209.19batch/s]

Training batches on cuda:  56%|█████▌    | 1312/2354 [00:06<00:04, 209.90batch/s]

Training batches on cuda:  57%|█████▋    | 1334/2354 [00:06<00:04, 212.60batch/s]

Training batches on cuda:  58%|█████▊    | 1356/2354 [00:06<00:04, 203.44batch/s]

Training batches on cuda:  59%|█████▊    | 1379/2354 [00:06<00:04, 209.27batch/s]

Training batches on cuda:  60%|█████▉    | 1401/2354 [00:06<00:04, 211.00batch/s]

Training batches on cuda:  60%|██████    | 1424/2354 [00:07<00:04, 214.67batch/s]

Training batches on cuda:  61%|██████▏   | 1447/2354 [00:07<00:04, 216.63batch/s]

Training batches on cuda:  62%|██████▏   | 1470/2354 [00:07<00:04, 218.07batch/s]

Training batches on cuda:  63%|██████▎   | 1492/2354 [00:07<00:04, 215.01batch/s]

Training batches on cuda:  64%|██████▍   | 1514/2354 [00:07<00:03, 213.38batch/s]

Training batches on cuda:  65%|██████▌   | 1536/2354 [00:07<00:03, 214.60batch/s]

Training batches on cuda:  66%|██████▌   | 1559/2354 [00:07<00:03, 216.46batch/s]

Training batches on cuda:  67%|██████▋   | 1581/2354 [00:07<00:03, 206.10batch/s]

Training batches on cuda:  68%|██████▊   | 1603/2354 [00:07<00:03, 209.47batch/s]

Training batches on cuda:  69%|██████▉   | 1626/2354 [00:08<00:03, 213.94batch/s]

Training batches on cuda:  70%|███████   | 1649/2354 [00:08<00:03, 215.83batch/s]

Training batches on cuda:  71%|███████   | 1672/2354 [00:08<00:03, 217.02batch/s]

Training batches on cuda:  72%|███████▏  | 1695/2354 [00:08<00:03, 218.53batch/s]

Training batches on cuda:  73%|███████▎  | 1718/2354 [00:08<00:02, 221.51batch/s]

Training batches on cuda:  74%|███████▍  | 1741/2354 [00:08<00:02, 218.90batch/s]

Training batches on cuda:  75%|███████▍  | 1763/2354 [00:08<00:02, 214.99batch/s]

Training batches on cuda:  76%|███████▌  | 1785/2354 [00:08<00:02, 213.90batch/s]

Training batches on cuda:  77%|███████▋  | 1808/2354 [00:08<00:02, 215.90batch/s]

Training batches on cuda:  78%|███████▊  | 1830/2354 [00:08<00:02, 211.31batch/s]

Training batches on cuda:  79%|███████▊  | 1852/2354 [00:09<00:02, 202.90batch/s]

Training batches on cuda:  80%|███████▉  | 1873/2354 [00:09<00:02, 201.61batch/s]

Training batches on cuda:  80%|████████  | 1894/2354 [00:09<00:02, 179.58batch/s]

Training batches on cuda:  81%|████████▏ | 1915/2354 [00:09<00:02, 185.95batch/s]

Training batches on cuda:  82%|████████▏ | 1936/2354 [00:09<00:02, 190.39batch/s]

Training batches on cuda:  83%|████████▎ | 1956/2354 [00:09<00:02, 192.56batch/s]

Training batches on cuda:  84%|████████▍ | 1976/2354 [00:09<00:01, 193.68batch/s]

Training batches on cuda:  85%|████████▍ | 1996/2354 [00:09<00:01, 194.13batch/s]

Training batches on cuda:  86%|████████▌ | 2018/2354 [00:09<00:01, 199.83batch/s]

Training batches on cuda:  87%|████████▋ | 2039/2354 [00:10<00:01, 198.87batch/s]

Training batches on cuda:  87%|████████▋ | 2059/2354 [00:10<00:01, 197.93batch/s]

Training batches on cuda:  88%|████████▊ | 2079/2354 [00:10<00:01, 198.00batch/s]

Training batches on cuda:  89%|████████▉ | 2099/2354 [00:10<00:01, 193.99batch/s]

Training batches on cuda:  90%|█████████ | 2119/2354 [00:10<00:01, 190.95batch/s]

Training batches on cuda:  91%|█████████ | 2139/2354 [00:10<00:01, 190.70batch/s]

Training batches on cuda:  92%|█████████▏| 2159/2354 [00:10<00:01, 193.02batch/s]

Training batches on cuda:  93%|█████████▎| 2179/2354 [00:10<00:00, 190.71batch/s]

Training batches on cuda:  93%|█████████▎| 2199/2354 [00:10<00:00, 192.82batch/s]

Training batches on cuda:  94%|█████████▍| 2219/2354 [00:11<00:00, 191.77batch/s]

Training batches on cuda:  95%|█████████▌| 2241/2354 [00:11<00:00, 198.13batch/s]

Training batches on cuda:  96%|█████████▌| 2263/2354 [00:11<00:00, 200.79batch/s]

Training batches on cuda:  97%|█████████▋| 2284/2354 [00:11<00:00, 202.78batch/s]

Training batches on cuda:  98%|█████████▊| 2306/2354 [00:11<00:00, 205.22batch/s]

Training batches on cuda:  99%|█████████▉| 2327/2354 [00:11<00:00, 202.31batch/s]

Training batches on cuda: 100%|█████████▉| 2348/2354 [00:11<00:00, 198.86batch/s]

                                                                                 
Training epochs on cuda:  70%|███████   | 7/10 [01:36<00:35, 11.99s/epoch, loss=0.0162, prev_loss=0.0162]
Training epochs on cuda:  80%|████████  | 8/10 [01:36<00:23, 11.95s/epoch, loss=0.0162, prev_loss=0.0162]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 14/2354 [00:00<00:17, 134.60batch/s]

Training batches on cuda:   1%|▏         | 31/2354 [00:00<00:16, 143.37batch/s]

Training batches on cuda:   2%|▏         | 49/2354 [00:00<00:15, 151.11batch/s]

Training batches on cuda:   3%|▎         | 67/2354 [00:00<00:14, 157.22batch/s]

Training batches on cuda:   4%|▎         | 86/2354 [00:00<00:13, 165.50batch/s]

Training batches on cuda:   5%|▍         | 107/2354 [00:00<00:12, 174.95batch/s]

Training batches on cuda:   5%|▌         | 128/2354 [00:00<00:12, 182.35batch/s]

Training batches on cuda:   6%|▋         | 149/2354 [00:00<00:11, 187.82batch/s]

Training batches on cuda:   7%|▋         | 171/2354 [00:00<00:11, 195.23batch/s]

Training batches on cuda:   8%|▊         | 194/2354 [00:01<00:10, 202.42batch/s]

Training batches on cuda:   9%|▉         | 215/2354 [00:01<00:10, 199.21batch/s]

Training batches on cuda:  10%|█         | 236/2354 [00:01<00:10, 200.50batch/s]

Training batches on cuda:  11%|█         | 256/2354 [00:01<00:10, 198.69batch/s]

Training batches on cuda:  12%|█▏        | 276/2354 [00:01<00:10, 195.26batch/s]

Training batches on cuda:  13%|█▎        | 298/2354 [00:01<00:10, 199.70batch/s]

Training batches on cuda:  14%|█▎        | 319/2354 [00:01<00:10, 200.57batch/s]

Training batches on cuda:  14%|█▍        | 341/2354 [00:01<00:09, 205.59batch/s]

Training batches on cuda:  15%|█▌        | 363/2354 [00:01<00:09, 208.66batch/s]

Training batches on cuda:  16%|█▋        | 384/2354 [00:01<00:10, 194.45batch/s]

Training batches on cuda:  17%|█▋        | 404/2354 [00:02<00:10, 191.25batch/s]

Training batches on cuda:  18%|█▊        | 425/2354 [00:02<00:09, 195.03batch/s]

Training batches on cuda:  19%|█▉        | 447/2354 [00:02<00:09, 201.25batch/s]

Training batches on cuda:  20%|█▉        | 470/2354 [00:02<00:09, 206.81batch/s]

Training batches on cuda:  21%|██        | 491/2354 [00:02<00:10, 183.82batch/s]

Training batches on cuda:  22%|██▏       | 510/2354 [00:02<00:10, 177.31batch/s]

Training batches on cuda:  22%|██▏       | 529/2354 [00:02<00:10, 176.29batch/s]

Training batches on cuda:  23%|██▎       | 551/2354 [00:02<00:09, 187.05batch/s]

Training batches on cuda:  24%|██▍       | 574/2354 [00:02<00:09, 197.36batch/s]

Training batches on cuda:  25%|██▌       | 596/2354 [00:03<00:08, 202.31batch/s]

Training batches on cuda:  26%|██▋       | 618/2354 [00:03<00:08, 204.70batch/s]

Training batches on cuda:  27%|██▋       | 640/2354 [00:03<00:08, 208.50batch/s]

Training batches on cuda:  28%|██▊       | 662/2354 [00:03<00:08, 204.78batch/s]

Training batches on cuda:  29%|██▉       | 684/2354 [00:03<00:08, 208.59batch/s]

Training batches on cuda:  30%|██▉       | 706/2354 [00:03<00:07, 209.87batch/s]

Training batches on cuda:  31%|███       | 728/2354 [00:03<00:07, 209.64batch/s]

Training batches on cuda:  32%|███▏      | 750/2354 [00:03<00:08, 192.90batch/s]

Training batches on cuda:  33%|███▎      | 770/2354 [00:03<00:08, 180.77batch/s]

Training batches on cuda:  34%|███▎      | 789/2354 [00:04<00:08, 181.28batch/s]

Training batches on cuda:  34%|███▍      | 808/2354 [00:04<00:08, 177.74batch/s]

Training batches on cuda:  35%|███▌      | 826/2354 [00:04<00:08, 176.09batch/s]

Training batches on cuda:  36%|███▌      | 846/2354 [00:04<00:08, 182.33batch/s]

Training batches on cuda:  37%|███▋      | 867/2354 [00:04<00:07, 187.89batch/s]

Training batches on cuda:  38%|███▊      | 887/2354 [00:04<00:07, 189.05batch/s]

Training batches on cuda:  39%|███▊      | 909/2354 [00:04<00:07, 195.27batch/s]

Training batches on cuda:  40%|███▉      | 931/2354 [00:04<00:07, 200.32batch/s]

Training batches on cuda:  40%|████      | 953/2354 [00:04<00:06, 205.16batch/s]

Training batches on cuda:  41%|████▏     | 974/2354 [00:05<00:06, 204.10batch/s]

Training batches on cuda:  42%|████▏     | 995/2354 [00:05<00:06, 201.84batch/s]

Training batches on cuda:  43%|████▎     | 1017/2354 [00:05<00:06, 206.15batch/s]

Training batches on cuda:  44%|████▍     | 1040/2354 [00:05<00:06, 211.26batch/s]

Training batches on cuda:  45%|████▌     | 1062/2354 [00:05<00:06, 211.61batch/s]

Training batches on cuda:  46%|████▌     | 1084/2354 [00:05<00:06, 192.00batch/s]

Training batches on cuda:  47%|████▋     | 1104/2354 [00:05<00:06, 180.10batch/s]

Training batches on cuda:  48%|████▊     | 1125/2354 [00:05<00:06, 187.15batch/s]

Training batches on cuda:  49%|████▊     | 1145/2354 [00:05<00:06, 179.17batch/s]

Training batches on cuda:  49%|████▉     | 1164/2354 [00:06<00:06, 179.28batch/s]

Training batches on cuda:  50%|█████     | 1183/2354 [00:06<00:06, 178.14batch/s]

Training batches on cuda:  51%|█████     | 1202/2354 [00:06<00:06, 179.05batch/s]

Training batches on cuda:  52%|█████▏    | 1221/2354 [00:06<00:06, 181.56batch/s]

Training batches on cuda:  53%|█████▎    | 1243/2354 [00:06<00:05, 189.61batch/s]

Training batches on cuda:  54%|█████▍    | 1266/2354 [00:06<00:05, 198.04batch/s]

Training batches on cuda:  55%|█████▍    | 1288/2354 [00:06<00:05, 201.10batch/s]

Training batches on cuda:  56%|█████▌    | 1309/2354 [00:06<00:05, 185.26batch/s]

Training batches on cuda:  56%|█████▋    | 1328/2354 [00:06<00:05, 179.09batch/s]

Training batches on cuda:  57%|█████▋    | 1347/2354 [00:07<00:05, 174.41batch/s]

Training batches on cuda:  58%|█████▊    | 1365/2354 [00:07<00:05, 172.79batch/s]

Training batches on cuda:  59%|█████▉    | 1383/2354 [00:07<00:05, 169.89batch/s]

Training batches on cuda:  60%|█████▉    | 1401/2354 [00:07<00:05, 172.72batch/s]

Training batches on cuda:  60%|██████    | 1420/2354 [00:07<00:05, 175.61batch/s]

Training batches on cuda:  61%|██████    | 1438/2354 [00:07<00:05, 164.28batch/s]

Training batches on cuda:  62%|██████▏   | 1456/2354 [00:07<00:05, 167.93batch/s]

Training batches on cuda:  63%|██████▎   | 1476/2354 [00:07<00:05, 175.49batch/s]

Training batches on cuda:  63%|██████▎   | 1494/2354 [00:07<00:04, 176.01batch/s]

Training batches on cuda:  64%|██████▍   | 1513/2354 [00:07<00:04, 179.05batch/s]

Training batches on cuda:  65%|██████▌   | 1532/2354 [00:08<00:04, 170.06batch/s]

Training batches on cuda:  66%|██████▌   | 1550/2354 [00:08<00:04, 171.60batch/s]

Training batches on cuda:  67%|██████▋   | 1569/2354 [00:08<00:04, 175.06batch/s]

Training batches on cuda:  67%|██████▋   | 1588/2354 [00:08<00:04, 179.14batch/s]

Training batches on cuda:  68%|██████▊   | 1607/2354 [00:08<00:04, 181.43batch/s]

Training batches on cuda:  69%|██████▉   | 1626/2354 [00:08<00:04, 181.70batch/s]

Training batches on cuda:  70%|██████▉   | 1645/2354 [00:08<00:03, 179.94batch/s]

Training batches on cuda:  71%|███████   | 1664/2354 [00:08<00:03, 178.15batch/s]

Training batches on cuda:  71%|███████▏  | 1683/2354 [00:08<00:03, 181.50batch/s]

Training batches on cuda:  72%|███████▏  | 1702/2354 [00:09<00:03, 181.09batch/s]

Training batches on cuda:  73%|███████▎  | 1721/2354 [00:09<00:03, 178.72batch/s]

Training batches on cuda:  74%|███████▍  | 1739/2354 [00:09<00:03, 178.91batch/s]

Training batches on cuda:  75%|███████▍  | 1757/2354 [00:09<00:03, 177.69batch/s]

Training batches on cuda:  75%|███████▌  | 1776/2354 [00:09<00:03, 179.33batch/s]

Training batches on cuda:  76%|███████▋  | 1796/2354 [00:09<00:03, 183.61batch/s]

Training batches on cuda:  77%|███████▋  | 1815/2354 [00:09<00:02, 181.92batch/s]

Training batches on cuda:  78%|███████▊  | 1835/2354 [00:09<00:02, 184.83batch/s]

Training batches on cuda:  79%|███████▉  | 1856/2354 [00:09<00:02, 190.97batch/s]

Training batches on cuda:  80%|███████▉  | 1879/2354 [00:09<00:02, 200.40batch/s]

Training batches on cuda:  81%|████████  | 1901/2354 [00:10<00:02, 205.82batch/s]

Training batches on cuda:  82%|████████▏ | 1922/2354 [00:10<00:02, 205.97batch/s]

Training batches on cuda:  83%|████████▎ | 1943/2354 [00:10<00:02, 189.55batch/s]

Training batches on cuda:  83%|████████▎ | 1963/2354 [00:10<00:02, 189.59batch/s]

Training batches on cuda:  84%|████████▍ | 1983/2354 [00:10<00:01, 188.96batch/s]

Training batches on cuda:  85%|████████▌ | 2004/2354 [00:10<00:01, 194.77batch/s]

Training batches on cuda:  86%|████████▌ | 2024/2354 [00:10<00:01, 188.85batch/s]

Training batches on cuda:  87%|████████▋ | 2044/2354 [00:10<00:01, 186.26batch/s]

Training batches on cuda:  88%|████████▊ | 2063/2354 [00:10<00:01, 179.49batch/s]

Training batches on cuda:  88%|████████▊ | 2082/2354 [00:11<00:01, 179.35batch/s]

Training batches on cuda:  89%|████████▉ | 2101/2354 [00:11<00:01, 178.93batch/s]

Training batches on cuda:  90%|█████████ | 2121/2354 [00:11<00:01, 183.07batch/s]

Training batches on cuda:  91%|█████████ | 2142/2354 [00:11<00:01, 190.24batch/s]

Training batches on cuda:  92%|█████████▏| 2167/2354 [00:11<00:00, 203.43batch/s]

Training batches on cuda:  93%|█████████▎| 2188/2354 [00:11<00:00, 203.02batch/s]

Training batches on cuda:  94%|█████████▍| 2209/2354 [00:11<00:00, 203.28batch/s]

Training batches on cuda:  95%|█████████▍| 2230/2354 [00:11<00:00, 198.33batch/s]

Training batches on cuda:  96%|█████████▌| 2252/2354 [00:11<00:00, 202.38batch/s]

Training batches on cuda:  97%|█████████▋| 2273/2354 [00:11<00:00, 201.43batch/s]

Training batches on cuda:  97%|█████████▋| 2295/2354 [00:12<00:00, 204.82batch/s]

Training batches on cuda:  98%|█████████▊| 2316/2354 [00:12<00:00, 200.77batch/s]

Training batches on cuda:  99%|█████████▉| 2340/2354 [00:12<00:00, 209.47batch/s]

                                                                                 
Training epochs on cuda:  80%|████████  | 8/10 [01:49<00:23, 11.95s/epoch, loss=0.0162, prev_loss=0.0162]
Training epochs on cuda:  90%|█████████ | 9/10 [01:49<00:12, 12.15s/epoch, loss=0.0162, prev_loss=0.0162]

Training batches on cuda:   0%|          | 0/2354 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 14/2354 [00:00<00:16, 139.15batch/s]

Training batches on cuda:   2%|▏         | 37/2354 [00:00<00:14, 156.56batch/s]

Training batches on cuda:   3%|▎         | 59/2354 [00:00<00:13, 170.45batch/s]

Training batches on cuda:   3%|▎         | 81/2354 [00:00<00:12, 181.37batch/s]

Training batches on cuda:   4%|▍         | 103/2354 [00:00<00:11, 189.56batch/s]

Training batches on cuda:   5%|▌         | 126/2354 [00:00<00:11, 199.95batch/s]

Training batches on cuda:   6%|▌         | 146/2354 [00:00<00:11, 199.65batch/s]

Training batches on cuda:   7%|▋         | 169/2354 [00:00<00:10, 207.15batch/s]

Training batches on cuda:   8%|▊         | 190/2354 [00:00<00:10, 207.21batch/s]

Training batches on cuda:   9%|▉         | 212/2354 [00:01<00:10, 207.89batch/s]

Training batches on cuda:  10%|▉         | 233/2354 [00:01<00:10, 205.58batch/s]

Training batches on cuda:  11%|█         | 255/2354 [00:01<00:10, 208.47batch/s]

Training batches on cuda:  12%|█▏        | 279/2354 [00:01<00:09, 214.81batch/s]

Training batches on cuda:  13%|█▎        | 302/2354 [00:01<00:09, 216.68batch/s]

Training batches on cuda:  14%|█▍        | 326/2354 [00:01<00:09, 220.45batch/s]

Training batches on cuda:  15%|█▍        | 349/2354 [00:01<00:09, 203.87batch/s]

Training batches on cuda:  16%|█▌        | 370/2354 [00:01<00:09, 201.26batch/s]

Training batches on cuda:  17%|█▋        | 393/2354 [00:01<00:09, 207.14batch/s]

Training batches on cuda:  18%|█▊        | 415/2354 [00:01<00:09, 209.32batch/s]

Training batches on cuda:  19%|█▊        | 438/2354 [00:02<00:08, 214.65batch/s]

Training batches on cuda:  20%|█▉        | 460/2354 [00:02<00:08, 215.31batch/s]

Training batches on cuda:  20%|██        | 482/2354 [00:02<00:08, 212.08batch/s]

Training batches on cuda:  21%|██▏       | 504/2354 [00:02<00:09, 193.08batch/s]

Training batches on cuda:  22%|██▏       | 524/2354 [00:02<00:09, 189.94batch/s]

Training batches on cuda:  23%|██▎       | 544/2354 [00:02<00:09, 185.29batch/s]

Training batches on cuda:  24%|██▍       | 563/2354 [00:02<00:09, 182.17batch/s]

Training batches on cuda:  25%|██▍       | 582/2354 [00:02<00:10, 170.18batch/s]

Training batches on cuda:  25%|██▌       | 600/2354 [00:02<00:10, 172.58batch/s]

Training batches on cuda:  26%|██▋       | 618/2354 [00:03<00:10, 167.21batch/s]

Training batches on cuda:  27%|██▋       | 640/2354 [00:03<00:09, 177.36batch/s]

Training batches on cuda:  28%|██▊       | 659/2354 [00:03<00:09, 171.95batch/s]

Training batches on cuda:  29%|██▉       | 677/2354 [00:03<00:09, 173.81batch/s]

Training batches on cuda:  30%|██▉       | 697/2354 [00:03<00:09, 179.30batch/s]

Training batches on cuda:  30%|███       | 716/2354 [00:03<00:09, 180.18batch/s]

Training batches on cuda:  31%|███       | 735/2354 [00:03<00:08, 180.92batch/s]

Training batches on cuda:  32%|███▏      | 755/2354 [00:03<00:08, 184.00batch/s]

Training batches on cuda:  33%|███▎      | 776/2354 [00:03<00:08, 188.61batch/s]

Training batches on cuda:  34%|███▍      | 796/2354 [00:04<00:08, 189.96batch/s]

Training batches on cuda:  35%|███▍      | 817/2354 [00:04<00:07, 193.76batch/s]

Training batches on cuda:  36%|███▌      | 837/2354 [00:04<00:07, 190.66batch/s]

Training batches on cuda:  36%|███▋      | 857/2354 [00:04<00:07, 191.42batch/s]

Training batches on cuda:  37%|███▋      | 877/2354 [00:04<00:07, 188.05batch/s]

Training batches on cuda:  38%|███▊      | 896/2354 [00:04<00:07, 183.76batch/s]

Training batches on cuda:  39%|███▉      | 917/2354 [00:04<00:07, 188.83batch/s]

Training batches on cuda:  40%|███▉      | 938/2354 [00:04<00:07, 193.73batch/s]

Training batches on cuda:  41%|████      | 958/2354 [00:04<00:07, 194.54batch/s]

Training batches on cuda:  42%|████▏     | 978/2354 [00:05<00:07, 193.28batch/s]

Training batches on cuda:  42%|████▏     | 998/2354 [00:05<00:07, 172.06batch/s]

Training batches on cuda:  43%|████▎     | 1018/2354 [00:05<00:07, 178.74batch/s]

Training batches on cuda:  44%|████▍     | 1037/2354 [00:05<00:07, 181.79batch/s]

Training batches on cuda:  45%|████▍     | 1056/2354 [00:05<00:07, 180.84batch/s]

Training batches on cuda:  46%|████▌     | 1075/2354 [00:05<00:07, 173.08batch/s]

Training batches on cuda:  47%|████▋     | 1100/2354 [00:05<00:06, 189.72batch/s]

Training batches on cuda:  48%|████▊     | 1125/2354 [00:05<00:06, 203.46batch/s]

Training batches on cuda:  49%|████▊     | 1147/2354 [00:05<00:05, 201.48batch/s]

Training batches on cuda:  50%|████▉     | 1168/2354 [00:06<00:06, 196.69batch/s]

Training batches on cuda:  51%|█████     | 1189/2354 [00:06<00:05, 197.73batch/s]

Training batches on cuda:  51%|█████▏    | 1210/2354 [00:06<00:05, 192.52batch/s]

Training batches on cuda:  52%|█████▏    | 1230/2354 [00:06<00:05, 193.72batch/s]

Training batches on cuda:  53%|█████▎    | 1250/2354 [00:06<00:05, 187.63batch/s]

Training batches on cuda:  54%|█████▍    | 1270/2354 [00:06<00:05, 189.83batch/s]

Training batches on cuda:  55%|█████▍    | 1290/2354 [00:06<00:05, 188.08batch/s]

Training batches on cuda:  56%|█████▌    | 1311/2354 [00:06<00:05, 192.18batch/s]

Training batches on cuda:  57%|█████▋    | 1333/2354 [00:06<00:05, 199.00batch/s]

Training batches on cuda:  58%|█████▊    | 1355/2354 [00:06<00:04, 204.14batch/s]

Training batches on cuda:  59%|█████▊    | 1378/2354 [00:07<00:04, 208.52batch/s]

Training batches on cuda:  59%|█████▉    | 1400/2354 [00:07<00:04, 209.82batch/s]

Training batches on cuda:  60%|██████    | 1422/2354 [00:07<00:04, 209.44batch/s]

Training batches on cuda:  61%|██████▏   | 1443/2354 [00:07<00:04, 209.11batch/s]

Training batches on cuda:  62%|██████▏   | 1465/2354 [00:07<00:04, 210.82batch/s]

Training batches on cuda:  63%|██████▎   | 1487/2354 [00:07<00:04, 209.13batch/s]

Training batches on cuda:  64%|██████▍   | 1508/2354 [00:07<00:04, 205.06batch/s]

Training batches on cuda:  65%|██████▍   | 1529/2354 [00:07<00:04, 201.14batch/s]

Training batches on cuda:  66%|██████▌   | 1550/2354 [00:07<00:04, 192.55batch/s]

Training batches on cuda:  67%|██████▋   | 1570/2354 [00:08<00:04, 191.72batch/s]

Training batches on cuda:  68%|██████▊   | 1590/2354 [00:08<00:03, 193.19batch/s]

Training batches on cuda:  68%|██████▊   | 1611/2354 [00:08<00:03, 197.48batch/s]

Training batches on cuda:  69%|██████▉   | 1634/2354 [00:08<00:03, 203.97batch/s]

Training batches on cuda:  70%|███████   | 1655/2354 [00:08<00:03, 186.20batch/s]

Training batches on cuda:  71%|███████   | 1674/2354 [00:08<00:03, 180.46batch/s]

Training batches on cuda:  72%|███████▏  | 1693/2354 [00:08<00:03, 177.30batch/s]

Training batches on cuda:  73%|███████▎  | 1711/2354 [00:08<00:03, 175.95batch/s]

Training batches on cuda:  73%|███████▎  | 1730/2354 [00:08<00:03, 178.19batch/s]

Training batches on cuda:  74%|███████▍  | 1749/2354 [00:09<00:03, 179.23batch/s]

Training batches on cuda:  75%|███████▌  | 1768/2354 [00:09<00:03, 179.85batch/s]

Training batches on cuda:  76%|███████▌  | 1787/2354 [00:09<00:03, 179.33batch/s]

Training batches on cuda:  77%|███████▋  | 1806/2354 [00:09<00:03, 180.36batch/s]

Training batches on cuda:  78%|███████▊  | 1825/2354 [00:09<00:02, 181.13batch/s]

Training batches on cuda:  78%|███████▊  | 1844/2354 [00:09<00:02, 182.58batch/s]

Training batches on cuda:  79%|███████▉  | 1863/2354 [00:09<00:02, 182.28batch/s]

Training batches on cuda:  80%|███████▉  | 1882/2354 [00:09<00:02, 181.87batch/s]

Training batches on cuda:  81%|████████  | 1901/2354 [00:09<00:02, 182.67batch/s]

Training batches on cuda:  82%|████████▏ | 1920/2354 [00:09<00:02, 183.34batch/s]

Training batches on cuda:  82%|████████▏ | 1939/2354 [00:10<00:02, 184.13batch/s]

Training batches on cuda:  83%|████████▎ | 1958/2354 [00:10<00:02, 180.62batch/s]

Training batches on cuda:  84%|████████▍ | 1981/2354 [00:10<00:01, 191.74batch/s]

Training batches on cuda:  85%|████████▌ | 2001/2354 [00:10<00:01, 183.88batch/s]

Training batches on cuda:  86%|████████▌ | 2020/2354 [00:10<00:01, 180.21batch/s]

Training batches on cuda:  87%|████████▋ | 2039/2354 [00:10<00:01, 181.61batch/s]

Training batches on cuda:  87%|████████▋ | 2058/2354 [00:10<00:01, 183.03batch/s]

Training batches on cuda:  88%|████████▊ | 2078/2354 [00:10<00:01, 187.01batch/s]

Training batches on cuda:  89%|████████▉ | 2097/2354 [00:10<00:01, 186.91batch/s]

Training batches on cuda:  90%|████████▉ | 2118/2354 [00:11<00:01, 192.38batch/s]

Training batches on cuda:  91%|█████████ | 2141/2354 [00:11<00:01, 200.57batch/s]

Training batches on cuda:  92%|█████████▏| 2164/2354 [00:11<00:00, 207.15batch/s]

Training batches on cuda:  93%|█████████▎| 2187/2354 [00:11<00:00, 211.59batch/s]

Training batches on cuda:  94%|█████████▍| 2210/2354 [00:11<00:00, 214.36batch/s]

Training batches on cuda:  95%|█████████▍| 2233/2354 [00:11<00:00, 217.80batch/s]

Training batches on cuda:  96%|█████████▌| 2255/2354 [00:11<00:00, 217.43batch/s]

Training batches on cuda:  97%|█████████▋| 2278/2354 [00:11<00:00, 219.15batch/s]

Training batches on cuda:  98%|█████████▊| 2301/2354 [00:11<00:00, 220.89batch/s]

Training batches on cuda:  99%|█████████▊| 2324/2354 [00:11<00:00, 219.39batch/s]

Training batches on cuda: 100%|█████████▉| 2347/2354 [00:12<00:00, 221.91batch/s]

                                                                                 
Training epochs on cuda:  90%|█████████ | 9/10 [02:01<00:12, 12.15s/epoch, loss=0.0162, prev_loss=0.0162]2021-09-16:17:51:04,732 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:51:05,715 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:51:20,598 INFO     [evaluator.py:657] Evaluation took 14.88s seconds
2021-09-16:17:51:20,751 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 16.99s/epoch, loss=0.0162, prev_loss=0.0162]
Training epochs on cuda: 100%|██████████| 10/10 [02:17<00:00, 13.74s/epoch, loss=0.0162, prev_loss=0.0162]
2021-09-16:17:51:20,752 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp84iue329'
2021-09-16:17:51:20,815 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp84iue329' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.35ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:13, 2.51ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.55ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.57ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.61ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:17:51:35,646 INFO     [evaluator.py:657] Evaluation took 14.82s seconds
[I 2021-09-16 17:51:35,651] Trial 16 finished with value: 0.004112538602934923 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.33941887670739546, 'optimizer.lr': 0.015277149086790946, 'negative_sampler.num_negs_per_pos': 2, 'training.batch_size': 132}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:51:35,672 WARNING  [api.py:823] No random seed is specified. Setting to 788430147.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:06, 28.37batch/s]

Training batches on cuda:   4%|▍         | 8/191 [00:00<00:05, 31.37batch/s]

Training batches on cuda:   7%|▋         | 13/191 [00:00<00:05, 34.14batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:04, 35.69batch/s]

Training batches on cuda:  12%|█▏        | 22/191 [00:00<00:04, 36.97batch/s]

Training batches on cuda:  14%|█▍        | 27/191 [00:00<00:04, 38.30batch/s]

Training batches on cuda:  17%|█▋        | 32/191 [00:00<00:04, 38.79batch/s]

Training batches on cuda:  19%|█▉        | 37/191 [00:00<00:03, 39.38batch/s]

Training batches on cuda:  22%|██▏       | 42/191 [00:01<00:03, 39.78batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:03, 39.70batch/s]

Training batches on cuda:  27%|██▋       | 52/191 [00:01<00:03, 40.16batch/s]

Training batches on cuda:  29%|██▉       | 56/191 [00:01<00:03, 40.02batch/s]

Training batches on cuda:  31%|███▏      | 60/191 [00:01<00:03, 39.84batch/s]

Training batches on cuda:  34%|███▎      | 64/191 [00:01<00:03, 39.51batch/s]

Training batches on cuda:  36%|███▌      | 69/191 [00:01<00:03, 39.70batch/s]

Training batches on cuda:  38%|███▊      | 73/191 [00:01<00:03, 37.51batch/s]

Training batches on cuda:  40%|████      | 77/191 [00:01<00:02, 38.18batch/s]

Training batches on cuda:  42%|████▏     | 81/191 [00:02<00:02, 38.69batch/s]

Training batches on cuda:  45%|████▍     | 85/191 [00:02<00:02, 38.84batch/s]

Training batches on cuda:  47%|████▋     | 90/191 [00:02<00:02, 39.20batch/s]

Training batches on cuda:  49%|████▉     | 94/191 [00:02<00:02, 38.81batch/s]

Training batches on cuda:  52%|█████▏    | 99/191 [00:02<00:02, 39.97batch/s]

Training batches on cuda:  54%|█████▍    | 104/191 [00:02<00:02, 39.91batch/s]

Training batches on cuda:  57%|█████▋    | 109/191 [00:02<00:02, 39.15batch/s]

Training batches on cuda:  59%|█████▉    | 113/191 [00:02<00:02, 38.76batch/s]

Training batches on cuda:  61%|██████▏   | 117/191 [00:02<00:01, 38.30batch/s]

Training batches on cuda:  63%|██████▎   | 121/191 [00:03<00:01, 37.61batch/s]

Training batches on cuda:  66%|██████▌   | 126/191 [00:03<00:01, 38.86batch/s]

Training batches on cuda:  69%|██████▊   | 131/191 [00:03<00:01, 39.63batch/s]

Training batches on cuda:  71%|███████   | 135/191 [00:03<00:01, 39.51batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 39.71batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 40.10batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:01, 39.84batch/s]

Training batches on cuda:  81%|████████  | 154/191 [00:03<00:00, 39.16batch/s]

Training batches on cuda:  83%|████████▎ | 158/191 [00:04<00:00, 38.58batch/s]

Training batches on cuda:  85%|████████▌ | 163/191 [00:04<00:00, 39.45batch/s]

Training batches on cuda:  88%|████████▊ | 168/191 [00:04<00:00, 39.92batch/s]

Training batches on cuda:  91%|█████████ | 173/191 [00:04<00:00, 40.08batch/s]

Training batches on cuda:  93%|█████████▎| 178/191 [00:04<00:00, 40.25batch/s]

Training batches on cuda:  96%|█████████▌| 183/191 [00:04<00:00, 40.03batch/s]

Training batches on cuda:  98%|█████████▊| 188/191 [00:04<00:00, 40.29batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.00189, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:46,  5.11s/epoch, loss=0.00189, prev_loss=nan]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:06, 28.79batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:05, 31.38batch/s]

Training batches on cuda:   6%|▌         | 11/191 [00:00<00:05, 33.04batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:05, 34.27batch/s]

Training batches on cuda:  10%|▉         | 19/191 [00:00<00:04, 35.59batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:04, 36.71batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:04, 38.01batch/s]

Training batches on cuda:  17%|█▋        | 32/191 [00:00<00:04, 38.42batch/s]

Training batches on cuda:  19%|█▉        | 37/191 [00:00<00:03, 38.96batch/s]

Training batches on cuda:  22%|██▏       | 42/191 [00:01<00:03, 39.63batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:03, 39.87batch/s]

Training batches on cuda:  27%|██▋       | 52/191 [00:01<00:03, 40.04batch/s]

Training batches on cuda:  30%|██▉       | 57/191 [00:01<00:03, 40.26batch/s]

Training batches on cuda:  32%|███▏      | 62/191 [00:01<00:03, 40.61batch/s]

Training batches on cuda:  35%|███▌      | 67/191 [00:01<00:03, 40.39batch/s]

Training batches on cuda:  38%|███▊      | 72/191 [00:01<00:03, 39.38batch/s]

Training batches on cuda:  40%|████      | 77/191 [00:01<00:02, 40.00batch/s]

Training batches on cuda:  43%|████▎     | 82/191 [00:02<00:02, 40.00batch/s]

Training batches on cuda:  46%|████▌     | 87/191 [00:02<00:02, 39.32batch/s]

Training batches on cuda:  48%|████▊     | 92/191 [00:02<00:02, 39.58batch/s]

Training batches on cuda:  50%|█████     | 96/191 [00:02<00:02, 39.17batch/s]

Training batches on cuda:  53%|█████▎    | 101/191 [00:02<00:02, 39.44batch/s]

Training batches on cuda:  55%|█████▍    | 105/191 [00:02<00:02, 39.12batch/s]

Training batches on cuda:  58%|█████▊    | 110/191 [00:02<00:02, 39.86batch/s]

Training batches on cuda:  60%|██████    | 115/191 [00:02<00:01, 40.87batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:03<00:01, 41.49batch/s]

Training batches on cuda:  65%|██████▌   | 125/191 [00:03<00:01, 41.77batch/s]

Training batches on cuda:  68%|██████▊   | 130/191 [00:03<00:01, 41.93batch/s]

Training batches on cuda:  71%|███████   | 135/191 [00:03<00:01, 42.08batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 42.10batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 42.36batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:00, 42.43batch/s]

Training batches on cuda:  81%|████████  | 155/191 [00:03<00:00, 42.43batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:03<00:00, 42.38batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:04<00:00, 42.53batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:04<00:00, 42.59batch/s]

Training batches on cuda:  92%|█████████▏| 175/191 [00:04<00:00, 42.50batch/s]

Training batches on cuda:  94%|█████████▍| 180/191 [00:04<00:00, 42.38batch/s]

Training batches on cuda:  97%|█████████▋| 185/191 [00:04<00:00, 42.34batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:04<00:00, 42.15batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:09<00:46,  5.11s/epoch, loss=0.00188, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:40,  5.04s/epoch, loss=0.00188, prev_loss=nan]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:06, 27.73batch/s]

Training batches on cuda:   4%|▍         | 8/191 [00:00<00:05, 30.81batch/s]

Training batches on cuda:   7%|▋         | 13/191 [00:00<00:05, 33.36batch/s]

Training batches on cuda:   9%|▉         | 18/191 [00:00<00:04, 35.35batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:04, 36.78batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:04, 37.95batch/s]

Training batches on cuda:  17%|█▋        | 33/191 [00:00<00:04, 38.84batch/s]

Training batches on cuda:  20%|█▉        | 38/191 [00:00<00:03, 39.51batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:01<00:03, 39.82batch/s]

Training batches on cuda:  25%|██▌       | 48/191 [00:01<00:03, 40.33batch/s]

Training batches on cuda:  28%|██▊       | 53/191 [00:01<00:03, 40.61batch/s]

Training batches on cuda:  30%|███       | 58/191 [00:01<00:03, 40.56batch/s]

Training batches on cuda:  33%|███▎      | 63/191 [00:01<00:03, 40.65batch/s]

Training batches on cuda:  36%|███▌      | 68/191 [00:01<00:03, 40.75batch/s]

Training batches on cuda:  38%|███▊      | 73/191 [00:01<00:02, 41.01batch/s]

Training batches on cuda:  41%|████      | 78/191 [00:01<00:02, 38.27batch/s]

Training batches on cuda:  43%|████▎     | 83/191 [00:02<00:02, 38.62batch/s]

Training batches on cuda:  46%|████▌     | 88/191 [00:02<00:02, 39.25batch/s]

Training batches on cuda:  48%|████▊     | 92/191 [00:02<00:02, 39.47batch/s]

Training batches on cuda:  50%|█████     | 96/191 [00:02<00:02, 39.21batch/s]

Training batches on cuda:  53%|█████▎    | 101/191 [00:02<00:02, 39.69batch/s]

Training batches on cuda:  55%|█████▌    | 106/191 [00:02<00:02, 39.88batch/s]

Training batches on cuda:  58%|█████▊    | 110/191 [00:02<00:02, 39.42batch/s]

Training batches on cuda:  60%|██████    | 115/191 [00:02<00:01, 39.95batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:03<00:01, 40.16batch/s]

Training batches on cuda:  65%|██████▌   | 125/191 [00:03<00:01, 40.50batch/s]

Training batches on cuda:  68%|██████▊   | 130/191 [00:03<00:01, 40.65batch/s]

Training batches on cuda:  71%|███████   | 135/191 [00:03<00:01, 40.81batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 40.59batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 40.72batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:01, 40.76batch/s]

Training batches on cuda:  81%|████████  | 155/191 [00:03<00:00, 40.85batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:03<00:00, 40.96batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:04<00:00, 41.12batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:04<00:00, 40.96batch/s]

Training batches on cuda:  92%|█████████▏| 175/191 [00:04<00:00, 40.73batch/s]

Training batches on cuda:  94%|█████████▍| 180/191 [00:04<00:00, 40.45batch/s]

Training batches on cuda:  97%|█████████▋| 185/191 [00:04<00:00, 40.38batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:04<00:00, 40.63batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:14<00:40,  5.04s/epoch, loss=0.00188, prev_loss=0.00188]
Training epochs on cuda:  30%|███       | 3/10 [00:14<00:35,  5.02s/epoch, loss=0.00188, prev_loss=0.00188]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/191 [00:00<00:09, 18.97batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:08, 22.37batch/s]

Training batches on cuda:   6%|▌         | 11/191 [00:00<00:06, 25.99batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:06, 28.91batch/s]

Training batches on cuda:  10%|█         | 20/191 [00:00<00:05, 31.68batch/s]

Training batches on cuda:  13%|█▎        | 25/191 [00:00<00:04, 34.26batch/s]

Training batches on cuda:  16%|█▌        | 30/191 [00:00<00:04, 36.18batch/s]

Training batches on cuda:  18%|█▊        | 35/191 [00:00<00:04, 37.42batch/s]

Training batches on cuda:  20%|██        | 39/191 [00:01<00:04, 37.57batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:01<00:03, 37.23batch/s]

Training batches on cuda:  25%|██▌       | 48/191 [00:01<00:03, 38.21batch/s]

Training batches on cuda:  27%|██▋       | 52/191 [00:01<00:03, 38.12batch/s]

Training batches on cuda:  29%|██▉       | 56/191 [00:01<00:03, 38.48batch/s]

Training batches on cuda:  31%|███▏      | 60/191 [00:01<00:03, 38.09batch/s]

Training batches on cuda:  34%|███▎      | 64/191 [00:01<00:03, 38.10batch/s]

Training batches on cuda:  36%|███▌      | 68/191 [00:01<00:03, 38.29batch/s]

Training batches on cuda:  38%|███▊      | 72/191 [00:01<00:03, 38.28batch/s]

Training batches on cuda:  40%|███▉      | 76/191 [00:02<00:03, 36.21batch/s]

Training batches on cuda:  42%|████▏     | 80/191 [00:02<00:03, 36.02batch/s]

Training batches on cuda:  44%|████▍     | 84/191 [00:02<00:02, 35.92batch/s]

Training batches on cuda:  47%|████▋     | 89/191 [00:02<00:02, 37.39batch/s]

Training batches on cuda:  49%|████▊     | 93/191 [00:02<00:02, 37.96batch/s]

Training batches on cuda:  51%|█████▏    | 98/191 [00:02<00:02, 38.80batch/s]

Training batches on cuda:  54%|█████▍    | 103/191 [00:02<00:02, 39.32batch/s]

Training batches on cuda:  57%|█████▋    | 108/191 [00:02<00:02, 40.06batch/s]

Training batches on cuda:  59%|█████▉    | 113/191 [00:02<00:01, 40.80batch/s]

Training batches on cuda:  62%|██████▏   | 118/191 [00:03<00:01, 41.02batch/s]

Training batches on cuda:  64%|██████▍   | 123/191 [00:03<00:01, 39.12batch/s]

Training batches on cuda:  66%|██████▋   | 127/191 [00:03<00:01, 38.99batch/s]

Training batches on cuda:  69%|██████▊   | 131/191 [00:03<00:01, 37.97batch/s]

Training batches on cuda:  71%|███████   | 136/191 [00:03<00:01, 38.77batch/s]

Training batches on cuda:  74%|███████▍  | 141/191 [00:03<00:01, 39.73batch/s]

Training batches on cuda:  76%|███████▋  | 146/191 [00:03<00:01, 40.07batch/s]

Training batches on cuda:  79%|███████▉  | 151/191 [00:03<00:01, 39.00batch/s]

Training batches on cuda:  82%|████████▏ | 156/191 [00:04<00:00, 39.76batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:04<00:00, 39.74batch/s]

Training batches on cuda:  86%|████████▌ | 164/191 [00:04<00:00, 39.02batch/s]

Training batches on cuda:  88%|████████▊ | 168/191 [00:04<00:00, 39.25batch/s]

Training batches on cuda:  91%|█████████ | 173/191 [00:04<00:00, 39.62batch/s]

Training batches on cuda:  93%|█████████▎| 178/191 [00:04<00:00, 39.88batch/s]

Training batches on cuda:  95%|█████████▌| 182/191 [00:04<00:00, 39.81batch/s]

Training batches on cuda:  97%|█████████▋| 186/191 [00:04<00:00, 39.77batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:04<00:00, 39.21batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:20<00:35,  5.02s/epoch, loss=0.00188, prev_loss=0.00188]
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:30,  5.09s/epoch, loss=0.00188, prev_loss=0.00188]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:07, 24.17batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:06, 27.23batch/s]

Training batches on cuda:   6%|▌         | 11/191 [00:00<00:05, 30.07batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:05, 32.46batch/s]

Training batches on cuda:  10%|▉         | 19/191 [00:00<00:05, 34.37batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:04, 35.69batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:04, 37.14batch/s]

Training batches on cuda:  17%|█▋        | 33/191 [00:00<00:04, 38.08batch/s]

Training batches on cuda:  20%|█▉        | 38/191 [00:00<00:03, 39.03batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:01<00:03, 39.37batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:03, 39.48batch/s]

Training batches on cuda:  27%|██▋       | 51/191 [00:01<00:03, 38.39batch/s]

Training batches on cuda:  29%|██▉       | 56/191 [00:01<00:03, 39.01batch/s]

Training batches on cuda:  32%|███▏      | 61/191 [00:01<00:03, 39.42batch/s]

Training batches on cuda:  35%|███▍      | 66/191 [00:01<00:03, 39.54batch/s]

Training batches on cuda:  37%|███▋      | 70/191 [00:01<00:03, 35.24batch/s]

Training batches on cuda:  39%|███▊      | 74/191 [00:01<00:03, 35.63batch/s]

Training batches on cuda:  41%|████      | 78/191 [00:02<00:03, 36.51batch/s]

Training batches on cuda:  43%|████▎     | 82/191 [00:02<00:02, 36.99batch/s]

Training batches on cuda:  46%|████▌     | 87/191 [00:02<00:02, 37.92batch/s]

Training batches on cuda:  48%|████▊     | 91/191 [00:02<00:02, 37.82batch/s]

Training batches on cuda:  50%|█████     | 96/191 [00:02<00:02, 38.65batch/s]

Training batches on cuda:  52%|█████▏    | 100/191 [00:02<00:02, 38.92batch/s]

Training batches on cuda:  54%|█████▍    | 104/191 [00:02<00:02, 38.96batch/s]

Training batches on cuda:  57%|█████▋    | 108/191 [00:02<00:02, 38.24batch/s]

Training batches on cuda:  59%|█████▊    | 112/191 [00:02<00:02, 38.49batch/s]

Training batches on cuda:  61%|██████    | 116/191 [00:03<00:01, 38.49batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:03<00:01, 38.21batch/s]

Training batches on cuda:  65%|██████▍   | 124/191 [00:03<00:01, 38.03batch/s]

Training batches on cuda:  68%|██████▊   | 129/191 [00:03<00:01, 38.71batch/s]

Training batches on cuda:  70%|███████   | 134/191 [00:03<00:01, 39.59batch/s]

Training batches on cuda:  73%|███████▎  | 139/191 [00:03<00:01, 40.23batch/s]

Training batches on cuda:  75%|███████▌  | 144/191 [00:03<00:01, 40.47batch/s]

Training batches on cuda:  78%|███████▊  | 149/191 [00:03<00:01, 40.73batch/s]

Training batches on cuda:  81%|████████  | 154/191 [00:03<00:00, 41.20batch/s]

Training batches on cuda:  83%|████████▎ | 159/191 [00:04<00:00, 40.76batch/s]

Training batches on cuda:  86%|████████▌ | 164/191 [00:04<00:00, 41.01batch/s]

Training batches on cuda:  88%|████████▊ | 169/191 [00:04<00:00, 40.06batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:04<00:00, 38.22batch/s]

Training batches on cuda:  94%|█████████▎| 179/191 [00:04<00:00, 39.43batch/s]

Training batches on cuda:  96%|█████████▋| 184/191 [00:04<00:00, 39.90batch/s]

Training batches on cuda:  99%|█████████▉| 189/191 [00:04<00:00, 40.23batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:25<00:30,  5.09s/epoch, loss=0.00188, prev_loss=0.00188]
Training epochs on cuda:  50%|█████     | 5/10 [00:25<00:25,  5.12s/epoch, loss=0.00188, prev_loss=0.00188]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/191 [00:00<00:20,  9.24batch/s]

Training batches on cuda:   3%|▎         | 5/191 [00:00<00:15, 11.96batch/s]

Training batches on cuda:   5%|▍         | 9/191 [00:00<00:12, 14.97batch/s]

Training batches on cuda:   7%|▋         | 13/191 [00:00<00:09, 17.98batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:08, 21.42batch/s]

Training batches on cuda:  11%|█         | 21/191 [00:00<00:06, 24.64batch/s]

Training batches on cuda:  13%|█▎        | 25/191 [00:00<00:06, 27.40batch/s]

Training batches on cuda:  15%|█▌        | 29/191 [00:00<00:05, 30.24batch/s]

Training batches on cuda:  17%|█▋        | 33/191 [00:00<00:04, 32.63batch/s]

Training batches on cuda:  20%|█▉        | 38/191 [00:01<00:04, 34.66batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:01<00:04, 36.34batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:03, 37.13batch/s]

Training batches on cuda:  27%|██▋       | 52/191 [00:01<00:03, 38.65batch/s]

Training batches on cuda:  30%|██▉       | 57/191 [00:01<00:03, 39.66batch/s]

Training batches on cuda:  32%|███▏      | 62/191 [00:01<00:03, 37.49batch/s]

Training batches on cuda:  35%|███▌      | 67/191 [00:01<00:03, 38.74batch/s]

Training batches on cuda:  38%|███▊      | 72/191 [00:01<00:03, 39.18batch/s]

Training batches on cuda:  40%|███▉      | 76/191 [00:02<00:02, 39.38batch/s]

Training batches on cuda:  42%|████▏     | 80/191 [00:02<00:02, 38.87batch/s]

Training batches on cuda:  45%|████▍     | 85/191 [00:02<00:02, 38.81batch/s]

Training batches on cuda:  47%|████▋     | 89/191 [00:02<00:02, 38.12batch/s]

Training batches on cuda:  49%|████▊     | 93/191 [00:02<00:02, 38.51batch/s]

Training batches on cuda:  51%|█████▏    | 98/191 [00:02<00:02, 39.14batch/s]

Training batches on cuda:  53%|█████▎    | 102/191 [00:02<00:02, 37.44batch/s]

Training batches on cuda:  56%|█████▌    | 107/191 [00:02<00:02, 38.23batch/s]

Training batches on cuda:  59%|█████▊    | 112/191 [00:02<00:02, 38.88batch/s]

Training batches on cuda:  61%|██████▏   | 117/191 [00:03<00:01, 39.27batch/s]

Training batches on cuda:  64%|██████▍   | 122/191 [00:03<00:01, 39.69batch/s]

Training batches on cuda:  66%|██████▋   | 127/191 [00:03<00:01, 39.87batch/s]

Training batches on cuda:  69%|██████▉   | 132/191 [00:03<00:01, 40.13batch/s]

Training batches on cuda:  72%|███████▏  | 137/191 [00:03<00:01, 40.62batch/s]

Training batches on cuda:  74%|███████▍  | 142/191 [00:03<00:01, 41.15batch/s]

Training batches on cuda:  77%|███████▋  | 147/191 [00:03<00:01, 41.61batch/s]

Training batches on cuda:  80%|███████▉  | 152/191 [00:03<00:00, 40.47batch/s]

Training batches on cuda:  82%|████████▏ | 157/191 [00:04<00:00, 39.99batch/s]

Training batches on cuda:  85%|████████▍ | 162/191 [00:04<00:00, 40.26batch/s]

Training batches on cuda:  87%|████████▋ | 167/191 [00:04<00:00, 39.79batch/s]

Training batches on cuda:  90%|████████▉ | 171/191 [00:04<00:00, 39.21batch/s]

Training batches on cuda:  92%|█████████▏| 175/191 [00:04<00:00, 38.48batch/s]

Training batches on cuda:  94%|█████████▎| 179/191 [00:04<00:00, 37.43batch/s]

Training batches on cuda:  96%|█████████▌| 183/191 [00:04<00:00, 37.98batch/s]

Training batches on cuda:  98%|█████████▊| 187/191 [00:04<00:00, 38.42batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:25,  5.12s/epoch, loss=0.00188, prev_loss=0.00188]
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:20,  5.16s/epoch, loss=0.00188, prev_loss=0.00188]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/191 [00:00<00:11, 16.51batch/s]

Training batches on cuda:   3%|▎         | 5/191 [00:00<00:09, 19.04batch/s]

Training batches on cuda:   5%|▍         | 9/191 [00:00<00:08, 22.12batch/s]

Training batches on cuda:   7%|▋         | 13/191 [00:00<00:07, 24.53batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:06, 27.39batch/s]

Training batches on cuda:  12%|█▏        | 22/191 [00:00<00:05, 30.38batch/s]

Training batches on cuda:  14%|█▎        | 26/191 [00:00<00:05, 32.72batch/s]

Training batches on cuda:  16%|█▌        | 30/191 [00:00<00:04, 34.58batch/s]

Training batches on cuda:  18%|█▊        | 35/191 [00:01<00:04, 36.51batch/s]

Training batches on cuda:  21%|██        | 40/191 [00:01<00:04, 37.46batch/s]

Training batches on cuda:  24%|██▎       | 45/191 [00:01<00:03, 38.72batch/s]

Training batches on cuda:  26%|██▌       | 50/191 [00:01<00:03, 39.42batch/s]

Training batches on cuda:  29%|██▉       | 55/191 [00:01<00:03, 40.32batch/s]

Training batches on cuda:  31%|███▏      | 60/191 [00:01<00:03, 40.44batch/s]

Training batches on cuda:  34%|███▍      | 65/191 [00:01<00:03, 39.75batch/s]

Training batches on cuda:  36%|███▌      | 69/191 [00:01<00:03, 39.80batch/s]

Training batches on cuda:  39%|███▊      | 74/191 [00:01<00:02, 40.05batch/s]

Training batches on cuda:  41%|████▏     | 79/191 [00:02<00:02, 40.05batch/s]

Training batches on cuda:  44%|████▍     | 84/191 [00:02<00:02, 40.19batch/s]

Training batches on cuda:  47%|████▋     | 89/191 [00:02<00:02, 40.16batch/s]

Training batches on cuda:  49%|████▉     | 94/191 [00:02<00:02, 40.28batch/s]

Training batches on cuda:  52%|█████▏    | 99/191 [00:02<00:02, 40.32batch/s]

Training batches on cuda:  54%|█████▍    | 104/191 [00:02<00:02, 40.22batch/s]

Training batches on cuda:  57%|█████▋    | 109/191 [00:02<00:02, 40.28batch/s]

Training batches on cuda:  60%|█████▉    | 114/191 [00:02<00:01, 40.27batch/s]

Training batches on cuda:  62%|██████▏   | 119/191 [00:03<00:01, 40.31batch/s]

Training batches on cuda:  65%|██████▍   | 124/191 [00:03<00:01, 40.33batch/s]

Training batches on cuda:  68%|██████▊   | 129/191 [00:03<00:01, 40.52batch/s]

Training batches on cuda:  70%|███████   | 134/191 [00:03<00:01, 40.62batch/s]

Training batches on cuda:  73%|███████▎  | 139/191 [00:03<00:01, 40.68batch/s]

Training batches on cuda:  75%|███████▌  | 144/191 [00:03<00:01, 39.58batch/s]

Training batches on cuda:  77%|███████▋  | 148/191 [00:03<00:01, 38.74batch/s]

Training batches on cuda:  80%|███████▉  | 152/191 [00:03<00:01, 38.27batch/s]

Training batches on cuda:  82%|████████▏ | 156/191 [00:04<00:00, 38.56batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:04<00:00, 38.68batch/s]

Training batches on cuda:  86%|████████▌ | 164/191 [00:04<00:00, 39.05batch/s]

Training batches on cuda:  88%|████████▊ | 169/191 [00:04<00:00, 40.18batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:04<00:00, 40.37batch/s]

Training batches on cuda:  94%|█████████▎| 179/191 [00:04<00:00, 40.04batch/s]

Training batches on cuda:  96%|█████████▋| 184/191 [00:04<00:00, 40.43batch/s]

Training batches on cuda:  99%|█████████▉| 189/191 [00:04<00:00, 39.74batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:35<00:20,  5.16s/epoch, loss=0.00188, prev_loss=0.00188]
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:15,  5.14s/epoch, loss=0.00188, prev_loss=0.00188]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:06, 27.09batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:06, 29.47batch/s]

Training batches on cuda:   6%|▌         | 11/191 [00:00<00:06, 29.95batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:05, 30.90batch/s]

Training batches on cuda:  10%|▉         | 19/191 [00:00<00:05, 32.71batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:04, 34.50batch/s]

Training batches on cuda:  14%|█▍        | 27/191 [00:00<00:04, 35.07batch/s]

Training batches on cuda:  16%|█▌        | 31/191 [00:00<00:04, 36.32batch/s]

Training batches on cuda:  19%|█▉        | 36/191 [00:01<00:04, 37.48batch/s]

Training batches on cuda:  21%|██▏       | 41/191 [00:01<00:03, 39.10batch/s]

Training batches on cuda:  24%|██▍       | 46/191 [00:01<00:03, 39.68batch/s]

Training batches on cuda:  27%|██▋       | 51/191 [00:01<00:03, 39.95batch/s]

Training batches on cuda:  29%|██▉       | 55/191 [00:01<00:03, 36.36batch/s]

Training batches on cuda:  31%|███       | 59/191 [00:01<00:03, 37.12batch/s]

Training batches on cuda:  33%|███▎      | 63/191 [00:01<00:03, 37.76batch/s]

Training batches on cuda:  36%|███▌      | 68/191 [00:01<00:03, 38.63batch/s]

Training batches on cuda:  38%|███▊      | 72/191 [00:01<00:03, 38.96batch/s]

Training batches on cuda:  40%|███▉      | 76/191 [00:02<00:02, 38.85batch/s]

Training batches on cuda:  42%|████▏     | 81/191 [00:02<00:02, 39.53batch/s]

Training batches on cuda:  45%|████▌     | 86/191 [00:02<00:02, 39.90batch/s]

Training batches on cuda:  48%|████▊     | 91/191 [00:02<00:02, 40.19batch/s]

Training batches on cuda:  50%|█████     | 96/191 [00:02<00:02, 40.53batch/s]

Training batches on cuda:  53%|█████▎    | 101/191 [00:02<00:02, 39.35batch/s]

Training batches on cuda:  55%|█████▍    | 105/191 [00:02<00:02, 38.45batch/s]

Training batches on cuda:  58%|█████▊    | 110/191 [00:02<00:02, 39.05batch/s]

Training batches on cuda:  60%|██████    | 115/191 [00:03<00:01, 39.56batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:03<00:01, 39.93batch/s]

Training batches on cuda:  65%|██████▌   | 125/191 [00:03<00:01, 40.15batch/s]

Training batches on cuda:  68%|██████▊   | 130/191 [00:03<00:01, 39.83batch/s]

Training batches on cuda:  71%|███████   | 135/191 [00:03<00:01, 40.19batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 40.17batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 40.04batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:01, 40.00batch/s]

Training batches on cuda:  81%|████████  | 155/191 [00:04<00:00, 39.22batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:04<00:00, 39.72batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:04<00:00, 39.99batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:04<00:00, 40.39batch/s]

Training batches on cuda:  92%|█████████▏| 175/191 [00:04<00:00, 40.05batch/s]

Training batches on cuda:  94%|█████████▍| 180/191 [00:04<00:00, 40.06batch/s]

Training batches on cuda:  97%|█████████▋| 185/191 [00:04<00:00, 39.48batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:04<00:00, 39.97batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:40<00:15,  5.14s/epoch, loss=0.00188, prev_loss=0.00188]
Training epochs on cuda:  80%|████████  | 8/10 [00:40<00:10,  5.16s/epoch, loss=0.00188, prev_loss=0.00188]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/191 [00:00<00:09, 19.07batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:08, 22.49batch/s]

Training batches on cuda:   5%|▌         | 10/191 [00:00<00:06, 25.88batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:06, 29.18batch/s]

Training batches on cuda:  10%|█         | 20/191 [00:00<00:05, 32.09batch/s]

Training batches on cuda:  13%|█▎        | 25/191 [00:00<00:04, 34.51batch/s]

Training batches on cuda:  16%|█▌        | 30/191 [00:00<00:04, 36.41batch/s]

Training batches on cuda:  18%|█▊        | 35/191 [00:00<00:04, 37.78batch/s]

Training batches on cuda:  21%|██        | 40/191 [00:01<00:03, 39.09batch/s]

Training batches on cuda:  24%|██▎       | 45/191 [00:01<00:03, 40.22batch/s]

Training batches on cuda:  26%|██▌       | 50/191 [00:01<00:03, 40.89batch/s]

Training batches on cuda:  29%|██▉       | 55/191 [00:01<00:03, 37.42batch/s]

Training batches on cuda:  31%|███▏      | 60/191 [00:01<00:03, 38.81batch/s]

Training batches on cuda:  34%|███▍      | 65/191 [00:01<00:03, 39.82batch/s]

Training batches on cuda:  37%|███▋      | 70/191 [00:01<00:02, 40.60batch/s]

Training batches on cuda:  39%|███▉      | 75/191 [00:01<00:02, 41.01batch/s]

Training batches on cuda:  42%|████▏     | 80/191 [00:02<00:02, 41.26batch/s]

Training batches on cuda:  45%|████▍     | 85/191 [00:02<00:02, 41.62batch/s]

Training batches on cuda:  47%|████▋     | 90/191 [00:02<00:02, 41.83batch/s]

Training batches on cuda:  50%|████▉     | 95/191 [00:02<00:02, 41.92batch/s]

Training batches on cuda:  52%|█████▏    | 100/191 [00:02<00:02, 41.97batch/s]

Training batches on cuda:  55%|█████▍    | 105/191 [00:02<00:02, 41.82batch/s]

Training batches on cuda:  58%|█████▊    | 110/191 [00:02<00:01, 41.83batch/s]

Training batches on cuda:  60%|██████    | 115/191 [00:02<00:01, 41.99batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:02<00:01, 42.09batch/s]

Training batches on cuda:  65%|██████▌   | 125/191 [00:03<00:01, 41.96batch/s]

Training batches on cuda:  68%|██████▊   | 130/191 [00:03<00:01, 42.03batch/s]

Training batches on cuda:  71%|███████   | 135/191 [00:03<00:01, 41.98batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 42.04batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 41.95batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:00, 41.97batch/s]

Training batches on cuda:  81%|████████  | 155/191 [00:03<00:00, 41.80batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:03<00:00, 41.79batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:04<00:00, 41.66batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:04<00:00, 41.43batch/s]

Training batches on cuda:  92%|█████████▏| 175/191 [00:04<00:00, 41.66batch/s]

Training batches on cuda:  94%|█████████▍| 180/191 [00:04<00:00, 42.00batch/s]

Training batches on cuda:  97%|█████████▋| 185/191 [00:04<00:00, 41.64batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:04<00:00, 41.97batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:45<00:10,  5.16s/epoch, loss=0.00188, prev_loss=0.00188]
Training epochs on cuda:  90%|█████████ | 9/10 [00:45<00:05,  5.07s/epoch, loss=0.00188, prev_loss=0.00188]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:06, 27.51batch/s]

Training batches on cuda:   4%|▍         | 8/191 [00:00<00:06, 30.43batch/s]

Training batches on cuda:   7%|▋         | 13/191 [00:00<00:05, 33.29batch/s]

Training batches on cuda:   9%|▉         | 18/191 [00:00<00:04, 35.65batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:04, 37.26batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:04, 38.69batch/s]

Training batches on cuda:  17%|█▋        | 33/191 [00:00<00:04, 39.49batch/s]

Training batches on cuda:  19%|█▉        | 37/191 [00:00<00:03, 39.64batch/s]

Training batches on cuda:  22%|██▏       | 42/191 [00:01<00:03, 40.07batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:03, 40.82batch/s]

Training batches on cuda:  27%|██▋       | 52/191 [00:01<00:03, 41.10batch/s]

Training batches on cuda:  30%|██▉       | 57/191 [00:01<00:03, 41.58batch/s]

Training batches on cuda:  32%|███▏      | 62/191 [00:01<00:03, 41.66batch/s]

Training batches on cuda:  35%|███▌      | 67/191 [00:01<00:03, 37.54batch/s]

Training batches on cuda:  38%|███▊      | 72/191 [00:01<00:03, 38.81batch/s]

Training batches on cuda:  40%|████      | 77/191 [00:01<00:02, 39.94batch/s]

Training batches on cuda:  43%|████▎     | 82/191 [00:02<00:02, 39.90batch/s]

Training batches on cuda:  46%|████▌     | 87/191 [00:02<00:02, 40.64batch/s]

Training batches on cuda:  48%|████▊     | 92/191 [00:02<00:02, 41.14batch/s]

Training batches on cuda:  51%|█████     | 97/191 [00:02<00:02, 41.58batch/s]

Training batches on cuda:  53%|█████▎    | 102/191 [00:02<00:02, 41.97batch/s]

Training batches on cuda:  56%|█████▌    | 107/191 [00:02<00:01, 42.37batch/s]

Training batches on cuda:  59%|█████▊    | 112/191 [00:02<00:01, 42.44batch/s]

Training batches on cuda:  61%|██████▏   | 117/191 [00:02<00:01, 42.16batch/s]

Training batches on cuda:  64%|██████▍   | 122/191 [00:02<00:01, 42.28batch/s]

Training batches on cuda:  66%|██████▋   | 127/191 [00:03<00:01, 42.28batch/s]

Training batches on cuda:  69%|██████▉   | 132/191 [00:03<00:01, 42.30batch/s]

Training batches on cuda:  72%|███████▏  | 137/191 [00:03<00:01, 42.44batch/s]

Training batches on cuda:  74%|███████▍  | 142/191 [00:03<00:01, 42.59batch/s]

Training batches on cuda:  77%|███████▋  | 147/191 [00:03<00:01, 42.51batch/s]

Training batches on cuda:  80%|███████▉  | 152/191 [00:03<00:00, 41.70batch/s]

Training batches on cuda:  82%|████████▏ | 157/191 [00:03<00:00, 41.76batch/s]

Training batches on cuda:  85%|████████▍ | 162/191 [00:03<00:00, 41.85batch/s]

Training batches on cuda:  87%|████████▋ | 167/191 [00:04<00:00, 42.01batch/s]

Training batches on cuda:  90%|█████████ | 172/191 [00:04<00:00, 42.18batch/s]

Training batches on cuda:  93%|█████████▎| 177/191 [00:04<00:00, 42.42batch/s]

Training batches on cuda:  95%|█████████▌| 182/191 [00:04<00:00, 42.01batch/s]

Training batches on cuda:  98%|█████████▊| 187/191 [00:04<00:00, 42.18batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:50<00:05,  5.07s/epoch, loss=0.00188, prev_loss=0.00188]2021-09-16:17:52:26,956 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:52:27,971 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:52:42,912 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:17:52:43,42 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00,  9.84s/epoch, loss=0.00188, prev_loss=0.00188]
Training epochs on cuda: 100%|██████████| 10/10 [01:06<00:00,  6.68s/epoch, loss=0.00188, prev_loss=0.00188]
2021-09-16:17:52:43,42 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpwouicja9'
2021-09-16:17:52:43,109 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpwouicja9' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.62ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.60ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.61ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:52:57,927 INFO     [evaluator.py:657] Evaluation took 14.80s seconds
[I 2021-09-16 17:52:57,932] Trial 17 finished with value: 0.0020684916366471295 and parameters: {'loss.margin': 3, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.16983071438474986, 'optimizer.lr': 0.07833556324126592, 'negative_sampler.num_negs_per_pos': 37, 'training.batch_size': 1632}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:52:57,952 WARNING  [api.py:823] No random seed is specified. Setting to 3123884019.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/334 [00:00<00:04, 78.14batch/s]

Training batches on cuda:   6%|▌         | 19/334 [00:00<00:03, 84.53batch/s]

Training batches on cuda:   9%|▉         | 30/334 [00:00<00:03, 90.34batch/s]

Training batches on cuda:  12%|█▏        | 41/334 [00:00<00:03, 93.56batch/s]

Training batches on cuda:  16%|█▌        | 52/334 [00:00<00:02, 97.06batch/s]

Training batches on cuda:  19%|█▉        | 63/334 [00:00<00:02, 100.04batch/s]

Training batches on cuda:  22%|██▏       | 74/334 [00:00<00:02, 101.23batch/s]

Training batches on cuda:  25%|██▌       | 85/334 [00:00<00:02, 102.32batch/s]

Training batches on cuda:  29%|██▊       | 96/334 [00:00<00:02, 103.18batch/s]

Training batches on cuda:  32%|███▏      | 107/334 [00:01<00:02, 100.82batch/s]

Training batches on cuda:  35%|███▌      | 118/334 [00:01<00:02, 103.28batch/s]

Training batches on cuda:  39%|███▊      | 129/334 [00:01<00:01, 105.14batch/s]

Training batches on cuda:  42%|████▏     | 140/334 [00:01<00:01, 105.47batch/s]

Training batches on cuda:  46%|████▌     | 152/334 [00:01<00:01, 107.39batch/s]

Training batches on cuda:  49%|████▉     | 163/334 [00:01<00:01, 107.83batch/s]

Training batches on cuda:  52%|█████▏    | 175/334 [00:01<00:01, 109.18batch/s]

Training batches on cuda:  56%|█████▌    | 187/334 [00:01<00:01, 110.01batch/s]

Training batches on cuda:  60%|█████▉    | 199/334 [00:01<00:01, 110.44batch/s]

Training batches on cuda:  63%|██████▎   | 211/334 [00:01<00:01, 110.27batch/s]

Training batches on cuda:  67%|██████▋   | 223/334 [00:02<00:01, 110.16batch/s]

Training batches on cuda:  70%|███████   | 235/334 [00:02<00:00, 110.49batch/s]

Training batches on cuda:  74%|███████▍  | 247/334 [00:02<00:00, 110.24batch/s]

Training batches on cuda:  78%|███████▊  | 259/334 [00:02<00:00, 109.95batch/s]

Training batches on cuda:  81%|████████  | 270/334 [00:02<00:00, 109.83batch/s]

Training batches on cuda:  84%|████████▍ | 282/334 [00:02<00:00, 110.31batch/s]

Training batches on cuda:  88%|████████▊ | 294/334 [00:02<00:00, 110.38batch/s]

Training batches on cuda:  92%|█████████▏| 306/334 [00:02<00:00, 110.02batch/s]

Training batches on cuda:  95%|█████████▌| 318/334 [00:02<00:00, 110.11batch/s]

Training batches on cuda:  99%|█████████▉| 330/334 [00:03<00:00, 110.29batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.00221, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:29,  3.32s/epoch, loss=0.00221, prev_loss=nan]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 5/334 [00:00<00:06, 49.04batch/s]

Training batches on cuda:   4%|▍         | 15/334 [00:00<00:05, 57.36batch/s]

Training batches on cuda:   7%|▋         | 25/334 [00:00<00:04, 65.12batch/s]

Training batches on cuda:  11%|█         | 36/334 [00:00<00:04, 73.83batch/s]

Training batches on cuda:  14%|█▍        | 47/334 [00:00<00:03, 80.35batch/s]

Training batches on cuda:  17%|█▋        | 58/334 [00:00<00:03, 87.18batch/s]

Training batches on cuda:  21%|██        | 69/334 [00:00<00:02, 91.42batch/s]

Training batches on cuda:  24%|██▍       | 80/334 [00:00<00:02, 95.23batch/s]

Training batches on cuda:  27%|██▋       | 91/334 [00:00<00:02, 98.43batch/s]

Training batches on cuda:  31%|███       | 102/334 [00:01<00:02, 98.92batch/s]

Training batches on cuda:  34%|███▍      | 113/334 [00:01<00:02, 99.84batch/s]

Training batches on cuda:  37%|███▋      | 124/334 [00:01<00:02, 101.99batch/s]

Training batches on cuda:  40%|████      | 135/334 [00:01<00:01, 101.03batch/s]

Training batches on cuda:  44%|████▎     | 146/334 [00:01<00:01, 97.56batch/s] 

Training batches on cuda:  47%|████▋     | 157/334 [00:01<00:01, 99.47batch/s]

Training batches on cuda:  50%|█████     | 168/334 [00:01<00:01, 101.92batch/s]

Training batches on cuda:  54%|█████▎    | 179/334 [00:01<00:01, 101.21batch/s]

Training batches on cuda:  57%|█████▋    | 190/334 [00:01<00:01, 101.00batch/s]

Training batches on cuda:  60%|██████    | 201/334 [00:02<00:01, 99.38batch/s] 

Training batches on cuda:  63%|██████▎   | 212/334 [00:02<00:01, 101.06batch/s]

Training batches on cuda:  67%|██████▋   | 223/334 [00:02<00:01, 103.32batch/s]

Training batches on cuda:  71%|███████   | 236/334 [00:02<00:00, 108.97batch/s]

Training batches on cuda:  75%|███████▍  | 249/334 [00:02<00:00, 112.76batch/s]

Training batches on cuda:  78%|███████▊  | 261/334 [00:02<00:00, 114.52batch/s]

Training batches on cuda:  82%|████████▏ | 273/334 [00:02<00:00, 115.38batch/s]

Training batches on cuda:  85%|████████▌ | 285/334 [00:02<00:00, 110.53batch/s]

Training batches on cuda:  89%|████████▉ | 297/334 [00:02<00:00, 106.47batch/s]

Training batches on cuda:  92%|█████████▏| 308/334 [00:02<00:00, 106.32batch/s]

Training batches on cuda:  96%|█████████▌| 319/334 [00:03<00:00, 107.06batch/s]

Training batches on cuda:  99%|█████████▉| 331/334 [00:03<00:00, 109.79batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:29,  3.32s/epoch, loss=0.00218, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:26,  3.37s/epoch, loss=0.00218, prev_loss=nan]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/334 [00:00<00:04, 73.54batch/s]

Training batches on cuda:   6%|▌         | 19/334 [00:00<00:03, 80.34batch/s]

Training batches on cuda:   9%|▉         | 30/334 [00:00<00:03, 86.81batch/s]

Training batches on cuda:  12%|█▏        | 41/334 [00:00<00:03, 92.26batch/s]

Training batches on cuda:  16%|█▌        | 52/334 [00:00<00:02, 96.01batch/s]

Training batches on cuda:  19%|█▉        | 63/334 [00:00<00:02, 99.54batch/s]

Training batches on cuda:  22%|██▏       | 74/334 [00:00<00:02, 101.63batch/s]

Training batches on cuda:  25%|██▌       | 84/334 [00:00<00:02, 100.70batch/s]

Training batches on cuda:  28%|██▊       | 95/334 [00:00<00:02, 102.84batch/s]

Training batches on cuda:  32%|███▏      | 106/334 [00:01<00:02, 103.90batch/s]

Training batches on cuda:  35%|███▌      | 117/334 [00:01<00:02, 101.94batch/s]

Training batches on cuda:  38%|███▊      | 128/334 [00:01<00:02, 101.23batch/s]

Training batches on cuda:  42%|████▏     | 139/334 [00:01<00:01, 103.12batch/s]

Training batches on cuda:  45%|████▍     | 150/334 [00:01<00:01, 99.37batch/s] 

Training batches on cuda:  48%|████▊     | 160/334 [00:01<00:01, 97.89batch/s]

Training batches on cuda:  51%|█████     | 171/334 [00:01<00:01, 100.42batch/s]

Training batches on cuda:  54%|█████▍    | 182/334 [00:01<00:01, 99.05batch/s] 

Training batches on cuda:  58%|█████▊    | 195/334 [00:01<00:01, 104.43batch/s]

Training batches on cuda:  62%|██████▏   | 207/334 [00:02<00:01, 106.95batch/s]

Training batches on cuda:  66%|██████▌   | 220/334 [00:02<00:01, 110.36batch/s]

Training batches on cuda:  69%|██████▉   | 232/334 [00:02<00:00, 109.16batch/s]

Training batches on cuda:  73%|███████▎  | 243/334 [00:02<00:00, 96.35batch/s] 

Training batches on cuda:  76%|███████▌  | 253/334 [00:02<00:00, 94.95batch/s]

Training batches on cuda:  79%|███████▉  | 264/334 [00:02<00:00, 98.52batch/s]

Training batches on cuda:  82%|████████▏ | 275/334 [00:02<00:00, 97.11batch/s]

Training batches on cuda:  86%|████████▌ | 286/334 [00:02<00:00, 99.41batch/s]

Training batches on cuda:  89%|████████▉ | 297/334 [00:02<00:00, 98.96batch/s]

Training batches on cuda:  92%|█████████▏| 308/334 [00:03<00:00, 100.61batch/s]

Training batches on cuda:  96%|█████████▌| 320/334 [00:03<00:00, 104.14batch/s]

Training batches on cuda:  99%|█████████▉| 331/334 [00:03<00:00, 103.90batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:10<00:26,  3.37s/epoch, loss=0.00218, prev_loss=0.00218]
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:23,  3.42s/epoch, loss=0.00218, prev_loss=0.00218]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/334 [00:00<00:04, 79.44batch/s]

Training batches on cuda:   6%|▋         | 21/334 [00:00<00:03, 89.38batch/s]

Training batches on cuda:  10%|█         | 34/334 [00:00<00:03, 98.58batch/s]

Training batches on cuda:  13%|█▎        | 45/334 [00:00<00:02, 101.06batch/s]

Training batches on cuda:  17%|█▋        | 57/334 [00:00<00:02, 104.93batch/s]

Training batches on cuda:  21%|██        | 69/334 [00:00<00:02, 108.74batch/s]

Training batches on cuda:  24%|██▍       | 81/334 [00:00<00:02, 111.54batch/s]

Training batches on cuda:  28%|██▊       | 92/334 [00:00<00:03, 78.45batch/s] 

Training batches on cuda:  31%|███       | 104/334 [00:01<00:02, 86.14batch/s]

Training batches on cuda:  35%|███▌      | 117/334 [00:01<00:02, 95.21batch/s]

Training batches on cuda:  39%|███▉      | 130/334 [00:01<00:01, 102.18batch/s]

Training batches on cuda:  43%|████▎     | 143/334 [00:01<00:01, 108.35batch/s]

Training batches on cuda:  47%|████▋     | 156/334 [00:01<00:01, 112.23batch/s]

Training batches on cuda:  50%|█████     | 168/334 [00:01<00:01, 113.99batch/s]

Training batches on cuda:  54%|█████▍    | 181/334 [00:01<00:01, 116.84batch/s]

Training batches on cuda:  58%|█████▊    | 193/334 [00:01<00:01, 117.08batch/s]

Training batches on cuda:  61%|██████▏   | 205/334 [00:01<00:01, 115.18batch/s]

Training batches on cuda:  65%|██████▍   | 217/334 [00:02<00:01, 112.80batch/s]

Training batches on cuda:  69%|██████▉   | 230/334 [00:02<00:00, 113.42batch/s]

Training batches on cuda:  72%|███████▏  | 242/334 [00:02<00:00, 112.62batch/s]

Training batches on cuda:  76%|███████▌  | 254/334 [00:02<00:00, 110.48batch/s]

Training batches on cuda:  80%|███████▉  | 266/334 [00:02<00:00, 109.62batch/s]

Training batches on cuda:  83%|████████▎ | 277/334 [00:02<00:00, 108.78batch/s]

Training batches on cuda:  86%|████████▌ | 288/334 [00:02<00:00, 107.74batch/s]

Training batches on cuda:  90%|████████▉ | 299/334 [00:02<00:00, 107.50batch/s]

Training batches on cuda:  93%|█████████▎| 311/334 [00:02<00:00, 108.53batch/s]

Training batches on cuda:  96%|█████████▋| 322/334 [00:02<00:00, 107.73batch/s]

Training batches on cuda: 100%|██████████| 334/334 [00:03<00:00, 108.52batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:23,  3.42s/epoch, loss=0.00218, prev_loss=0.00218]
Training epochs on cuda:  40%|████      | 4/10 [00:13<00:20,  3.38s/epoch, loss=0.00218, prev_loss=0.00218]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 9/334 [00:00<00:03, 83.83batch/s]

Training batches on cuda:   6%|▌         | 20/334 [00:00<00:03, 89.03batch/s]

Training batches on cuda:   9%|▉         | 31/334 [00:00<00:03, 94.01batch/s]

Training batches on cuda:  12%|█▏        | 41/334 [00:00<00:03, 93.99batch/s]

Training batches on cuda:  16%|█▌        | 52/334 [00:00<00:02, 96.00batch/s]

Training batches on cuda:  19%|█▊        | 62/334 [00:00<00:02, 96.52batch/s]

Training batches on cuda:  22%|██▏       | 75/334 [00:00<00:02, 101.43batch/s]

Training batches on cuda:  26%|██▌       | 86/334 [00:00<00:02, 103.65batch/s]

Training batches on cuda:  29%|██▉       | 98/334 [00:00<00:02, 105.58batch/s]

Training batches on cuda:  33%|███▎      | 109/334 [00:01<00:02, 104.30batch/s]

Training batches on cuda:  36%|███▌      | 120/334 [00:01<00:02, 105.78batch/s]

Training batches on cuda:  40%|███▉      | 132/334 [00:01<00:01, 107.54batch/s]

Training batches on cuda:  43%|████▎     | 144/334 [00:01<00:01, 108.40batch/s]

Training batches on cuda:  46%|████▋     | 155/334 [00:01<00:01, 107.61batch/s]

Training batches on cuda:  50%|█████     | 167/334 [00:01<00:01, 109.54batch/s]

Training batches on cuda:  54%|█████▎    | 179/334 [00:01<00:01, 112.19batch/s]

Training batches on cuda:  57%|█████▋    | 191/334 [00:01<00:01, 112.92batch/s]

Training batches on cuda:  61%|██████    | 203/334 [00:01<00:01, 114.77batch/s]

Training batches on cuda:  65%|██████▍   | 216/334 [00:01<00:01, 117.84batch/s]

Training batches on cuda:  68%|██████▊   | 228/334 [00:02<00:00, 108.73batch/s]

Training batches on cuda:  72%|███████▏  | 240/334 [00:02<00:00, 104.68batch/s]

Training batches on cuda:  76%|███████▌  | 253/334 [00:02<00:00, 109.47batch/s]

Training batches on cuda:  80%|███████▉  | 266/334 [00:02<00:00, 112.87batch/s]

Training batches on cuda:  83%|████████▎ | 278/334 [00:02<00:00, 89.40batch/s] 

Training batches on cuda:  87%|████████▋ | 291/334 [00:02<00:00, 97.53batch/s]

Training batches on cuda:  91%|█████████ | 303/334 [00:02<00:00, 101.02batch/s]

Training batches on cuda:  94%|█████████▍| 314/334 [00:02<00:00, 102.46batch/s]

Training batches on cuda:  97%|█████████▋| 325/334 [00:03<00:00, 100.04batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:20,  3.38s/epoch, loss=0.00218, prev_loss=0.00218]
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:16,  3.40s/epoch, loss=0.00218, prev_loss=0.00218]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 10/334 [00:00<00:03, 97.67batch/s]

Training batches on cuda:   7%|▋         | 22/334 [00:00<00:03, 102.46batch/s]

Training batches on cuda:  10%|█         | 35/334 [00:00<00:02, 108.16batch/s]

Training batches on cuda:  14%|█▍        | 48/334 [00:00<00:02, 112.20batch/s]

Training batches on cuda:  18%|█▊        | 61/334 [00:00<00:02, 115.78batch/s]

Training batches on cuda:  22%|██▏       | 74/334 [00:00<00:02, 118.13batch/s]

Training batches on cuda:  26%|██▌       | 87/334 [00:00<00:02, 120.45batch/s]

Training batches on cuda:  30%|██▉       | 100/334 [00:00<00:01, 121.35batch/s]

Training batches on cuda:  34%|███▍      | 113/334 [00:00<00:01, 122.81batch/s]

Training batches on cuda:  37%|███▋      | 125/334 [00:01<00:01, 120.55batch/s]

Training batches on cuda:  41%|████      | 137/334 [00:01<00:01, 119.92batch/s]

Training batches on cuda:  45%|████▍     | 149/334 [00:01<00:01, 110.02batch/s]

Training batches on cuda:  48%|████▊     | 161/334 [00:01<00:01, 106.00batch/s]

Training batches on cuda:  51%|█████▏    | 172/334 [00:01<00:01, 106.82batch/s]

Training batches on cuda:  55%|█████▍    | 183/334 [00:01<00:01, 107.02batch/s]

Training batches on cuda:  58%|█████▊    | 194/334 [00:01<00:01, 105.14batch/s]

Training batches on cuda:  61%|██████▏   | 205/334 [00:01<00:01, 103.35batch/s]

Training batches on cuda:  65%|██████▍   | 217/334 [00:01<00:01, 105.56batch/s]

Training batches on cuda:  69%|██████▉   | 230/334 [00:02<00:00, 109.78batch/s]

Training batches on cuda:  72%|███████▏  | 242/334 [00:02<00:00, 111.88batch/s]

Training batches on cuda:  76%|███████▌  | 254/334 [00:02<00:00, 113.92batch/s]

Training batches on cuda:  80%|███████▉  | 267/334 [00:02<00:00, 116.59batch/s]

Training batches on cuda:  84%|████████▎ | 279/334 [00:02<00:00, 115.44batch/s]

Training batches on cuda:  87%|████████▋ | 291/334 [00:02<00:00, 116.65batch/s]

Training batches on cuda:  91%|█████████ | 304/334 [00:02<00:00, 117.85batch/s]

Training batches on cuda:  95%|█████████▍| 316/334 [00:02<00:00, 117.84batch/s]

Training batches on cuda:  99%|█████████▊| 329/334 [00:02<00:00, 118.65batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:20<00:16,  3.40s/epoch, loss=0.00218, prev_loss=0.00218]
Training epochs on cuda:  60%|██████    | 6/10 [00:20<00:13,  3.31s/epoch, loss=0.00218, prev_loss=0.00218]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/334 [00:00<00:04, 75.74batch/s]

Training batches on cuda:   5%|▌         | 18/334 [00:00<00:03, 81.28batch/s]

Training batches on cuda:   9%|▉         | 31/334 [00:00<00:03, 90.26batch/s]

Training batches on cuda:  13%|█▎        | 43/334 [00:00<00:03, 96.85batch/s]

Training batches on cuda:  16%|█▋        | 55/334 [00:00<00:02, 100.74batch/s]

Training batches on cuda:  20%|██        | 67/334 [00:00<00:02, 103.98batch/s]

Training batches on cuda:  24%|██▎       | 79/334 [00:00<00:02, 106.71batch/s]

Training batches on cuda:  27%|██▋       | 91/334 [00:00<00:02, 108.35batch/s]

Training batches on cuda:  31%|███       | 102/334 [00:00<00:02, 108.80batch/s]

Training batches on cuda:  34%|███▍      | 113/334 [00:01<00:02, 92.91batch/s] 

Training batches on cuda:  37%|███▋      | 123/334 [00:01<00:02, 94.46batch/s]

Training batches on cuda:  40%|████      | 134/334 [00:01<00:02, 97.45batch/s]

Training batches on cuda:  43%|████▎     | 145/334 [00:01<00:01, 100.73batch/s]

Training batches on cuda:  47%|████▋     | 157/334 [00:01<00:01, 103.74batch/s]

Training batches on cuda:  50%|█████     | 168/334 [00:01<00:01, 105.00batch/s]

Training batches on cuda:  54%|█████▎    | 179/334 [00:01<00:01, 106.10batch/s]

Training batches on cuda:  57%|█████▋    | 190/334 [00:01<00:01, 106.13batch/s]

Training batches on cuda:  60%|██████    | 201/334 [00:01<00:01, 105.87batch/s]

Training batches on cuda:  63%|██████▎   | 212/334 [00:02<00:01, 104.60batch/s]

Training batches on cuda:  67%|██████▋   | 223/334 [00:02<00:01, 103.13batch/s]

Training batches on cuda:  70%|███████   | 234/334 [00:02<00:00, 103.16batch/s]

Training batches on cuda:  74%|███████▎  | 246/334 [00:02<00:00, 107.27batch/s]

Training batches on cuda:  78%|███████▊  | 259/334 [00:02<00:00, 112.01batch/s]

Training batches on cuda:  81%|████████▏ | 272/334 [00:02<00:00, 115.44batch/s]

Training batches on cuda:  85%|████████▌ | 284/334 [00:02<00:00, 116.48batch/s]

Training batches on cuda:  89%|████████▉ | 297/334 [00:02<00:00, 119.83batch/s]

Training batches on cuda:  93%|█████████▎| 310/334 [00:02<00:00, 114.71batch/s]

Training batches on cuda:  97%|█████████▋| 323/334 [00:02<00:00, 116.42batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:23<00:13,  3.31s/epoch, loss=0.00218, prev_loss=0.00218]
Training epochs on cuda:  70%|███████   | 7/10 [00:23<00:09,  3.29s/epoch, loss=0.00218, prev_loss=0.00218]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/334 [00:00<00:04, 74.99batch/s]

Training batches on cuda:   6%|▌         | 19/334 [00:00<00:03, 81.44batch/s]

Training batches on cuda:   9%|▉         | 30/334 [00:00<00:03, 86.79batch/s]

Training batches on cuda:  12%|█▏        | 41/334 [00:00<00:03, 90.62batch/s]

Training batches on cuda:  15%|█▌        | 51/334 [00:00<00:03, 92.63batch/s]

Training batches on cuda:  18%|█▊        | 61/334 [00:00<00:02, 92.84batch/s]

Training batches on cuda:  21%|██▏       | 71/334 [00:00<00:02, 94.38batch/s]

Training batches on cuda:  25%|██▍       | 82/334 [00:00<00:02, 97.34batch/s]

Training batches on cuda:  28%|██▊       | 93/334 [00:00<00:02, 99.54batch/s]

Training batches on cuda:  32%|███▏      | 106/334 [00:01<00:02, 105.06batch/s]

Training batches on cuda:  35%|███▌      | 117/334 [00:01<00:02, 104.51batch/s]

Training batches on cuda:  39%|███▊      | 129/334 [00:01<00:01, 105.67batch/s]

Training batches on cuda:  42%|████▏     | 141/334 [00:01<00:01, 107.44batch/s]

Training batches on cuda:  46%|████▌     | 152/334 [00:01<00:01, 107.68batch/s]

Training batches on cuda:  49%|████▉     | 164/334 [00:01<00:01, 109.18batch/s]

Training batches on cuda:  53%|█████▎    | 176/334 [00:01<00:01, 111.11batch/s]

Training batches on cuda:  56%|█████▋    | 188/334 [00:01<00:01, 110.83batch/s]

Training batches on cuda:  60%|█████▉    | 200/334 [00:01<00:01, 109.20batch/s]

Training batches on cuda:  63%|██████▎   | 212/334 [00:02<00:01, 111.57batch/s]

Training batches on cuda:  67%|██████▋   | 224/334 [00:02<00:00, 113.14batch/s]

Training batches on cuda:  71%|███████   | 237/334 [00:02<00:00, 116.36batch/s]

Training batches on cuda:  75%|███████▍  | 249/334 [00:02<00:00, 115.76batch/s]

Training batches on cuda:  78%|███████▊  | 261/334 [00:02<00:00, 107.22batch/s]

Training batches on cuda:  81%|████████▏ | 272/334 [00:02<00:00, 96.57batch/s] 

Training batches on cuda:  85%|████████▍ | 283/334 [00:02<00:00, 96.95batch/s]

Training batches on cuda:  88%|████████▊ | 293/334 [00:02<00:00, 84.55batch/s]

Training batches on cuda:  91%|█████████ | 304/334 [00:02<00:00, 90.18batch/s]

Training batches on cuda:  94%|█████████▍| 314/334 [00:03<00:00, 92.69batch/s]

Training batches on cuda:  97%|█████████▋| 324/334 [00:03<00:00, 94.51batch/s]

Training batches on cuda: 100%|██████████| 334/334 [00:03<00:00, 95.31batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:26<00:09,  3.29s/epoch, loss=0.00218, prev_loss=0.00218]
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:06,  3.37s/epoch, loss=0.00218, prev_loss=0.00218]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 10/334 [00:00<00:03, 99.26batch/s]

Training batches on cuda:   7%|▋         | 23/334 [00:00<00:02, 105.34batch/s]

Training batches on cuda:  11%|█         | 36/334 [00:00<00:02, 111.34batch/s]

Training batches on cuda:  15%|█▍        | 49/334 [00:00<00:02, 114.12batch/s]

Training batches on cuda:  19%|█▊        | 62/334 [00:00<00:02, 116.84batch/s]

Training batches on cuda:  22%|██▏       | 75/334 [00:00<00:02, 120.16batch/s]

Training batches on cuda:  26%|██▋       | 88/334 [00:00<00:02, 122.08batch/s]

Training batches on cuda:  30%|███       | 101/334 [00:00<00:01, 123.66batch/s]

Training batches on cuda:  34%|███▍      | 114/334 [00:00<00:01, 124.60batch/s]

Training batches on cuda:  38%|███▊      | 127/334 [00:01<00:01, 125.64batch/s]

Training batches on cuda:  42%|████▏     | 140/334 [00:01<00:01, 124.15batch/s]

Training batches on cuda:  46%|████▌     | 153/334 [00:01<00:01, 119.04batch/s]

Training batches on cuda:  49%|████▉     | 165/334 [00:01<00:01, 113.15batch/s]

Training batches on cuda:  53%|█████▎    | 177/334 [00:01<00:01, 112.32batch/s]

Training batches on cuda:  57%|█████▋    | 189/334 [00:01<00:01, 112.08batch/s]

Training batches on cuda:  60%|██████    | 201/334 [00:01<00:01, 110.13batch/s]

Training batches on cuda:  64%|██████▍   | 213/334 [00:01<00:01, 107.28batch/s]

Training batches on cuda:  67%|██████▋   | 224/334 [00:01<00:01, 106.09batch/s]

Training batches on cuda:  70%|███████   | 235/334 [00:02<00:00, 105.54batch/s]

Training batches on cuda:  74%|███████▎  | 246/334 [00:02<00:00, 106.47batch/s]

Training batches on cuda:  77%|███████▋  | 257/334 [00:02<00:00, 106.64batch/s]

Training batches on cuda:  80%|████████  | 268/334 [00:02<00:00, 106.36batch/s]

Training batches on cuda:  84%|████████▎ | 279/334 [00:02<00:00, 105.53batch/s]

Training batches on cuda:  87%|████████▋ | 290/334 [00:02<00:00, 105.49batch/s]

Training batches on cuda:  90%|█████████ | 301/334 [00:02<00:00, 105.17batch/s]

Training batches on cuda:  93%|█████████▎| 312/334 [00:02<00:00, 105.07batch/s]

Training batches on cuda:  97%|█████████▋| 323/334 [00:02<00:00, 104.79batch/s]

Training batches on cuda: 100%|██████████| 334/334 [00:02<00:00, 104.63batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:30<00:06,  3.37s/epoch, loss=0.00218, prev_loss=0.00218]
Training epochs on cuda:  90%|█████████ | 9/10 [00:30<00:03,  3.31s/epoch, loss=0.00218, prev_loss=0.00218]

Training batches on cuda:   0%|          | 0/334 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 10/334 [00:00<00:03, 98.84batch/s]

Training batches on cuda:   7%|▋         | 22/334 [00:00<00:03, 103.73batch/s]

Training batches on cuda:  10%|█         | 35/334 [00:00<00:02, 109.16batch/s]

Training batches on cuda:  15%|█▍        | 49/334 [00:00<00:02, 114.90batch/s]

Training batches on cuda:  19%|█▊        | 62/334 [00:00<00:02, 118.82batch/s]

Training batches on cuda:  22%|██▏       | 73/334 [00:00<00:02, 109.28batch/s]

Training batches on cuda:  25%|██▌       | 85/334 [00:00<00:02, 109.88batch/s]

Training batches on cuda:  29%|██▉       | 97/334 [00:00<00:02, 110.81batch/s]

Training batches on cuda:  32%|███▏      | 108/334 [00:00<00:02, 108.64batch/s]

Training batches on cuda:  36%|███▌      | 120/334 [00:01<00:01, 109.90batch/s]

Training batches on cuda:  40%|███▉      | 132/334 [00:01<00:01, 110.29batch/s]

Training batches on cuda:  43%|████▎     | 144/334 [00:01<00:01, 110.38batch/s]

Training batches on cuda:  47%|████▋     | 156/334 [00:01<00:01, 111.10batch/s]

Training batches on cuda:  50%|█████     | 168/334 [00:01<00:01, 109.56batch/s]

Training batches on cuda:  54%|█████▍    | 180/334 [00:01<00:01, 110.06batch/s]

Training batches on cuda:  57%|█████▋    | 191/334 [00:01<00:01, 108.53batch/s]

Training batches on cuda:  60%|██████    | 202/334 [00:01<00:01, 108.04batch/s]

Training batches on cuda:  64%|██████▍   | 214/334 [00:01<00:01, 108.88batch/s]

Training batches on cuda:  67%|██████▋   | 225/334 [00:02<00:00, 109.12batch/s]

Training batches on cuda:  71%|███████   | 236/334 [00:02<00:00, 108.59batch/s]

Training batches on cuda:  74%|███████▍  | 248/334 [00:02<00:00, 109.15batch/s]

Training batches on cuda:  78%|███████▊  | 259/334 [00:02<00:00, 109.32batch/s]

Training batches on cuda:  81%|████████  | 270/334 [00:02<00:00, 105.85batch/s]

Training batches on cuda:  84%|████████▍ | 281/334 [00:02<00:00, 106.57batch/s]

Training batches on cuda:  87%|████████▋ | 292/334 [00:02<00:00, 107.40batch/s]

Training batches on cuda:  91%|█████████ | 303/334 [00:02<00:00, 107.61batch/s]

Training batches on cuda:  94%|█████████▍| 314/334 [00:02<00:00, 107.90batch/s]

Training batches on cuda:  97%|█████████▋| 325/334 [00:02<00:00, 108.26batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:33<00:03,  3.31s/epoch, loss=0.00218, prev_loss=0.00218]2021-09-16:17:53:31,783 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:53:32,799 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:53:47,728 INFO     [evaluator.py:657] Evaluation took 14.93s seconds
2021-09-16:17:53:47,859 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:49<00:00,  8.12s/epoch, loss=0.00218, prev_loss=0.00218]
Training epochs on cuda: 100%|██████████| 10/10 [00:49<00:00,  4.95s/epoch, loss=0.00218, prev_loss=0.00218]
2021-09-16:17:53:47,860 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpw1jz0hpv'
2021-09-16:17:53:47,924 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw1jz0hpv' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.53ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.65ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:54:02,710 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 17:54:02,715] Trial 18 finished with value: 0.0020957045788257966 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.7336349959023134, 'optimizer.lr': 0.035254551854115086, 'negative_sampler.num_negs_per_pos': 12, 'training.batch_size': 932}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:54:02,736 WARNING  [api.py:823] No random seed is specified. Setting to 343706307.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 107.82batch/s]

Training batches on cuda:   4%|▎         | 26/720 [00:00<00:05, 117.69batch/s]

Training batches on cuda:   6%|▌         | 42/720 [00:00<00:05, 127.00batch/s]

Training batches on cuda:   8%|▊         | 58/720 [00:00<00:04, 134.44batch/s]

Training batches on cuda:  10%|█         | 74/720 [00:00<00:04, 139.34batch/s]

Training batches on cuda:  12%|█▏        | 89/720 [00:00<00:04, 138.38batch/s]

Training batches on cuda:  14%|█▍        | 104/720 [00:00<00:04, 140.05batch/s]

Training batches on cuda:  17%|█▋        | 120/720 [00:00<00:04, 144.12batch/s]

Training batches on cuda:  19%|█▉        | 138/720 [00:00<00:03, 151.31batch/s]

Training batches on cuda:  22%|██▏       | 156/720 [00:01<00:03, 157.87batch/s]

Training batches on cuda:  24%|██▍       | 175/720 [00:01<00:03, 165.19batch/s]

Training batches on cuda:  27%|██▋       | 194/720 [00:01<00:03, 171.42batch/s]

Training batches on cuda:  30%|██▉       | 214/720 [00:01<00:02, 178.78batch/s]

Training batches on cuda:  32%|███▎      | 234/720 [00:01<00:02, 184.62batch/s]

Training batches on cuda:  35%|███▌      | 253/720 [00:01<00:02, 184.79batch/s]

Training batches on cuda:  38%|███▊      | 272/720 [00:01<00:03, 117.03batch/s]

Training batches on cuda:  40%|████      | 290/720 [00:01<00:03, 130.71batch/s]

Training batches on cuda:  43%|████▎     | 308/720 [00:02<00:02, 142.35batch/s]

Training batches on cuda:  45%|████▌     | 327/720 [00:02<00:02, 152.24batch/s]

Training batches on cuda:  48%|████▊     | 347/720 [00:02<00:02, 162.49batch/s]

Training batches on cuda:  51%|█████     | 366/720 [00:02<00:02, 169.33batch/s]

Training batches on cuda:  53%|█████▎    | 385/720 [00:02<00:01, 173.07batch/s]

Training batches on cuda:  56%|█████▌    | 404/720 [00:02<00:01, 172.23batch/s]

Training batches on cuda:  59%|█████▊    | 422/720 [00:02<00:01, 171.87batch/s]

Training batches on cuda:  61%|██████    | 440/720 [00:02<00:01, 170.07batch/s]

Training batches on cuda:  64%|██████▍   | 459/720 [00:02<00:01, 174.62batch/s]

Training batches on cuda:  66%|██████▋   | 477/720 [00:03<00:01, 169.42batch/s]

Training batches on cuda:  69%|██████▉   | 496/720 [00:03<00:01, 175.02batch/s]

Training batches on cuda:  72%|███████▏  | 515/720 [00:03<00:01, 178.01batch/s]

Training batches on cuda:  74%|███████▍  | 533/720 [00:03<00:01, 173.50batch/s]

Training batches on cuda:  77%|███████▋  | 551/720 [00:03<00:00, 171.11batch/s]

Training batches on cuda:  79%|███████▉  | 569/720 [00:03<00:00, 169.48batch/s]

Training batches on cuda:  82%|████████▏ | 587/720 [00:03<00:00, 168.23batch/s]

Training batches on cuda:  84%|████████▍ | 605/720 [00:03<00:00, 168.83batch/s]

Training batches on cuda:  87%|████████▋ | 624/720 [00:03<00:00, 173.58batch/s]

Training batches on cuda:  89%|████████▉ | 643/720 [00:03<00:00, 176.31batch/s]

Training batches on cuda:  92%|█████████▏| 662/720 [00:04<00:00, 178.07batch/s]

Training batches on cuda:  95%|█████████▍| 681/720 [00:04<00:00, 179.39batch/s]

Training batches on cuda:  97%|█████████▋| 700/720 [00:04<00:00, 179.18batch/s]

Training batches on cuda: 100%|█████████▉| 718/720 [00:04<00:00, 163.07batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.00711, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:41,  4.62s/epoch, loss=0.00711, prev_loss=nan]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 105.31batch/s]

Training batches on cuda:   4%|▍         | 27/720 [00:00<00:05, 116.25batch/s]

Training batches on cuda:   6%|▌         | 44/720 [00:00<00:05, 127.78batch/s]

Training batches on cuda:   8%|▊         | 61/720 [00:00<00:04, 137.79batch/s]

Training batches on cuda:  11%|█         | 79/720 [00:00<00:04, 146.14batch/s]

Training batches on cuda:  13%|█▎        | 97/720 [00:00<00:04, 154.18batch/s]

Training batches on cuda:  16%|█▌        | 116/720 [00:00<00:03, 163.38batch/s]

Training batches on cuda:  19%|█▊        | 134/720 [00:00<00:03, 167.51batch/s]

Training batches on cuda:  21%|██▏       | 153/720 [00:00<00:03, 171.42batch/s]

Training batches on cuda:  24%|██▍       | 172/720 [00:01<00:03, 175.17batch/s]

Training batches on cuda:  27%|██▋       | 192/720 [00:01<00:02, 179.59batch/s]

Training batches on cuda:  29%|██▉       | 210/720 [00:01<00:02, 178.85batch/s]

Training batches on cuda:  32%|███▏      | 228/720 [00:01<00:02, 177.61batch/s]

Training batches on cuda:  34%|███▍      | 246/720 [00:01<00:02, 176.03batch/s]

Training batches on cuda:  37%|███▋      | 264/720 [00:01<00:02, 175.33batch/s]

Training batches on cuda:  39%|███▉      | 282/720 [00:01<00:02, 174.61batch/s]

Training batches on cuda:  42%|████▏     | 300/720 [00:01<00:02, 174.93batch/s]

Training batches on cuda:  44%|████▍     | 318/720 [00:01<00:02, 175.08batch/s]

Training batches on cuda:  47%|████▋     | 336/720 [00:02<00:02, 173.52batch/s]

Training batches on cuda:  49%|████▉     | 354/720 [00:02<00:02, 122.00batch/s]

Training batches on cuda:  51%|█████▏    | 369/720 [00:02<00:02, 127.94batch/s]

Training batches on cuda:  53%|█████▎    | 385/720 [00:02<00:02, 134.60batch/s]

Training batches on cuda:  56%|█████▌    | 400/720 [00:02<00:02, 138.33batch/s]

Training batches on cuda:  58%|█████▊    | 417/720 [00:02<00:02, 144.46batch/s]

Training batches on cuda:  60%|██████    | 433/720 [00:02<00:01, 143.62batch/s]

Training batches on cuda:  62%|██████▎   | 450/720 [00:02<00:01, 149.42batch/s]

Training batches on cuda:  65%|██████▍   | 466/720 [00:02<00:01, 147.04batch/s]

Training batches on cuda:  67%|██████▋   | 481/720 [00:03<00:01, 146.88batch/s]

Training batches on cuda:  69%|██████▉   | 496/720 [00:03<00:01, 145.91batch/s]

Training batches on cuda:  71%|███████▏  | 513/720 [00:03<00:01, 151.21batch/s]

Training batches on cuda:  74%|███████▍  | 531/720 [00:03<00:01, 156.73batch/s]

Training batches on cuda:  76%|███████▋  | 550/720 [00:03<00:01, 163.81batch/s]

Training batches on cuda:  79%|███████▉  | 569/720 [00:03<00:00, 169.43batch/s]

Training batches on cuda:  82%|████████▏ | 588/720 [00:03<00:00, 174.47batch/s]

Training batches on cuda:  84%|████████▍ | 607/720 [00:03<00:00, 177.74batch/s]

Training batches on cuda:  87%|████████▋ | 626/720 [00:03<00:00, 180.17batch/s]

Training batches on cuda:  90%|████████▉ | 645/720 [00:03<00:00, 176.10batch/s]

Training batches on cuda:  92%|█████████▏| 663/720 [00:04<00:00, 172.60batch/s]

Training batches on cuda:  95%|█████████▍| 681/720 [00:04<00:00, 169.26batch/s]

Training batches on cuda:  97%|█████████▋| 698/720 [00:04<00:00, 166.73batch/s]

Training batches on cuda:  99%|█████████▉| 715/720 [00:04<00:00, 162.44batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:09<00:41,  4.62s/epoch, loss=0.00709, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:37,  4.63s/epoch, loss=0.00709, prev_loss=nan]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 14/720 [00:00<00:05, 139.69batch/s]

Training batches on cuda:   5%|▍         | 33/720 [00:00<00:04, 150.81batch/s]

Training batches on cuda:   7%|▋         | 52/720 [00:00<00:04, 159.24batch/s]

Training batches on cuda:  10%|█         | 72/720 [00:00<00:03, 167.98batch/s]

Training batches on cuda:  13%|█▎        | 91/720 [00:00<00:03, 172.44batch/s]

Training batches on cuda:  15%|█▌        | 110/720 [00:00<00:03, 174.94batch/s]

Training batches on cuda:  18%|█▊        | 128/720 [00:00<00:03, 176.13batch/s]

Training batches on cuda:  20%|██        | 147/720 [00:00<00:03, 178.28batch/s]

Training batches on cuda:  23%|██▎       | 166/720 [00:00<00:03, 180.79batch/s]

Training batches on cuda:  26%|██▌       | 184/720 [00:01<00:02, 179.81batch/s]

Training batches on cuda:  28%|██▊       | 204/720 [00:01<00:02, 185.28batch/s]

Training batches on cuda:  31%|███       | 223/720 [00:01<00:02, 186.30batch/s]

Training batches on cuda:  34%|███▎      | 242/720 [00:01<00:02, 186.97batch/s]

Training batches on cuda:  36%|███▋      | 261/720 [00:01<00:02, 183.55batch/s]

Training batches on cuda:  39%|███▉      | 280/720 [00:01<00:02, 176.50batch/s]

Training batches on cuda:  41%|████▏     | 298/720 [00:01<00:02, 168.09batch/s]

Training batches on cuda:  44%|████▍     | 315/720 [00:01<00:02, 165.28batch/s]

Training batches on cuda:  46%|████▌     | 332/720 [00:01<00:02, 160.35batch/s]

Training batches on cuda:  48%|████▊     | 349/720 [00:02<00:02, 154.89batch/s]

Training batches on cuda:  51%|█████     | 365/720 [00:02<00:02, 153.99batch/s]

Training batches on cuda:  53%|█████▎    | 383/720 [00:02<00:02, 160.00batch/s]

Training batches on cuda:  56%|█████▌    | 402/720 [00:02<00:01, 166.76batch/s]

Training batches on cuda:  58%|█████▊    | 419/720 [00:02<00:02, 120.97batch/s]

Training batches on cuda:  61%|██████    | 438/720 [00:02<00:02, 135.19batch/s]

Training batches on cuda:  64%|██████▍   | 459/720 [00:02<00:01, 150.97batch/s]

Training batches on cuda:  67%|██████▋   | 481/720 [00:02<00:01, 164.93batch/s]

Training batches on cuda:  70%|██████▉   | 503/720 [00:02<00:01, 178.19batch/s]

Training batches on cuda:  73%|███████▎  | 525/720 [00:03<00:01, 187.42batch/s]

Training batches on cuda:  76%|███████▌  | 547/720 [00:03<00:00, 194.10batch/s]

Training batches on cuda:  79%|███████▉  | 568/720 [00:03<00:00, 198.41batch/s]

Training batches on cuda:  82%|████████▏ | 591/720 [00:03<00:00, 204.53batch/s]

Training batches on cuda:  85%|████████▌ | 614/720 [00:03<00:00, 209.83batch/s]

Training batches on cuda:  88%|████████▊ | 637/720 [00:03<00:00, 213.62batch/s]

Training batches on cuda:  92%|█████████▏| 659/720 [00:03<00:00, 215.02batch/s]

Training batches on cuda:  95%|█████████▍| 681/720 [00:03<00:00, 214.42batch/s]

Training batches on cuda:  98%|█████████▊| 703/720 [00:03<00:00, 214.25batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:37,  4.63s/epoch, loss=0.00709, prev_loss=0.00709]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:31,  4.48s/epoch, loss=0.00709, prev_loss=0.00709]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 15/720 [00:00<00:04, 146.30batch/s]

Training batches on cuda:   5%|▍         | 35/720 [00:00<00:04, 157.50batch/s]

Training batches on cuda:   7%|▋         | 51/720 [00:00<00:04, 155.22batch/s]

Training batches on cuda:   9%|▉         | 65/720 [00:00<00:04, 150.29batch/s]

Training batches on cuda:  11%|█         | 80/720 [00:00<00:04, 148.37batch/s]

Training batches on cuda:  13%|█▎        | 96/720 [00:00<00:04, 148.79batch/s]

Training batches on cuda:  15%|█▌        | 111/720 [00:00<00:04, 148.97batch/s]

Training batches on cuda:  18%|█▊        | 128/720 [00:00<00:03, 154.54batch/s]

Training batches on cuda:  20%|█▉        | 143/720 [00:00<00:03, 152.75batch/s]

Training batches on cuda:  22%|██▏       | 160/720 [00:01<00:03, 156.58batch/s]

Training batches on cuda:  24%|██▍       | 176/720 [00:01<00:03, 157.03batch/s]

Training batches on cuda:  27%|██▋       | 192/720 [00:01<00:03, 156.78batch/s]

Training batches on cuda:  29%|██▉       | 209/720 [00:01<00:03, 158.55batch/s]

Training batches on cuda:  31%|███▏      | 226/720 [00:01<00:03, 161.44batch/s]

Training batches on cuda:  34%|███▍      | 245/720 [00:01<00:02, 166.96batch/s]

Training batches on cuda:  37%|███▋      | 264/720 [00:01<00:02, 171.75batch/s]

Training batches on cuda:  39%|███▉      | 282/720 [00:01<00:02, 158.71batch/s]

Training batches on cuda:  42%|████▏     | 299/720 [00:01<00:02, 156.85batch/s]

Training batches on cuda:  44%|████▍     | 316/720 [00:01<00:02, 159.22batch/s]

Training batches on cuda:  46%|████▋     | 333/720 [00:02<00:02, 158.91batch/s]

Training batches on cuda:  48%|████▊     | 349/720 [00:02<00:02, 158.63batch/s]

Training batches on cuda:  51%|█████     | 367/720 [00:02<00:02, 163.50batch/s]

Training batches on cuda:  54%|█████▎    | 386/720 [00:02<00:01, 169.79batch/s]

Training batches on cuda:  56%|█████▋    | 405/720 [00:02<00:01, 174.74batch/s]

Training batches on cuda:  59%|█████▉    | 424/720 [00:02<00:01, 176.78batch/s]

Training batches on cuda:  62%|██████▏   | 443/720 [00:02<00:01, 178.53batch/s]

Training batches on cuda:  64%|██████▍   | 463/720 [00:02<00:01, 181.82batch/s]

Training batches on cuda:  67%|██████▋   | 482/720 [00:02<00:01, 183.78batch/s]

Training batches on cuda:  70%|██████▉   | 501/720 [00:03<00:01, 184.91batch/s]

Training batches on cuda:  72%|███████▏  | 520/720 [00:03<00:01, 186.20batch/s]

Training batches on cuda:  75%|███████▍  | 539/720 [00:03<00:00, 185.36batch/s]

Training batches on cuda:  78%|███████▊  | 558/720 [00:03<00:00, 167.48batch/s]

Training batches on cuda:  80%|████████  | 576/720 [00:03<00:00, 164.69batch/s]

Training batches on cuda:  82%|████████▏ | 593/720 [00:03<00:00, 163.31batch/s]

Training batches on cuda:  85%|████████▍ | 610/720 [00:03<00:00, 164.46batch/s]

Training batches on cuda:  87%|████████▋ | 627/720 [00:03<00:00, 163.48batch/s]

Training batches on cuda:  90%|████████▉ | 647/720 [00:03<00:00, 171.28batch/s]

Training batches on cuda:  93%|█████████▎| 667/720 [00:03<00:00, 177.96batch/s]

Training batches on cuda:  95%|█████████▌| 686/720 [00:04<00:00, 181.17batch/s]

Training batches on cuda:  98%|█████████▊| 705/720 [00:04<00:00, 183.25batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:31,  4.48s/epoch, loss=0.00709, prev_loss=0.00709]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:26,  4.49s/epoch, loss=0.00709, prev_loss=0.00709]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 13/720 [00:00<00:05, 124.28batch/s]

Training batches on cuda:   4%|▍         | 31/720 [00:00<00:05, 137.00batch/s]

Training batches on cuda:   7%|▋         | 50/720 [00:00<00:04, 148.78batch/s]

Training batches on cuda:  10%|▉         | 69/720 [00:00<00:04, 158.62batch/s]

Training batches on cuda:  12%|█▏        | 85/720 [00:00<00:03, 159.03batch/s]

Training batches on cuda:  14%|█▍        | 103/720 [00:00<00:03, 164.14batch/s]

Training batches on cuda:  17%|█▋        | 120/720 [00:00<00:03, 163.88batch/s]

Training batches on cuda:  19%|█▉        | 140/720 [00:00<00:03, 171.94batch/s]

Training batches on cuda:  22%|██▏       | 159/720 [00:00<00:03, 176.81batch/s]

Training batches on cuda:  25%|██▍       | 179/720 [00:01<00:02, 181.27batch/s]

Training batches on cuda:  28%|██▊       | 199/720 [00:01<00:02, 184.96batch/s]

Training batches on cuda:  30%|███       | 219/720 [00:01<00:02, 188.09batch/s]

Training batches on cuda:  33%|███▎      | 239/720 [00:01<00:02, 189.33batch/s]

Training batches on cuda:  36%|███▌      | 259/720 [00:01<00:02, 190.86batch/s]

Training batches on cuda:  39%|███▉      | 279/720 [00:01<00:02, 192.50batch/s]

Training batches on cuda:  42%|████▏     | 299/720 [00:01<00:02, 191.97batch/s]

Training batches on cuda:  44%|████▍     | 319/720 [00:01<00:02, 193.84batch/s]

Training batches on cuda:  47%|████▋     | 339/720 [00:01<00:01, 193.83batch/s]

Training batches on cuda:  50%|████▉     | 359/720 [00:01<00:01, 194.42batch/s]

Training batches on cuda:  53%|█████▎    | 380/720 [00:02<00:01, 196.84batch/s]

Training batches on cuda:  56%|█████▌    | 400/720 [00:02<00:01, 195.94batch/s]

Training batches on cuda:  58%|█████▊    | 420/720 [00:02<00:01, 194.61batch/s]

Training batches on cuda:  61%|██████    | 440/720 [00:02<00:01, 192.91batch/s]

Training batches on cuda:  64%|██████▍   | 460/720 [00:02<00:01, 192.69batch/s]

Training batches on cuda:  67%|██████▋   | 480/720 [00:02<00:01, 192.93batch/s]

Training batches on cuda:  69%|██████▉   | 500/720 [00:02<00:01, 194.98batch/s]

Training batches on cuda:  72%|███████▏  | 520/720 [00:02<00:01, 196.43batch/s]

Training batches on cuda:  75%|███████▌  | 541/720 [00:02<00:00, 197.98batch/s]

Training batches on cuda:  78%|███████▊  | 561/720 [00:02<00:00, 190.51batch/s]

Training batches on cuda:  81%|████████  | 581/720 [00:03<00:00, 182.55batch/s]

Training batches on cuda:  83%|████████▎ | 600/720 [00:03<00:00, 177.12batch/s]

Training batches on cuda:  86%|████████▌ | 618/720 [00:03<00:00, 172.90batch/s]

Training batches on cuda:  88%|████████▊ | 636/720 [00:03<00:00, 174.56batch/s]

Training batches on cuda:  91%|█████████ | 656/720 [00:03<00:00, 181.03batch/s]

Training batches on cuda:  94%|█████████▍| 675/720 [00:03<00:00, 162.91batch/s]

Training batches on cuda:  96%|█████████▌| 692/720 [00:03<00:00, 161.05batch/s]

Training batches on cuda:  98%|█████████▊| 709/720 [00:03<00:00, 135.95batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:26,  4.49s/epoch, loss=0.00709, prev_loss=0.00709]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:22,  4.41s/epoch, loss=0.00709, prev_loss=0.00709]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 14/720 [00:00<00:05, 136.13batch/s]

Training batches on cuda:   4%|▍         | 32/720 [00:00<00:04, 144.98batch/s]

Training batches on cuda:   7%|▋         | 50/720 [00:00<00:04, 152.64batch/s]

Training batches on cuda:   9%|▉         | 67/720 [00:00<00:04, 157.15batch/s]

Training batches on cuda:  12%|█▏        | 83/720 [00:00<00:04, 157.13batch/s]

Training batches on cuda:  14%|█▍        | 99/720 [00:00<00:03, 157.95batch/s]

Training batches on cuda:  16%|█▌        | 116/720 [00:00<00:03, 159.13batch/s]

Training batches on cuda:  19%|█▊        | 134/720 [00:00<00:03, 164.48batch/s]

Training batches on cuda:  21%|██▏       | 153/720 [00:00<00:03, 170.86batch/s]

Training batches on cuda:  24%|██▍       | 172/720 [00:01<00:03, 175.62batch/s]

Training batches on cuda:  27%|██▋       | 191/720 [00:01<00:03, 175.46batch/s]

Training batches on cuda:  29%|██▉       | 209/720 [00:01<00:03, 168.87batch/s]

Training batches on cuda:  31%|███▏      | 226/720 [00:01<00:02, 168.32batch/s]

Training batches on cuda:  34%|███▍      | 243/720 [00:01<00:02, 167.97batch/s]

Training batches on cuda:  37%|███▋      | 263/720 [00:01<00:02, 175.73batch/s]

Training batches on cuda:  39%|███▉      | 284/720 [00:01<00:02, 183.79batch/s]

Training batches on cuda:  42%|████▏     | 305/720 [00:01<00:02, 188.97batch/s]

Training batches on cuda:  45%|████▌     | 325/720 [00:01<00:02, 186.15batch/s]

Training batches on cuda:  48%|████▊     | 344/720 [00:01<00:02, 179.96batch/s]

Training batches on cuda:  50%|█████     | 363/720 [00:02<00:02, 176.20batch/s]

Training batches on cuda:  53%|█████▎    | 383/720 [00:02<00:01, 181.23batch/s]

Training batches on cuda:  56%|█████▌    | 402/720 [00:02<00:01, 180.11batch/s]

Training batches on cuda:  58%|█████▊    | 421/720 [00:02<00:01, 180.96batch/s]

Training batches on cuda:  61%|██████    | 440/720 [00:02<00:01, 178.72batch/s]

Training batches on cuda:  64%|██████▍   | 460/720 [00:02<00:01, 182.26batch/s]

Training batches on cuda:  67%|██████▋   | 479/720 [00:02<00:01, 180.40batch/s]

Training batches on cuda:  69%|██████▉   | 498/720 [00:02<00:01, 182.47batch/s]

Training batches on cuda:  72%|███████▏  | 517/720 [00:02<00:01, 184.51batch/s]

Training batches on cuda:  75%|███████▍  | 538/720 [00:03<00:00, 190.01batch/s]

Training batches on cuda:  78%|███████▊  | 558/720 [00:03<00:00, 192.53batch/s]

Training batches on cuda:  80%|████████  | 579/720 [00:03<00:00, 195.09batch/s]

Training batches on cuda:  84%|████████▎ | 602/720 [00:03<00:00, 203.60batch/s]

Training batches on cuda:  87%|████████▋ | 624/720 [00:03<00:00, 206.19batch/s]

Training batches on cuda:  90%|████████▉ | 645/720 [00:03<00:00, 199.78batch/s]

Training batches on cuda:  92%|█████████▎| 666/720 [00:03<00:00, 200.75batch/s]

Training batches on cuda:  95%|█████████▌| 687/720 [00:03<00:00, 202.14batch/s]

Training batches on cuda:  98%|█████████▊| 708/720 [00:03<00:00, 201.19batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:26<00:22,  4.41s/epoch, loss=0.00709, prev_loss=0.00709]
Training epochs on cuda:  60%|██████    | 6/10 [00:26<00:17,  4.32s/epoch, loss=0.00709, prev_loss=0.00709]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 106.95batch/s]

Training batches on cuda:   4%|▎         | 26/720 [00:00<00:06, 115.57batch/s]

Training batches on cuda:   6%|▌         | 43/720 [00:00<00:05, 126.90batch/s]

Training batches on cuda:   8%|▊         | 60/720 [00:00<00:04, 136.74batch/s]

Training batches on cuda:  10%|█         | 74/720 [00:00<00:04, 135.46batch/s]

Training batches on cuda:  13%|█▎        | 91/720 [00:00<00:04, 143.01batch/s]

Training batches on cuda:  15%|█▌        | 108/720 [00:00<00:04, 149.27batch/s]

Training batches on cuda:  17%|█▋        | 125/720 [00:00<00:03, 152.93batch/s]

Training batches on cuda:  20%|█▉        | 142/720 [00:00<00:03, 156.65batch/s]

Training batches on cuda:  22%|██▏       | 159/720 [00:01<00:03, 159.98batch/s]

Training batches on cuda:  25%|██▍       | 179/720 [00:01<00:03, 169.48batch/s]

Training batches on cuda:  28%|██▊       | 199/720 [00:01<00:02, 175.90batch/s]

Training batches on cuda:  30%|███       | 219/720 [00:01<00:02, 181.24batch/s]

Training batches on cuda:  33%|███▎      | 239/720 [00:01<00:02, 183.95batch/s]

Training batches on cuda:  36%|███▌      | 259/720 [00:01<00:02, 187.18batch/s]

Training batches on cuda:  39%|███▉      | 279/720 [00:01<00:02, 189.42batch/s]

Training batches on cuda:  42%|████▏     | 299/720 [00:01<00:02, 190.54batch/s]

Training batches on cuda:  44%|████▍     | 319/720 [00:01<00:02, 182.08batch/s]

Training batches on cuda:  47%|████▋     | 338/720 [00:02<00:02, 169.78batch/s]

Training batches on cuda:  49%|████▉     | 356/720 [00:02<00:02, 168.87batch/s]

Training batches on cuda:  52%|█████▏    | 374/720 [00:02<00:02, 170.39batch/s]

Training batches on cuda:  54%|█████▍    | 392/720 [00:02<00:01, 172.46batch/s]

Training batches on cuda:  57%|█████▋    | 412/720 [00:02<00:01, 178.31batch/s]

Training batches on cuda:  60%|██████    | 432/720 [00:02<00:01, 182.01batch/s]

Training batches on cuda:  63%|██████▎   | 452/720 [00:02<00:01, 185.82batch/s]

Training batches on cuda:  66%|██████▌   | 472/720 [00:02<00:01, 189.49batch/s]

Training batches on cuda:  69%|██████▊   | 494/720 [00:02<00:01, 196.08batch/s]

Training batches on cuda:  72%|███████▏  | 515/720 [00:02<00:01, 199.64batch/s]

Training batches on cuda:  74%|███████▍  | 536/720 [00:03<00:00, 201.17batch/s]

Training batches on cuda:  78%|███████▊  | 558/720 [00:03<00:00, 204.42batch/s]

Training batches on cuda:  81%|████████  | 580/720 [00:03<00:00, 206.76batch/s]

Training batches on cuda:  83%|████████▎ | 601/720 [00:03<00:00, 205.02batch/s]

Training batches on cuda:  86%|████████▋ | 622/720 [00:03<00:00, 201.82batch/s]

Training batches on cuda:  89%|████████▉ | 644/720 [00:03<00:00, 204.34batch/s]

Training batches on cuda:  92%|█████████▏| 665/720 [00:03<00:00, 198.85batch/s]

Training batches on cuda:  95%|█████████▌| 686/720 [00:03<00:00, 199.62batch/s]

Training batches on cuda:  98%|█████████▊| 707/720 [00:03<00:00, 197.35batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:17,  4.32s/epoch, loss=0.00709, prev_loss=0.00709]
Training epochs on cuda:  70%|███████   | 7/10 [00:30<00:12,  4.27s/epoch, loss=0.00709, prev_loss=0.00709]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 102.23batch/s]

Training batches on cuda:   4%|▍         | 30/720 [00:00<00:05, 118.42batch/s]

Training batches on cuda:   7%|▋         | 48/720 [00:00<00:05, 131.42batch/s]

Training batches on cuda:  10%|▉         | 69/720 [00:00<00:04, 146.58batch/s]

Training batches on cuda:  12%|█▏        | 88/720 [00:00<00:04, 156.98batch/s]

Training batches on cuda:  15%|█▌        | 108/720 [00:00<00:03, 166.74batch/s]

Training batches on cuda:  17%|█▋        | 125/720 [00:00<00:03, 166.47batch/s]

Training batches on cuda:  20%|██        | 145/720 [00:00<00:03, 173.26batch/s]

Training batches on cuda:  23%|██▎       | 164/720 [00:00<00:03, 176.33batch/s]

Training batches on cuda:  25%|██▌       | 183/720 [00:01<00:03, 178.48batch/s]

Training batches on cuda:  28%|██▊       | 202/720 [00:01<00:02, 180.85batch/s]

Training batches on cuda:  31%|███       | 221/720 [00:01<00:02, 181.59batch/s]

Training batches on cuda:  33%|███▎      | 240/720 [00:01<00:02, 183.23batch/s]

Training batches on cuda:  36%|███▌      | 260/720 [00:01<00:02, 185.18batch/s]

Training batches on cuda:  39%|███▉      | 279/720 [00:01<00:02, 183.23batch/s]

Training batches on cuda:  41%|████▏     | 298/720 [00:01<00:02, 180.90batch/s]

Training batches on cuda:  44%|████▍     | 317/720 [00:01<00:02, 180.83batch/s]

Training batches on cuda:  47%|████▋     | 337/720 [00:01<00:02, 184.19batch/s]

Training batches on cuda:  49%|████▉     | 356/720 [00:01<00:01, 183.90batch/s]

Training batches on cuda:  52%|█████▏    | 375/720 [00:02<00:01, 184.03batch/s]

Training batches on cuda:  55%|█████▍    | 394/720 [00:02<00:01, 178.70batch/s]

Training batches on cuda:  57%|█████▋    | 412/720 [00:02<00:01, 171.57batch/s]

Training batches on cuda:  60%|█████▉    | 430/720 [00:02<00:01, 164.04batch/s]

Training batches on cuda:  62%|██████▏   | 447/720 [00:02<00:01, 160.89batch/s]

Training batches on cuda:  64%|██████▍   | 464/720 [00:02<00:01, 160.87batch/s]

Training batches on cuda:  67%|██████▋   | 481/720 [00:02<00:01, 161.55batch/s]

Training batches on cuda:  69%|██████▉   | 499/720 [00:02<00:01, 164.22batch/s]

Training batches on cuda:  72%|███████▏  | 518/720 [00:02<00:01, 168.98batch/s]

Training batches on cuda:  75%|███████▍  | 537/720 [00:03<00:01, 171.84batch/s]

Training batches on cuda:  77%|███████▋  | 556/720 [00:03<00:00, 174.60batch/s]

Training batches on cuda:  80%|███████▉  | 574/720 [00:03<00:00, 175.05batch/s]

Training batches on cuda:  82%|████████▎ | 594/720 [00:03<00:00, 181.47batch/s]

Training batches on cuda:  85%|████████▌ | 613/720 [00:03<00:00, 181.76batch/s]

Training batches on cuda:  88%|████████▊ | 632/720 [00:03<00:00, 183.44batch/s]

Training batches on cuda:  90%|█████████ | 651/720 [00:03<00:00, 181.65batch/s]

Training batches on cuda:  93%|█████████▎| 670/720 [00:03<00:00, 182.85batch/s]

Training batches on cuda:  96%|█████████▌| 690/720 [00:03<00:00, 185.22batch/s]

Training batches on cuda:  98%|█████████▊| 709/720 [00:03<00:00, 186.63batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:34<00:12,  4.27s/epoch, loss=0.00709, prev_loss=0.00709]
Training epochs on cuda:  80%|████████  | 8/10 [00:34<00:08,  4.28s/epoch, loss=0.00709, prev_loss=0.00709]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 12/720 [00:00<00:06, 115.99batch/s]

Training batches on cuda:   4%|▍         | 30/720 [00:00<00:05, 129.51batch/s]

Training batches on cuda:   7%|▋         | 49/720 [00:00<00:04, 142.58batch/s]

Training batches on cuda:   9%|▉         | 68/720 [00:00<00:04, 153.54batch/s]

Training batches on cuda:  12%|█▏        | 86/720 [00:00<00:03, 158.56batch/s]

Training batches on cuda:  14%|█▍        | 103/720 [00:00<00:03, 161.14batch/s]

Training batches on cuda:  17%|█▋        | 121/720 [00:00<00:03, 163.76batch/s]

Training batches on cuda:  19%|█▉        | 138/720 [00:00<00:03, 164.11batch/s]

Training batches on cuda:  21%|██▏       | 154/720 [00:00<00:03, 162.36batch/s]

Training batches on cuda:  24%|██▍       | 171/720 [00:01<00:03, 163.66batch/s]

Training batches on cuda:  26%|██▌       | 188/720 [00:01<00:03, 165.39batch/s]

Training batches on cuda:  29%|██▊       | 206/720 [00:01<00:03, 168.87batch/s]

Training batches on cuda:  31%|███       | 224/720 [00:01<00:02, 170.50batch/s]

Training batches on cuda:  34%|███▎      | 242/720 [00:01<00:02, 172.29batch/s]

Training batches on cuda:  37%|███▋      | 263/720 [00:01<00:02, 180.59batch/s]

Training batches on cuda:  39%|███▉      | 282/720 [00:01<00:02, 178.64batch/s]

Training batches on cuda:  42%|████▏     | 300/720 [00:01<00:02, 178.12batch/s]

Training batches on cuda:  44%|████▍     | 319/720 [00:01<00:02, 179.23batch/s]

Training batches on cuda:  47%|████▋     | 339/720 [00:01<00:02, 182.86batch/s]

Training batches on cuda:  50%|████▉     | 359/720 [00:02<00:01, 185.77batch/s]

Training batches on cuda:  53%|█████▎    | 379/720 [00:02<00:01, 188.32batch/s]

Training batches on cuda:  55%|█████▌    | 399/720 [00:02<00:01, 190.00batch/s]

Training batches on cuda:  58%|█████▊    | 419/720 [00:02<00:01, 191.60batch/s]

Training batches on cuda:  61%|██████    | 439/720 [00:02<00:01, 190.17batch/s]

Training batches on cuda:  64%|██████▍   | 459/720 [00:02<00:01, 190.44batch/s]

Training batches on cuda:  67%|██████▋   | 480/720 [00:02<00:01, 193.62batch/s]

Training batches on cuda:  69%|██████▉   | 500/720 [00:02<00:01, 189.31batch/s]

Training batches on cuda:  72%|███████▏  | 519/720 [00:02<00:01, 186.34batch/s]

Training batches on cuda:  75%|███████▍  | 539/720 [00:03<00:00, 188.88batch/s]

Training batches on cuda:  78%|███████▊  | 558/720 [00:03<00:00, 187.82batch/s]

Training batches on cuda:  80%|████████  | 577/720 [00:03<00:00, 184.86batch/s]

Training batches on cuda:  83%|████████▎ | 596/720 [00:03<00:00, 183.88batch/s]

Training batches on cuda:  85%|████████▌ | 615/720 [00:03<00:00, 175.94batch/s]

Training batches on cuda:  88%|████████▊ | 633/720 [00:03<00:00, 169.21batch/s]

Training batches on cuda:  90%|█████████ | 651/720 [00:03<00:00, 158.88batch/s]

Training batches on cuda:  93%|█████████▎| 668/720 [00:03<00:00, 152.65batch/s]

Training batches on cuda:  95%|█████████▌| 684/720 [00:03<00:00, 149.27batch/s]

Training batches on cuda:  97%|█████████▋| 700/720 [00:04<00:00, 146.73batch/s]

Training batches on cuda: 100%|█████████▉| 717/720 [00:04<00:00, 151.05batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:39<00:08,  4.28s/epoch, loss=0.00709, prev_loss=0.00709]
Training epochs on cuda:  90%|█████████ | 9/10 [00:39<00:04,  4.30s/epoch, loss=0.00709, prev_loss=0.00709]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 106.99batch/s]

Training batches on cuda:   4%|▍         | 29/720 [00:00<00:05, 121.65batch/s]

Training batches on cuda:   7%|▋         | 47/720 [00:00<00:05, 133.35batch/s]

Training batches on cuda:   9%|▉         | 64/720 [00:00<00:04, 141.53batch/s]

Training batches on cuda:  12%|█▏        | 83/720 [00:00<00:04, 150.50batch/s]

Training batches on cuda:  14%|█▍        | 100/720 [00:00<00:04, 154.87batch/s]

Training batches on cuda:  16%|█▌        | 115/720 [00:00<00:03, 152.12batch/s]

Training batches on cuda:  18%|█▊        | 132/720 [00:00<00:03, 156.31batch/s]

Training batches on cuda:  21%|██        | 148/720 [00:00<00:03, 155.73batch/s]

Training batches on cuda:  23%|██▎       | 164/720 [00:01<00:03, 145.44batch/s]

Training batches on cuda:  25%|██▌       | 180/720 [00:01<00:03, 147.69batch/s]

Training batches on cuda:  27%|██▋       | 196/720 [00:01<00:03, 150.56batch/s]

Training batches on cuda:  30%|██▉       | 213/720 [00:01<00:03, 153.53batch/s]

Training batches on cuda:  32%|███▏      | 230/720 [00:01<00:03, 156.83batch/s]

Training batches on cuda:  34%|███▍      | 247/720 [00:01<00:02, 159.37batch/s]

Training batches on cuda:  37%|███▋      | 263/720 [00:01<00:02, 155.43batch/s]

Training batches on cuda:  39%|███▉      | 279/720 [00:01<00:02, 152.82batch/s]

Training batches on cuda:  41%|████      | 295/720 [00:01<00:02, 150.85batch/s]

Training batches on cuda:  43%|████▎     | 312/720 [00:02<00:02, 154.92batch/s]

Training batches on cuda:  46%|████▌     | 328/720 [00:02<00:02, 155.55batch/s]

Training batches on cuda:  48%|████▊     | 344/720 [00:02<00:02, 155.97batch/s]

Training batches on cuda:  50%|█████     | 361/720 [00:02<00:02, 158.05batch/s]

Training batches on cuda:  52%|█████▏    | 377/720 [00:02<00:02, 157.41batch/s]

Training batches on cuda:  55%|█████▍    | 393/720 [00:02<00:02, 155.73batch/s]

Training batches on cuda:  57%|█████▋    | 409/720 [00:02<00:02, 149.51batch/s]

Training batches on cuda:  59%|█████▉    | 428/720 [00:02<00:01, 159.08batch/s]

Training batches on cuda:  62%|██████▏   | 445/720 [00:02<00:01, 149.60batch/s]

Training batches on cuda:  64%|██████▍   | 461/720 [00:02<00:01, 143.95batch/s]

Training batches on cuda:  66%|██████▌   | 476/720 [00:03<00:01, 143.69batch/s]

Training batches on cuda:  68%|██████▊   | 493/720 [00:03<00:01, 150.36batch/s]

Training batches on cuda:  71%|███████   | 510/720 [00:03<00:01, 154.00batch/s]

Training batches on cuda:  73%|███████▎  | 528/720 [00:03<00:01, 158.94batch/s]

Training batches on cuda:  76%|███████▌  | 547/720 [00:03<00:01, 166.51batch/s]

Training batches on cuda:  79%|███████▊  | 566/720 [00:03<00:00, 171.06batch/s]

Training batches on cuda:  81%|████████▏ | 585/720 [00:03<00:00, 175.47batch/s]

Training batches on cuda:  84%|████████▍ | 605/720 [00:03<00:00, 181.31batch/s]

Training batches on cuda:  87%|████████▋ | 624/720 [00:03<00:00, 181.53batch/s]

Training batches on cuda:  89%|████████▉ | 643/720 [00:04<00:00, 182.12batch/s]

Training batches on cuda:  92%|█████████▏| 662/720 [00:04<00:00, 172.00batch/s]

Training batches on cuda:  94%|█████████▍| 680/720 [00:04<00:00, 165.21batch/s]

Training batches on cuda:  97%|█████████▋| 698/720 [00:04<00:00, 169.08batch/s]

Training batches on cuda: 100%|█████████▉| 717/720 [00:04<00:00, 173.64batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:43<00:04,  4.30s/epoch, loss=0.00709, prev_loss=0.00709]2021-09-16:17:54:46,938 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:54:47,989 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:55:02,925 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:17:55:03,60 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00,  9.27s/epoch, loss=0.00709, prev_loss=0.00709]
Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00,  5.99s/epoch, loss=0.00709, prev_loss=0.00709]
2021-09-16:17:55:03,61 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpeew_qx4l'
2021-09-16:17:55:03,122 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpeew_qx4l' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.34ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.62ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.61ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.62ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:55:17,937 INFO     [evaluator.py:657] Evaluation took 14.79s seconds
[I 2021-09-16 17:55:17,943] Trial 19 finished with value: 0.0004697554022580208 and parameters: {'loss.margin': 3, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.4723780534835031, 'optimizer.lr': 0.016251067186108192, 'negative_sampler.num_negs_per_pos': 5, 'training.batch_size': 432}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:55:17,963 WARNING  [api.py:823] No random seed is specified. Setting to 794288307.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 56.02batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:02, 63.94batch/s]

Training batches on cuda:  16%|█▌        | 26/161 [00:00<00:01, 70.88batch/s]

Training batches on cuda:  22%|██▏       | 36/161 [00:00<00:01, 76.78batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:01, 81.51batch/s]

Training batches on cuda:  35%|███▍      | 56/161 [00:00<00:01, 85.87batch/s]

Training batches on cuda:  41%|████      | 66/161 [00:00<00:01, 88.26batch/s]

Training batches on cuda:  47%|████▋     | 76/161 [00:00<00:00, 90.32batch/s]

Training batches on cuda:  53%|█████▎    | 86/161 [00:00<00:00, 91.56batch/s]

Training batches on cuda:  60%|█████▉    | 96/161 [00:01<00:00, 92.25batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 93.23batch/s]

Training batches on cuda:  72%|███████▏  | 116/161 [00:01<00:00, 93.99batch/s]

Training batches on cuda:  78%|███████▊  | 126/161 [00:01<00:00, 88.20batch/s]

Training batches on cuda:  84%|████████▍ | 135/161 [00:01<00:00, 82.26batch/s]

Training batches on cuda:  90%|█████████ | 145/161 [00:01<00:00, 84.98batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:01<00:00, 87.48batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:01<?, ?epoch/s, loss=0.00112, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:01<00:17,  1.98s/epoch, loss=0.00112, prev_loss=nan]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 59.01batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 64.98batch/s]

Training batches on cuda:  14%|█▎        | 22/161 [00:00<00:02, 65.21batch/s]

Training batches on cuda:  19%|█▊        | 30/161 [00:00<00:01, 68.75batch/s]

Training batches on cuda:  24%|██▎       | 38/161 [00:00<00:01, 71.03batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:01, 73.14batch/s]

Training batches on cuda:  34%|███▍      | 55/161 [00:00<00:01, 75.12batch/s]

Training batches on cuda:  40%|███▉      | 64/161 [00:00<00:01, 77.31batch/s]

Training batches on cuda:  46%|████▌     | 74/161 [00:00<00:01, 81.15batch/s]

Training batches on cuda:  52%|█████▏    | 84/161 [00:01<00:00, 84.54batch/s]

Training batches on cuda:  58%|█████▊    | 94/161 [00:01<00:00, 87.51batch/s]

Training batches on cuda:  65%|██████▍   | 104/161 [00:01<00:00, 86.73batch/s]

Training batches on cuda:  70%|███████   | 113/161 [00:01<00:00, 83.73batch/s]

Training batches on cuda:  76%|███████▌  | 122/161 [00:01<00:00, 81.79batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:01<00:00, 79.93batch/s]

Training batches on cuda:  87%|████████▋ | 140/161 [00:01<00:00, 78.75batch/s]

Training batches on cuda:  93%|█████████▎| 149/161 [00:01<00:00, 78.04batch/s]

Training batches on cuda:  99%|█████████▉| 159/161 [00:01<00:00, 81.67batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:17,  1.98s/epoch, loss=0.00111, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:16,  2.05s/epoch, loss=0.00111, prev_loss=nan]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 59.26batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:02, 66.13batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:01, 71.23batch/s]

Training batches on cuda:  22%|██▏       | 35/161 [00:00<00:01, 76.89batch/s]

Training batches on cuda:  27%|██▋       | 44/161 [00:00<00:01, 79.14batch/s]

Training batches on cuda:  34%|███▎      | 54/161 [00:00<00:01, 82.80batch/s]

Training batches on cuda:  39%|███▉      | 63/161 [00:00<00:01, 84.64batch/s]

Training batches on cuda:  45%|████▍     | 72/161 [00:00<00:01, 85.48batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:00<00:00, 85.51batch/s]

Training batches on cuda:  56%|█████▌    | 90/161 [00:01<00:00, 86.60batch/s]

Training batches on cuda:  61%|██████▏   | 99/161 [00:01<00:00, 85.67batch/s]

Training batches on cuda:  67%|██████▋   | 108/161 [00:01<00:00, 81.00batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:01<00:00, 77.48batch/s]

Training batches on cuda:  78%|███████▊  | 125/161 [00:01<00:00, 77.95batch/s]

Training batches on cuda:  83%|████████▎ | 133/161 [00:01<00:00, 76.19batch/s]

Training batches on cuda:  88%|████████▊ | 141/161 [00:01<00:00, 75.96batch/s]

Training batches on cuda:  93%|█████████▎| 149/161 [00:01<00:00, 76.59batch/s]

Training batches on cuda:  98%|█████████▊| 157/161 [00:01<00:00, 73.30batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:16,  2.05s/epoch, loss=0.00111, prev_loss=0.00111]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:14,  2.10s/epoch, loss=0.00111, prev_loss=0.00111]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 59.18batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 64.52batch/s]

Training batches on cuda:  15%|█▍        | 24/161 [00:00<00:01, 69.15batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:01, 73.09batch/s]

Training batches on cuda:  26%|██▌       | 42/161 [00:00<00:01, 75.89batch/s]

Training batches on cuda:  30%|███       | 49/161 [00:00<00:01, 66.48batch/s]

Training batches on cuda:  36%|███▌      | 58/161 [00:00<00:01, 70.86batch/s]

Training batches on cuda:  42%|████▏     | 67/161 [00:00<00:01, 73.94batch/s]

Training batches on cuda:  48%|████▊     | 77/161 [00:00<00:01, 78.19batch/s]

Training batches on cuda:  53%|█████▎    | 86/161 [00:01<00:00, 79.85batch/s]

Training batches on cuda:  59%|█████▉    | 95/161 [00:01<00:00, 82.28batch/s]

Training batches on cuda:  65%|██████▌   | 105/161 [00:01<00:00, 85.25batch/s]

Training batches on cuda:  71%|███████▏  | 115/161 [00:01<00:00, 88.28batch/s]

Training batches on cuda:  78%|███████▊  | 125/161 [00:01<00:00, 90.52batch/s]

Training batches on cuda:  84%|████████▍ | 135/161 [00:01<00:00, 92.23batch/s]

Training batches on cuda:  90%|█████████ | 145/161 [00:01<00:00, 92.94batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:01<00:00, 93.86batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:14,  2.10s/epoch, loss=0.00111, prev_loss=0.00111]
Training epochs on cuda:  40%|████      | 4/10 [00:08<00:12,  2.11s/epoch, loss=0.00111, prev_loss=0.00111]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 45.94batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 54.12batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:02, 62.33batch/s]

Training batches on cuda:  21%|██        | 34/161 [00:00<00:01, 67.62batch/s]

Training batches on cuda:  26%|██▌       | 42/161 [00:00<00:01, 68.63batch/s]

Training batches on cuda:  30%|███       | 49/161 [00:00<00:01, 68.92batch/s]

Training batches on cuda:  35%|███▍      | 56/161 [00:00<00:01, 68.01batch/s]

Training batches on cuda:  40%|███▉      | 64/161 [00:00<00:01, 70.32batch/s]

Training batches on cuda:  45%|████▌     | 73/161 [00:00<00:01, 75.01batch/s]

Training batches on cuda:  52%|█████▏    | 84/161 [00:01<00:00, 81.66batch/s]

Training batches on cuda:  59%|█████▉    | 95/161 [00:01<00:00, 87.22batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 91.97batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:01<00:00, 95.08batch/s]

Training batches on cuda:  80%|███████▉  | 128/161 [00:01<00:00, 97.71batch/s]

Training batches on cuda:  86%|████████▋ | 139/161 [00:01<00:00, 99.27batch/s]

Training batches on cuda:  93%|█████████▎| 150/161 [00:01<00:00, 100.96batch/s]

Training batches on cuda: 100%|██████████| 161/161 [00:01<00:00, 95.96batch/s] 

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:10<00:12,  2.11s/epoch, loss=0.00111, prev_loss=0.00111]
Training epochs on cuda:  50%|█████     | 5/10 [00:10<00:10,  2.07s/epoch, loss=0.00111, prev_loss=0.00111]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 56.45batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 62.33batch/s]

Training batches on cuda:  15%|█▍        | 24/161 [00:00<00:02, 67.79batch/s]

Training batches on cuda:  21%|██        | 34/161 [00:00<00:01, 73.61batch/s]

Training batches on cuda:  27%|██▋       | 44/161 [00:00<00:01, 78.63batch/s]

Training batches on cuda:  32%|███▏      | 52/161 [00:00<00:01, 76.53batch/s]

Training batches on cuda:  37%|███▋      | 60/161 [00:00<00:01, 75.71batch/s]

Training batches on cuda:  43%|████▎     | 69/161 [00:00<00:01, 78.94batch/s]

Training batches on cuda:  48%|████▊     | 77/161 [00:00<00:01, 76.68batch/s]

Training batches on cuda:  53%|█████▎    | 85/161 [00:01<00:01, 73.08batch/s]

Training batches on cuda:  58%|█████▊    | 94/161 [00:01<00:00, 77.05batch/s]

Training batches on cuda:  64%|██████▍   | 103/161 [00:01<00:00, 78.84batch/s]

Training batches on cuda:  71%|███████   | 114/161 [00:01<00:00, 84.32batch/s]

Training batches on cuda:  78%|███████▊  | 125/161 [00:01<00:00, 88.59batch/s]

Training batches on cuda:  84%|████████▍ | 135/161 [00:01<00:00, 84.01batch/s]

Training batches on cuda:  90%|█████████ | 145/161 [00:01<00:00, 87.49batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:01<00:00, 88.71batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:12<00:10,  2.07s/epoch, loss=0.00111, prev_loss=0.00111]
Training epochs on cuda:  60%|██████    | 6/10 [00:12<00:08,  2.09s/epoch, loss=0.00111, prev_loss=0.00111]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 53.47batch/s]

Training batches on cuda:   9%|▊         | 14/161 [00:00<00:02, 59.05batch/s]

Training batches on cuda:  14%|█▎        | 22/161 [00:00<00:02, 62.44batch/s]

Training batches on cuda:  19%|█▉        | 31/161 [00:00<00:01, 66.86batch/s]

Training batches on cuda:  26%|██▌       | 42/161 [00:00<00:01, 74.58batch/s]

Training batches on cuda:  32%|███▏      | 51/161 [00:00<00:01, 78.44batch/s]

Training batches on cuda:  38%|███▊      | 61/161 [00:00<00:01, 81.99batch/s]

Training batches on cuda:  44%|████▍     | 71/161 [00:00<00:01, 85.88batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:00<00:00, 89.36batch/s]

Training batches on cuda:  57%|█████▋    | 92/161 [00:01<00:00, 92.38batch/s]

Training batches on cuda:  63%|██████▎   | 102/161 [00:01<00:00, 94.10batch/s]

Training batches on cuda:  70%|██████▉   | 112/161 [00:01<00:00, 87.02batch/s]

Training batches on cuda:  75%|███████▌  | 121/161 [00:01<00:00, 85.65batch/s]

Training batches on cuda:  81%|████████  | 130/161 [00:01<00:00, 84.21batch/s]

Training batches on cuda:  86%|████████▋ | 139/161 [00:01<00:00, 84.02batch/s]

Training batches on cuda:  92%|█████████▏| 148/161 [00:01<00:00, 83.09batch/s]

Training batches on cuda:  98%|█████████▊| 157/161 [00:01<00:00, 83.18batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:14<00:08,  2.09s/epoch, loss=0.00111, prev_loss=0.00111]
Training epochs on cuda:  70%|███████   | 7/10 [00:14<00:06,  2.11s/epoch, loss=0.00111, prev_loss=0.00111]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/161 [00:00<00:04, 36.08batch/s]

Training batches on cuda:   7%|▋         | 11/161 [00:00<00:03, 40.88batch/s]

Training batches on cuda:  11%|█         | 18/161 [00:00<00:03, 46.67batch/s]

Training batches on cuda:  16%|█▌        | 26/161 [00:00<00:02, 52.97batch/s]

Training batches on cuda:  22%|██▏       | 35/161 [00:00<00:02, 59.64batch/s]

Training batches on cuda:  28%|██▊       | 45/161 [00:00<00:01, 66.88batch/s]

Training batches on cuda:  34%|███▍      | 55/161 [00:00<00:01, 73.78batch/s]

Training batches on cuda:  40%|███▉      | 64/161 [00:00<00:01, 76.68batch/s]

Training batches on cuda:  45%|████▌     | 73/161 [00:00<00:01, 78.87batch/s]

Training batches on cuda:  51%|█████     | 82/161 [00:01<00:00, 80.84batch/s]

Training batches on cuda:  57%|█████▋    | 91/161 [00:01<00:00, 82.13batch/s]

Training batches on cuda:  62%|██████▏   | 100/161 [00:01<00:00, 80.72batch/s]

Training batches on cuda:  68%|██████▊   | 109/161 [00:01<00:00, 78.04batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:01<00:00, 76.00batch/s]

Training batches on cuda:  78%|███████▊  | 126/161 [00:01<00:00, 78.36batch/s]

Training batches on cuda:  84%|████████▍ | 135/161 [00:01<00:00, 81.07batch/s]

Training batches on cuda:  89%|████████▉ | 144/161 [00:01<00:00, 83.02batch/s]

Training batches on cuda:  95%|█████████▌| 153/161 [00:01<00:00, 84.89batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:06,  2.11s/epoch, loss=0.00111, prev_loss=0.00111]
Training epochs on cuda:  80%|████████  | 8/10 [00:17<00:04,  2.14s/epoch, loss=0.00111, prev_loss=0.00111]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 59.04batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 65.10batch/s]

Training batches on cuda:  15%|█▍        | 24/161 [00:00<00:01, 70.72batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:01, 75.08batch/s]

Training batches on cuda:  27%|██▋       | 43/161 [00:00<00:01, 79.13batch/s]

Training batches on cuda:  32%|███▏      | 52/161 [00:00<00:01, 82.01batch/s]

Training batches on cuda:  38%|███▊      | 61/161 [00:00<00:01, 84.25batch/s]

Training batches on cuda:  43%|████▎     | 70/161 [00:00<00:01, 85.54batch/s]

Training batches on cuda:  49%|████▉     | 79/161 [00:00<00:00, 86.26batch/s]

Training batches on cuda:  55%|█████▍    | 88/161 [00:01<00:00, 83.44batch/s]

Training batches on cuda:  60%|██████    | 97/161 [00:01<00:00, 82.09batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 80.14batch/s]

Training batches on cuda:  71%|███████   | 114/161 [00:01<00:00, 78.49batch/s]

Training batches on cuda:  76%|███████▌  | 122/161 [00:01<00:00, 78.45batch/s]

Training batches on cuda:  81%|████████  | 130/161 [00:01<00:00, 78.80batch/s]

Training batches on cuda:  86%|████████▋ | 139/161 [00:01<00:00, 79.31batch/s]

Training batches on cuda:  91%|█████████▏| 147/161 [00:01<00:00, 79.29batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:01<00:00, 78.96batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:19<00:04,  2.14s/epoch, loss=0.00111, prev_loss=0.00111]
Training epochs on cuda:  90%|█████████ | 9/10 [00:19<00:02,  2.17s/epoch, loss=0.00111, prev_loss=0.00111]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 47.90batch/s]

Training batches on cuda:   8%|▊         | 13/161 [00:00<00:02, 54.29batch/s]

Training batches on cuda:  14%|█▎        | 22/161 [00:00<00:02, 60.44batch/s]

Training batches on cuda:  19%|█▊        | 30/161 [00:00<00:02, 64.01batch/s]

Training batches on cuda:  24%|██▍       | 39/161 [00:00<00:01, 68.49batch/s]

Training batches on cuda:  30%|██▉       | 48/161 [00:00<00:01, 72.07batch/s]

Training batches on cuda:  35%|███▌      | 57/161 [00:00<00:01, 74.60batch/s]

Training batches on cuda:  40%|████      | 65/161 [00:00<00:01, 74.55batch/s]

Training batches on cuda:  46%|████▌     | 74/161 [00:00<00:01, 76.32batch/s]

Training batches on cuda:  51%|█████     | 82/161 [00:01<00:01, 74.58batch/s]

Training batches on cuda:  57%|█████▋    | 91/161 [00:01<00:00, 76.28batch/s]

Training batches on cuda:  61%|██████▏   | 99/161 [00:01<00:00, 77.12batch/s]

Training batches on cuda:  66%|██████▋   | 107/161 [00:01<00:00, 77.89batch/s]

Training batches on cuda:  71%|███████▏  | 115/161 [00:01<00:00, 77.62batch/s]

Training batches on cuda:  76%|███████▋  | 123/161 [00:01<00:00, 77.85batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:01<00:00, 76.73batch/s]

Training batches on cuda:  86%|████████▋ | 139/161 [00:01<00:00, 77.35batch/s]

Training batches on cuda:  91%|█████████▏| 147/161 [00:01<00:00, 77.12batch/s]

Training batches on cuda:  97%|█████████▋| 156/161 [00:02<00:00, 78.29batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:21<00:02,  2.17s/epoch, loss=0.00111, prev_loss=0.00111]2021-09-16:17:55:40,38 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:55:41,57 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:55:55,996 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:17:55:56,123 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:37<00:00,  7.05s/epoch, loss=0.00111, prev_loss=0.00111]
Training epochs on cuda: 100%|██████████| 10/10 [00:37<00:00,  3.77s/epoch, loss=0.00111, prev_loss=0.00111]
2021-09-16:17:55:56,124 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp78i8ncr9'
2021-09-16:17:55:56,196 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp78i8ncr9' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.52ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.60ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:56:10,996 INFO     [evaluator.py:657] Evaluation took 14.79s seconds
[I 2021-09-16 17:56:11,001] Trial 20 finished with value: 0.0035878156854735055 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.2718448394309829, 'optimizer.lr': 0.05255167315577165, 'negative_sampler.num_negs_per_pos': 2, 'training.batch_size': 1932}. Best is trial 12 with value: 0.9534737906551205.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:56:11,21 WARNING  [api.py:823] No random seed is specified. Setting to 174036689.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 14/720 [00:00<00:05, 138.59batch/s]

Training batches on cuda:   5%|▍         | 34/720 [00:00<00:04, 151.23batch/s]

Training batches on cuda:   7%|▋         | 53/720 [00:00<00:04, 160.18batch/s]

Training batches on cuda:  10%|█         | 73/720 [00:00<00:03, 169.60batch/s]

Training batches on cuda:  13%|█▎        | 92/720 [00:00<00:03, 173.79batch/s]

Training batches on cuda:  15%|█▌        | 111/720 [00:00<00:03, 176.72batch/s]

Training batches on cuda:  18%|█▊        | 129/720 [00:00<00:03, 177.64batch/s]

Training batches on cuda:  20%|██        | 146/720 [00:00<00:03, 174.90batch/s]

Training batches on cuda:  23%|██▎       | 165/720 [00:00<00:03, 177.08batch/s]

Training batches on cuda:  25%|██▌       | 183/720 [00:01<00:03, 177.42batch/s]

Training batches on cuda:  28%|██▊       | 201/720 [00:01<00:02, 176.98batch/s]

Training batches on cuda:  30%|███       | 219/720 [00:01<00:03, 163.89batch/s]

Training batches on cuda:  33%|███▎      | 236/720 [00:01<00:02, 163.48batch/s]

Training batches on cuda:  35%|███▌      | 253/720 [00:01<00:02, 164.63batch/s]

Training batches on cuda:  38%|███▊      | 273/720 [00:01<00:02, 172.08batch/s]

Training batches on cuda:  41%|████      | 292/720 [00:01<00:02, 176.51batch/s]

Training batches on cuda:  43%|████▎     | 312/720 [00:01<00:02, 181.38batch/s]

Training batches on cuda:  46%|████▌     | 332/720 [00:01<00:02, 183.26batch/s]

Training batches on cuda:  49%|████▉     | 353/720 [00:01<00:01, 189.02batch/s]

Training batches on cuda:  52%|█████▏    | 375/720 [00:02<00:01, 194.91batch/s]

Training batches on cuda:  55%|█████▌    | 396/720 [00:02<00:01, 198.91batch/s]

Training batches on cuda:  58%|█████▊    | 416/720 [00:02<00:01, 194.32batch/s]

Training batches on cuda:  61%|██████    | 436/720 [00:02<00:01, 192.17batch/s]

Training batches on cuda:  63%|██████▎   | 456/720 [00:02<00:01, 189.42batch/s]

Training batches on cuda:  66%|██████▌   | 476/720 [00:02<00:01, 186.51batch/s]

Training batches on cuda:  69%|██████▉   | 495/720 [00:02<00:01, 186.10batch/s]

Training batches on cuda:  71%|███████▏  | 514/720 [00:02<00:01, 186.57batch/s]

Training batches on cuda:  74%|███████▍  | 533/720 [00:02<00:01, 185.66batch/s]

Training batches on cuda:  77%|███████▋  | 552/720 [00:03<00:00, 172.84batch/s]

Training batches on cuda:  79%|███████▉  | 570/720 [00:03<00:00, 160.61batch/s]

Training batches on cuda:  82%|████████▏ | 589/720 [00:03<00:00, 167.44batch/s]

Training batches on cuda:  84%|████████▍ | 607/720 [00:03<00:00, 170.72batch/s]

Training batches on cuda:  87%|████████▋ | 627/720 [00:03<00:00, 177.40batch/s]

Training batches on cuda:  90%|████████▉ | 647/720 [00:03<00:00, 181.72batch/s]

Training batches on cuda:  92%|█████████▎| 666/720 [00:03<00:00, 181.35batch/s]

Training batches on cuda:  95%|█████████▌| 685/720 [00:03<00:00, 173.36batch/s]

Training batches on cuda:  98%|█████████▊| 703/720 [00:03<00:00, 165.88batch/s]

Training batches on cuda: 100%|██████████| 720/720 [00:04<00:00, 159.43batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.00305, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:38,  4.23s/epoch, loss=0.00305, prev_loss=nan]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 12/720 [00:00<00:06, 116.77batch/s]

Training batches on cuda:   4%|▍         | 28/720 [00:00<00:05, 126.03batch/s]

Training batches on cuda:   6%|▌         | 40/720 [00:00<00:05, 124.04batch/s]

Training batches on cuda:   8%|▊         | 58/720 [00:00<00:04, 136.67batch/s]

Training batches on cuda:  11%|█         | 76/720 [00:00<00:04, 146.26batch/s]

Training batches on cuda:  13%|█▎        | 93/720 [00:00<00:04, 151.97batch/s]

Training batches on cuda:  15%|█▌        | 110/720 [00:00<00:03, 155.42batch/s]

Training batches on cuda:  18%|█▊        | 128/720 [00:00<00:03, 160.13batch/s]

Training batches on cuda:  20%|██        | 147/720 [00:00<00:03, 166.84batch/s]

Training batches on cuda:  23%|██▎       | 166/720 [00:01<00:03, 172.50batch/s]

Training batches on cuda:  26%|██▌       | 185/720 [00:01<00:03, 177.27batch/s]

Training batches on cuda:  28%|██▊       | 203/720 [00:01<00:02, 177.10batch/s]

Training batches on cuda:  31%|███       | 221/720 [00:01<00:02, 176.50batch/s]

Training batches on cuda:  33%|███▎      | 240/720 [00:01<00:02, 178.80batch/s]

Training batches on cuda:  36%|███▌      | 259/720 [00:01<00:02, 181.48batch/s]

Training batches on cuda:  39%|███▊      | 278/720 [00:01<00:02, 182.51batch/s]

Training batches on cuda:  41%|████▏     | 297/720 [00:01<00:02, 183.01batch/s]

Training batches on cuda:  44%|████▍     | 316/720 [00:01<00:02, 183.49batch/s]

Training batches on cuda:  47%|████▋     | 335/720 [00:01<00:02, 182.33batch/s]

Training batches on cuda:  49%|████▉     | 354/720 [00:02<00:02, 175.93batch/s]

Training batches on cuda:  52%|█████▏    | 372/720 [00:02<00:01, 177.00batch/s]

Training batches on cuda:  54%|█████▍    | 390/720 [00:02<00:01, 176.18batch/s]

Training batches on cuda:  57%|█████▋    | 408/720 [00:02<00:01, 168.83batch/s]

Training batches on cuda:  59%|█████▉    | 425/720 [00:02<00:01, 168.16batch/s]

Training batches on cuda:  61%|██████▏   | 442/720 [00:02<00:01, 166.84batch/s]

Training batches on cuda:  64%|██████▍   | 460/720 [00:02<00:01, 169.83batch/s]

Training batches on cuda:  67%|██████▋   | 479/720 [00:02<00:01, 175.23batch/s]

Training batches on cuda:  69%|██████▉   | 499/720 [00:02<00:01, 180.53batch/s]

Training batches on cuda:  72%|███████▏  | 518/720 [00:02<00:01, 180.96batch/s]

Training batches on cuda:  75%|███████▍  | 537/720 [00:03<00:00, 183.30batch/s]

Training batches on cuda:  77%|███████▋  | 556/720 [00:03<00:00, 183.40batch/s]

Training batches on cuda:  80%|███████▉  | 575/720 [00:03<00:00, 181.72batch/s]

Training batches on cuda:  82%|████████▎ | 594/720 [00:03<00:00, 175.22batch/s]

Training batches on cuda:  85%|████████▌ | 613/720 [00:03<00:00, 178.63batch/s]

Training batches on cuda:  88%|████████▊ | 632/720 [00:03<00:00, 181.57batch/s]

Training batches on cuda:  90%|█████████ | 651/720 [00:03<00:00, 182.74batch/s]

Training batches on cuda:  93%|█████████▎| 670/720 [00:03<00:00, 183.86batch/s]

Training batches on cuda:  96%|█████████▌| 689/720 [00:03<00:00, 184.01batch/s]

Training batches on cuda:  98%|█████████▊| 708/720 [00:04<00:00, 182.18batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:38,  4.23s/epoch, loss=0.00303, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:33,  4.25s/epoch, loss=0.00303, prev_loss=nan]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 16/720 [00:00<00:04, 152.15batch/s]

Training batches on cuda:   5%|▌         | 36/720 [00:00<00:04, 162.06batch/s]

Training batches on cuda:   8%|▊         | 56/720 [00:00<00:03, 171.31batch/s]

Training batches on cuda:  11%|█         | 76/720 [00:00<00:03, 177.22batch/s]

Training batches on cuda:  13%|█▎        | 95/720 [00:00<00:03, 179.34batch/s]

Training batches on cuda:  16%|█▌        | 115/720 [00:00<00:03, 184.00batch/s]

Training batches on cuda:  19%|█▉        | 135/720 [00:00<00:03, 186.45batch/s]

Training batches on cuda:  21%|██▏       | 154/720 [00:00<00:03, 186.00batch/s]

Training batches on cuda:  24%|██▍       | 173/720 [00:00<00:02, 186.34batch/s]

Training batches on cuda:  27%|██▋       | 192/720 [00:01<00:03, 162.93batch/s]

Training batches on cuda:  29%|██▉       | 210/720 [00:01<00:03, 166.47batch/s]

Training batches on cuda:  32%|███▏      | 228/720 [00:01<00:02, 168.50batch/s]

Training batches on cuda:  34%|███▍      | 246/720 [00:01<00:02, 170.49batch/s]

Training batches on cuda:  37%|███▋      | 264/720 [00:01<00:02, 171.09batch/s]

Training batches on cuda:  39%|███▉      | 282/720 [00:01<00:02, 170.53batch/s]

Training batches on cuda:  42%|████▏     | 300/720 [00:01<00:02, 165.55batch/s]

Training batches on cuda:  44%|████▍     | 317/720 [00:01<00:02, 162.70batch/s]

Training batches on cuda:  46%|████▋     | 334/720 [00:01<00:02, 159.10batch/s]

Training batches on cuda:  49%|████▊     | 350/720 [00:02<00:02, 152.36batch/s]

Training batches on cuda:  51%|█████     | 366/720 [00:02<00:02, 154.36batch/s]

Training batches on cuda:  53%|█████▎    | 384/720 [00:02<00:02, 160.71batch/s]

Training batches on cuda:  56%|█████▌    | 401/720 [00:02<00:01, 162.16batch/s]

Training batches on cuda:  58%|█████▊    | 418/720 [00:02<00:01, 160.44batch/s]

Training batches on cuda:  60%|██████    | 435/720 [00:02<00:01, 160.54batch/s]

Training batches on cuda:  63%|██████▎   | 452/720 [00:02<00:01, 155.78batch/s]

Training batches on cuda:  65%|██████▌   | 468/720 [00:02<00:01, 145.76batch/s]

Training batches on cuda:  68%|██████▊   | 487/720 [00:02<00:01, 154.69batch/s]

Training batches on cuda:  70%|███████   | 504/720 [00:03<00:01, 158.27batch/s]

Training batches on cuda:  72%|███████▏  | 521/720 [00:03<00:01, 160.58batch/s]

Training batches on cuda:  75%|███████▍  | 538/720 [00:03<00:01, 162.42batch/s]

Training batches on cuda:  77%|███████▋  | 555/720 [00:03<00:01, 162.24batch/s]

Training batches on cuda:  80%|███████▉  | 573/720 [00:03<00:00, 165.31batch/s]

Training batches on cuda:  82%|████████▏ | 590/720 [00:03<00:00, 166.05batch/s]

Training batches on cuda:  84%|████████▍ | 607/720 [00:03<00:00, 166.55batch/s]

Training batches on cuda:  87%|████████▋ | 624/720 [00:03<00:00, 165.56batch/s]

Training batches on cuda:  89%|████████▉ | 641/720 [00:03<00:00, 164.79batch/s]

Training batches on cuda:  91%|█████████▏| 658/720 [00:03<00:00, 162.17batch/s]

Training batches on cuda:  94%|█████████▍| 675/720 [00:04<00:00, 157.31batch/s]

Training batches on cuda:  96%|█████████▌| 691/720 [00:04<00:00, 149.83batch/s]

Training batches on cuda:  98%|█████████▊| 707/720 [00:04<00:00, 151.29batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:33,  4.25s/epoch, loss=0.00298, prev_loss=0.00303]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:30,  4.33s/epoch, loss=0.00298, prev_loss=0.00303]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 9/720 [00:00<00:07, 89.93batch/s]

Training batches on cuda:   4%|▍         | 28/720 [00:00<00:06, 106.19batch/s]

Training batches on cuda:   6%|▌         | 43/720 [00:00<00:05, 114.99batch/s]

Training batches on cuda:   8%|▊         | 59/720 [00:00<00:05, 123.92batch/s]

Training batches on cuda:  10%|█         | 75/720 [00:00<00:04, 131.85batch/s]

Training batches on cuda:  13%|█▎        | 91/720 [00:00<00:04, 138.50batch/s]

Training batches on cuda:  15%|█▍        | 107/720 [00:00<00:04, 142.77batch/s]

Training batches on cuda:  17%|█▋        | 124/720 [00:00<00:03, 149.35batch/s]

Training batches on cuda:  19%|█▉        | 140/720 [00:00<00:03, 151.71batch/s]

Training batches on cuda:  22%|██▏       | 156/720 [00:01<00:03, 143.75batch/s]

Training batches on cuda:  24%|██▍       | 173/720 [00:01<00:03, 148.51batch/s]

Training batches on cuda:  26%|██▋       | 190/720 [00:01<00:03, 152.72batch/s]

Training batches on cuda:  29%|██▉       | 207/720 [00:01<00:03, 156.23batch/s]

Training batches on cuda:  31%|███       | 223/720 [00:01<00:03, 156.88batch/s]

Training batches on cuda:  33%|███▎      | 239/720 [00:01<00:03, 155.42batch/s]

Training batches on cuda:  35%|███▌      | 255/720 [00:01<00:03, 154.17batch/s]

Training batches on cuda:  38%|███▊      | 271/720 [00:01<00:02, 150.66batch/s]

Training batches on cuda:  40%|███▉      | 287/720 [00:01<00:02, 148.04batch/s]

Training batches on cuda:  42%|████▏     | 303/720 [00:02<00:02, 150.14batch/s]

Training batches on cuda:  44%|████▍     | 319/720 [00:02<00:02, 141.90batch/s]

Training batches on cuda:  46%|████▋     | 334/720 [00:02<00:02, 131.36batch/s]

Training batches on cuda:  48%|████▊     | 348/720 [00:02<00:02, 131.95batch/s]

Training batches on cuda:  50%|█████     | 362/720 [00:02<00:02, 131.62batch/s]

Training batches on cuda:  52%|█████▏    | 376/720 [00:02<00:02, 133.72batch/s]

Training batches on cuda:  55%|█████▍    | 393/720 [00:02<00:02, 141.31batch/s]

Training batches on cuda:  57%|█████▋    | 408/720 [00:02<00:02, 143.69batch/s]

Training batches on cuda:  59%|█████▉    | 424/720 [00:02<00:02, 146.25batch/s]

Training batches on cuda:  61%|██████    | 439/720 [00:02<00:01, 144.18batch/s]

Training batches on cuda:  63%|██████▎   | 456/720 [00:03<00:01, 150.52batch/s]

Training batches on cuda:  66%|██████▌   | 473/720 [00:03<00:01, 155.65batch/s]

Training batches on cuda:  68%|██████▊   | 490/720 [00:03<00:01, 157.96batch/s]

Training batches on cuda:  70%|███████   | 507/720 [00:03<00:01, 161.13batch/s]

Training batches on cuda:  73%|███████▎  | 524/720 [00:03<00:01, 161.00batch/s]

Training batches on cuda:  75%|███████▌  | 542/720 [00:03<00:01, 164.58batch/s]

Training batches on cuda:  78%|███████▊  | 559/720 [00:03<00:00, 165.82batch/s]

Training batches on cuda:  80%|████████  | 578/720 [00:03<00:00, 171.35batch/s]

Training batches on cuda:  83%|████████▎ | 597/720 [00:03<00:00, 174.11batch/s]

Training batches on cuda:  85%|████████▌ | 615/720 [00:04<00:00, 174.55batch/s]

Training batches on cuda:  88%|████████▊ | 633/720 [00:04<00:00, 173.12batch/s]

Training batches on cuda:  90%|█████████ | 651/720 [00:04<00:00, 173.95batch/s]

Training batches on cuda:  93%|█████████▎| 669/720 [00:04<00:00, 173.19batch/s]

Training batches on cuda:  95%|█████████▌| 687/720 [00:04<00:00, 166.41batch/s]

Training batches on cuda:  98%|█████████▊| 704/720 [00:04<00:00, 152.90batch/s]

Training batches on cuda: 100%|██████████| 720/720 [00:04<00:00, 152.43batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:30,  4.33s/epoch, loss=0.00285, prev_loss=0.00298]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:27,  4.51s/epoch, loss=0.00285, prev_loss=0.00298]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 106.36batch/s]

Training batches on cuda:   4%|▎         | 26/720 [00:00<00:06, 115.53batch/s]

Training batches on cuda:   6%|▌         | 42/720 [00:00<00:05, 125.26batch/s]

Training batches on cuda:   8%|▊         | 59/720 [00:00<00:04, 135.16batch/s]

Training batches on cuda:  10%|█         | 73/720 [00:00<00:04, 134.02batch/s]

Training batches on cuda:  12%|█▏        | 87/720 [00:00<00:04, 135.03batch/s]

Training batches on cuda:  14%|█▍        | 104/720 [00:00<00:04, 143.50batch/s]

Training batches on cuda:  17%|█▋        | 121/720 [00:00<00:03, 149.83batch/s]

Training batches on cuda:  19%|█▉        | 136/720 [00:00<00:04, 143.02batch/s]

Training batches on cuda:  21%|██▏       | 154/720 [00:01<00:03, 151.63batch/s]

Training batches on cuda:  24%|██▍       | 173/720 [00:01<00:03, 159.65batch/s]

Training batches on cuda:  27%|██▋       | 192/720 [00:01<00:03, 166.83batch/s]

Training batches on cuda:  29%|██▉       | 209/720 [00:01<00:03, 166.73batch/s]

Training batches on cuda:  32%|███▏      | 227/720 [00:01<00:02, 168.01batch/s]

Training batches on cuda:  34%|███▍      | 244/720 [00:01<00:02, 166.39batch/s]

Training batches on cuda:  36%|███▋      | 261/720 [00:01<00:03, 149.50batch/s]

Training batches on cuda:  38%|███▊      | 277/720 [00:01<00:02, 152.41batch/s]

Training batches on cuda:  41%|████      | 296/720 [00:01<00:02, 160.67batch/s]

Training batches on cuda:  44%|████▍     | 316/720 [00:02<00:02, 169.03batch/s]

Training batches on cuda:  47%|████▋     | 336/720 [00:02<00:02, 175.03batch/s]

Training batches on cuda:  49%|████▉     | 355/720 [00:02<00:02, 178.59batch/s]

Training batches on cuda:  52%|█████▏    | 374/720 [00:02<00:01, 173.97batch/s]

Training batches on cuda:  54%|█████▍    | 392/720 [00:02<00:01, 168.49batch/s]

Training batches on cuda:  57%|█████▋    | 410/720 [00:02<00:01, 171.23batch/s]

Training batches on cuda:  59%|█████▉    | 428/720 [00:02<00:01, 173.65batch/s]

Training batches on cuda:  62%|██████▏   | 446/720 [00:02<00:01, 171.17batch/s]

Training batches on cuda:  64%|██████▍   | 464/720 [00:02<00:01, 168.84batch/s]

Training batches on cuda:  67%|██████▋   | 481/720 [00:02<00:01, 165.15batch/s]

Training batches on cuda:  69%|██████▉   | 498/720 [00:03<00:01, 157.44batch/s]

Training batches on cuda:  71%|███████▏  | 514/720 [00:03<00:01, 155.65batch/s]

Training batches on cuda:  74%|███████▍  | 531/720 [00:03<00:01, 157.23batch/s]

Training batches on cuda:  76%|███████▌  | 548/720 [00:03<00:01, 158.48batch/s]

Training batches on cuda:  78%|███████▊  | 565/720 [00:03<00:00, 159.40batch/s]

Training batches on cuda:  81%|████████  | 582/720 [00:03<00:00, 160.12batch/s]

Training batches on cuda:  83%|████████▎ | 599/720 [00:03<00:00, 156.69batch/s]

Training batches on cuda:  85%|████████▌ | 615/720 [00:03<00:00, 147.09batch/s]

Training batches on cuda:  88%|████████▊ | 630/720 [00:03<00:00, 144.13batch/s]

Training batches on cuda:  90%|████████▉ | 647/720 [00:04<00:00, 150.15batch/s]

Training batches on cuda:  92%|█████████▏| 663/720 [00:04<00:00, 152.82batch/s]

Training batches on cuda:  94%|█████████▍| 679/720 [00:04<00:00, 153.89batch/s]

Training batches on cuda:  97%|█████████▋| 695/720 [00:04<00:00, 153.38batch/s]

Training batches on cuda:  99%|█████████▉| 711/720 [00:04<00:00, 155.22batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:27,  4.51s/epoch, loss=0.00239, prev_loss=0.00285]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:22,  4.57s/epoch, loss=0.00239, prev_loss=0.00285]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 10/720 [00:00<00:07, 97.97batch/s]

Training batches on cuda:   3%|▎         | 25/720 [00:00<00:06, 109.24batch/s]

Training batches on cuda:   6%|▌         | 43/720 [00:00<00:05, 122.29batch/s]

Training batches on cuda:   8%|▊         | 57/720 [00:00<00:05, 125.95batch/s]

Training batches on cuda:  10%|█         | 74/720 [00:00<00:04, 135.48batch/s]

Training batches on cuda:  13%|█▎        | 91/720 [00:00<00:04, 141.95batch/s]

Training batches on cuda:  15%|█▌        | 108/720 [00:00<00:04, 148.60batch/s]

Training batches on cuda:  18%|█▊        | 126/720 [00:00<00:03, 155.16batch/s]

Training batches on cuda:  20%|█▉        | 143/720 [00:00<00:03, 159.26batch/s]

Training batches on cuda:  22%|██▎       | 162/720 [00:01<00:03, 165.68batch/s]

Training batches on cuda:  25%|██▌       | 180/720 [00:01<00:03, 169.38batch/s]

Training batches on cuda:  28%|██▊       | 199/720 [00:01<00:02, 173.76batch/s]

Training batches on cuda:  30%|███       | 217/720 [00:01<00:02, 171.23batch/s]

Training batches on cuda:  33%|███▎      | 235/720 [00:01<00:02, 170.39batch/s]

Training batches on cuda:  35%|███▌      | 253/720 [00:01<00:02, 170.08batch/s]

Training batches on cuda:  38%|███▊      | 272/720 [00:01<00:02, 173.08batch/s]

Training batches on cuda:  40%|████      | 291/720 [00:01<00:02, 175.87batch/s]

Training batches on cuda:  43%|████▎     | 311/720 [00:01<00:02, 179.98batch/s]

Training batches on cuda:  46%|████▌     | 330/720 [00:01<00:02, 169.68batch/s]

Training batches on cuda:  49%|████▊     | 350/720 [00:02<00:02, 175.02batch/s]

Training batches on cuda:  51%|█████     | 368/720 [00:02<00:02, 164.11batch/s]

Training batches on cuda:  53%|█████▎    | 385/720 [00:02<00:02, 165.24batch/s]

Training batches on cuda:  56%|█████▌    | 402/720 [00:02<00:01, 162.96batch/s]

Training batches on cuda:  58%|█████▊    | 419/720 [00:02<00:01, 160.41batch/s]

Training batches on cuda:  61%|██████    | 436/720 [00:02<00:01, 159.53batch/s]

Training batches on cuda:  63%|██████▎   | 454/720 [00:02<00:01, 163.40batch/s]

Training batches on cuda:  66%|██████▌   | 472/720 [00:02<00:01, 167.78batch/s]

Training batches on cuda:  68%|██████▊   | 490/720 [00:02<00:01, 171.14batch/s]

Training batches on cuda:  71%|███████   | 508/720 [00:03<00:01, 172.66batch/s]

Training batches on cuda:  73%|███████▎  | 526/720 [00:03<00:01, 166.61batch/s]

Training batches on cuda:  76%|███████▌  | 544/720 [00:03<00:01, 168.97batch/s]

Training batches on cuda:  78%|███████▊  | 561/720 [00:03<00:00, 168.29batch/s]

Training batches on cuda:  81%|████████  | 580/720 [00:03<00:00, 172.01batch/s]

Training batches on cuda:  83%|████████▎ | 599/720 [00:03<00:00, 175.70batch/s]

Training batches on cuda:  86%|████████▌ | 617/720 [00:03<00:00, 175.98batch/s]

Training batches on cuda:  88%|████████▊ | 636/720 [00:03<00:00, 179.40batch/s]

Training batches on cuda:  91%|█████████ | 655/720 [00:03<00:00, 181.34batch/s]

Training batches on cuda:  94%|█████████▎| 674/720 [00:04<00:00, 181.05batch/s]

Training batches on cuda:  96%|█████████▋| 693/720 [00:04<00:00, 183.52batch/s]

Training batches on cuda:  99%|█████████▉| 712/720 [00:04<00:00, 183.93batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:27<00:22,  4.57s/epoch, loss=0.00181, prev_loss=0.00239]
Training epochs on cuda:  60%|██████    | 6/10 [00:27<00:18,  4.53s/epoch, loss=0.00181, prev_loss=0.00239]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 12/720 [00:00<00:05, 119.06batch/s]

Training batches on cuda:   4%|▍         | 31/720 [00:00<00:05, 133.02batch/s]

Training batches on cuda:   7%|▋         | 50/720 [00:00<00:04, 145.30batch/s]

Training batches on cuda:   9%|▉         | 67/720 [00:00<00:04, 151.44batch/s]

Training batches on cuda:  12%|█▏        | 83/720 [00:00<00:04, 153.26batch/s]

Training batches on cuda:  14%|█▍        | 99/720 [00:00<00:04, 155.20batch/s]

Training batches on cuda:  16%|█▋        | 117/720 [00:00<00:03, 159.16batch/s]

Training batches on cuda:  18%|█▊        | 133/720 [00:00<00:03, 153.73batch/s]

Training batches on cuda:  21%|██        | 148/720 [00:00<00:03, 143.60batch/s]

Training batches on cuda:  23%|██▎       | 163/720 [00:01<00:03, 139.52batch/s]

Training batches on cuda:  25%|██▍       | 178/720 [00:01<00:03, 142.30batch/s]

Training batches on cuda:  27%|██▋       | 194/720 [00:01<00:03, 145.66batch/s]

Training batches on cuda:  29%|██▉       | 211/720 [00:01<00:03, 152.08batch/s]

Training batches on cuda:  32%|███▏      | 228/720 [00:01<00:03, 156.33batch/s]

Training batches on cuda:  34%|███▍      | 247/720 [00:01<00:02, 163.14batch/s]

Training batches on cuda:  37%|███▋      | 264/720 [00:01<00:02, 156.20batch/s]

Training batches on cuda:  39%|███▉      | 282/720 [00:01<00:02, 161.94batch/s]

Training batches on cuda:  42%|████▏     | 302/720 [00:01<00:02, 170.76batch/s]

Training batches on cuda:  45%|████▍     | 322/720 [00:01<00:02, 177.91batch/s]

Training batches on cuda:  47%|████▋     | 341/720 [00:02<00:02, 181.16batch/s]

Training batches on cuda:  50%|█████     | 361/720 [00:02<00:01, 186.43batch/s]

Training batches on cuda:  53%|█████▎    | 381/720 [00:02<00:01, 190.10batch/s]

Training batches on cuda:  56%|█████▌    | 401/720 [00:02<00:01, 190.17batch/s]

Training batches on cuda:  58%|█████▊    | 421/720 [00:02<00:01, 191.49batch/s]

Training batches on cuda:  61%|██████▏   | 441/720 [00:02<00:01, 172.52batch/s]

Training batches on cuda:  64%|██████▍   | 460/720 [00:02<00:01, 176.33batch/s]

Training batches on cuda:  67%|██████▋   | 480/720 [00:02<00:01, 180.89batch/s]

Training batches on cuda:  69%|██████▉   | 500/720 [00:02<00:01, 185.34batch/s]

Training batches on cuda:  72%|███████▏  | 520/720 [00:03<00:01, 187.99batch/s]

Training batches on cuda:  75%|███████▍  | 539/720 [00:03<00:00, 184.77batch/s]

Training batches on cuda:  78%|███████▊  | 558/720 [00:03<00:00, 180.50batch/s]

Training batches on cuda:  80%|████████  | 577/720 [00:03<00:00, 179.56batch/s]

Training batches on cuda:  83%|████████▎ | 596/720 [00:03<00:00, 172.30batch/s]

Training batches on cuda:  85%|████████▌ | 615/720 [00:03<00:00, 174.68batch/s]

Training batches on cuda:  88%|████████▊ | 634/720 [00:03<00:00, 176.36batch/s]

Training batches on cuda:  91%|█████████ | 652/720 [00:03<00:00, 164.39batch/s]

Training batches on cuda:  93%|█████████▎| 669/720 [00:03<00:00, 164.17batch/s]

Training batches on cuda:  95%|█████████▌| 686/720 [00:04<00:00, 162.70batch/s]

Training batches on cuda:  98%|█████████▊| 703/720 [00:04<00:00, 162.02batch/s]

Training batches on cuda: 100%|██████████| 720/720 [00:04<00:00, 161.19batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:31<00:18,  4.53s/epoch, loss=0.0014, prev_loss=0.00181] 
Training epochs on cuda:  70%|███████   | 7/10 [00:31<00:13,  4.50s/epoch, loss=0.0014, prev_loss=0.00181]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 16/720 [00:00<00:04, 153.54batch/s]

Training batches on cuda:   5%|▌         | 36/720 [00:00<00:04, 163.83batch/s]

Training batches on cuda:   8%|▊         | 56/720 [00:00<00:03, 172.56batch/s]

Training batches on cuda:  11%|█         | 76/720 [00:00<00:03, 178.32batch/s]

Training batches on cuda:  13%|█▎        | 93/720 [00:00<00:03, 172.58batch/s]

Training batches on cuda:  15%|█▌        | 110/720 [00:00<00:03, 170.96batch/s]

Training batches on cuda:  18%|█▊        | 126/720 [00:00<00:03, 167.01batch/s]

Training batches on cuda:  20%|█▉        | 142/720 [00:00<00:03, 154.57batch/s]

Training batches on cuda:  22%|██▏       | 157/720 [00:00<00:03, 148.16batch/s]

Training batches on cuda:  24%|██▍       | 172/720 [00:01<00:03, 148.63batch/s]

Training batches on cuda:  26%|██▋       | 189/720 [00:01<00:03, 154.44batch/s]

Training batches on cuda:  28%|██▊       | 205/720 [00:01<00:03, 154.67batch/s]

Training batches on cuda:  31%|███       | 222/720 [00:01<00:03, 158.96batch/s]

Training batches on cuda:  33%|███▎      | 240/720 [00:01<00:02, 164.31batch/s]

Training batches on cuda:  36%|███▌      | 257/720 [00:01<00:02, 156.70batch/s]

Training batches on cuda:  38%|███▊      | 273/720 [00:01<00:02, 155.09batch/s]

Training batches on cuda:  40%|████      | 289/720 [00:01<00:02, 152.95batch/s]

Training batches on cuda:  42%|████▏     | 305/720 [00:01<00:02, 150.28batch/s]

Training batches on cuda:  45%|████▍     | 321/720 [00:02<00:02, 150.37batch/s]

Training batches on cuda:  47%|████▋     | 337/720 [00:02<00:02, 151.95batch/s]

Training batches on cuda:  49%|████▉     | 353/720 [00:02<00:02, 152.68batch/s]

Training batches on cuda:  51%|█████▏    | 369/720 [00:02<00:02, 152.80batch/s]

Training batches on cuda:  53%|█████▎    | 385/720 [00:02<00:02, 154.81batch/s]

Training batches on cuda:  56%|█████▌    | 402/720 [00:02<00:02, 157.17batch/s]

Training batches on cuda:  58%|█████▊    | 419/720 [00:02<00:01, 160.33batch/s]

Training batches on cuda:  61%|██████    | 437/720 [00:02<00:01, 165.49batch/s]

Training batches on cuda:  63%|██████▎   | 454/720 [00:02<00:01, 165.49batch/s]

Training batches on cuda:  65%|██████▌   | 471/720 [00:02<00:01, 151.93batch/s]

Training batches on cuda:  68%|██████▊   | 487/720 [00:03<00:01, 147.63batch/s]

Training batches on cuda:  70%|██████▉   | 502/720 [00:03<00:01, 130.55batch/s]

Training batches on cuda:  72%|███████▏  | 519/720 [00:03<00:01, 139.42batch/s]

Training batches on cuda:  74%|███████▍  | 536/720 [00:03<00:01, 145.86batch/s]

Training batches on cuda:  77%|███████▋  | 553/720 [00:03<00:01, 150.10batch/s]

Training batches on cuda:  79%|███████▉  | 569/720 [00:03<00:01, 149.79batch/s]

Training batches on cuda:  81%|████████▏ | 585/720 [00:03<00:00, 151.09batch/s]

Training batches on cuda:  83%|████████▎ | 601/720 [00:03<00:00, 151.64batch/s]

Training batches on cuda:  86%|████████▌ | 618/720 [00:03<00:00, 154.20batch/s]

Training batches on cuda:  88%|████████▊ | 635/720 [00:04<00:00, 156.39batch/s]

Training batches on cuda:  91%|█████████ | 652/720 [00:04<00:00, 157.84batch/s]

Training batches on cuda:  93%|█████████▎| 671/720 [00:04<00:00, 165.17batch/s]

Training batches on cuda:  96%|█████████▌| 689/720 [00:04<00:00, 167.58batch/s]

Training batches on cuda:  98%|█████████▊| 708/720 [00:04<00:00, 171.14batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:36<00:13,  4.50s/epoch, loss=0.00128, prev_loss=0.0014]
Training epochs on cuda:  80%|████████  | 8/10 [00:36<00:09,  4.57s/epoch, loss=0.00128, prev_loss=0.0014]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 10/720 [00:00<00:07, 96.30batch/s]

Training batches on cuda:   4%|▍         | 27/720 [00:00<00:06, 108.96batch/s]

Training batches on cuda:   6%|▋         | 46/720 [00:00<00:05, 124.82batch/s]

Training batches on cuda:   9%|▉         | 66/720 [00:00<00:04, 139.24batch/s]

Training batches on cuda:  12%|█▏        | 85/720 [00:00<00:04, 150.39batch/s]

Training batches on cuda:  14%|█▍        | 104/720 [00:00<00:03, 158.73batch/s]

Training batches on cuda:  17%|█▋        | 122/720 [00:00<00:03, 164.33batch/s]

Training batches on cuda:  19%|█▉        | 139/720 [00:00<00:03, 165.70batch/s]

Training batches on cuda:  22%|██▏       | 157/720 [00:00<00:03, 166.93batch/s]

Training batches on cuda:  24%|██▍       | 174/720 [00:01<00:03, 166.96batch/s]

Training batches on cuda:  27%|██▋       | 191/720 [00:01<00:03, 164.80batch/s]

Training batches on cuda:  29%|██▉       | 208/720 [00:01<00:03, 163.43batch/s]

Training batches on cuda:  31%|███▏      | 225/720 [00:01<00:03, 163.78batch/s]

Training batches on cuda:  34%|███▍      | 244/720 [00:01<00:02, 169.96batch/s]

Training batches on cuda:  37%|███▋      | 263/720 [00:01<00:02, 174.10batch/s]

Training batches on cuda:  39%|███▉      | 281/720 [00:01<00:02, 174.38batch/s]

Training batches on cuda:  42%|████▏     | 300/720 [00:01<00:02, 178.56batch/s]

Training batches on cuda:  45%|████▍     | 321/720 [00:01<00:02, 184.95batch/s]

Training batches on cuda:  47%|████▋     | 341/720 [00:01<00:02, 188.45batch/s]

Training batches on cuda:  50%|█████     | 360/720 [00:02<00:01, 180.66batch/s]

Training batches on cuda:  53%|█████▎    | 379/720 [00:02<00:01, 176.20batch/s]

Training batches on cuda:  55%|█████▌    | 397/720 [00:02<00:01, 171.67batch/s]

Training batches on cuda:  58%|█████▊    | 415/720 [00:02<00:01, 164.47batch/s]

Training batches on cuda:  60%|██████    | 432/720 [00:02<00:01, 164.37batch/s]

Training batches on cuda:  62%|██████▎   | 450/720 [00:02<00:01, 167.45batch/s]

Training batches on cuda:  65%|██████▍   | 467/720 [00:02<00:01, 167.52batch/s]

Training batches on cuda:  67%|██████▋   | 485/720 [00:02<00:01, 169.63batch/s]

Training batches on cuda:  70%|███████   | 505/720 [00:02<00:01, 175.52batch/s]

Training batches on cuda:  73%|███████▎  | 524/720 [00:03<00:01, 178.69batch/s]

Training batches on cuda:  75%|███████▌  | 543/720 [00:03<00:00, 180.38batch/s]

Training batches on cuda:  78%|███████▊  | 562/720 [00:03<00:00, 180.48batch/s]

Training batches on cuda:  81%|████████  | 581/720 [00:03<00:00, 178.92batch/s]

Training batches on cuda:  83%|████████▎ | 599/720 [00:03<00:00, 163.21batch/s]

Training batches on cuda:  86%|████████▌ | 618/720 [00:03<00:00, 167.96batch/s]

Training batches on cuda:  88%|████████▊ | 636/720 [00:03<00:00, 166.56batch/s]

Training batches on cuda:  91%|█████████ | 653/720 [00:03<00:00, 162.42batch/s]

Training batches on cuda:  93%|█████████▎| 670/720 [00:03<00:00, 161.22batch/s]

Training batches on cuda:  95%|█████████▌| 687/720 [00:04<00:00, 162.92batch/s]

Training batches on cuda:  98%|█████████▊| 704/720 [00:04<00:00, 162.85batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:40<00:09,  4.57s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda:  90%|█████████ | 9/10 [00:40<00:04,  4.53s/epoch, loss=0.00128, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 104.63batch/s]

Training batches on cuda:   4%|▍         | 29/720 [00:00<00:05, 119.12batch/s]

Training batches on cuda:   7%|▋         | 48/720 [00:00<00:05, 133.52batch/s]

Training batches on cuda:   9%|▉         | 68/720 [00:00<00:04, 147.28batch/s]

Training batches on cuda:  12%|█▏        | 87/720 [00:00<00:04, 157.71batch/s]

Training batches on cuda:  15%|█▍        | 107/720 [00:00<00:03, 167.97batch/s]

Training batches on cuda:  18%|█▊        | 126/720 [00:00<00:03, 173.88batch/s]

Training batches on cuda:  20%|██        | 145/720 [00:00<00:03, 177.40batch/s]

Training batches on cuda:  23%|██▎       | 165/720 [00:00<00:03, 181.50batch/s]

Training batches on cuda:  26%|██▌       | 184/720 [00:01<00:02, 182.87batch/s]

Training batches on cuda:  28%|██▊       | 203/720 [00:01<00:02, 181.52batch/s]

Training batches on cuda:  31%|███       | 222/720 [00:01<00:02, 177.18batch/s]

Training batches on cuda:  33%|███▎      | 240/720 [00:01<00:02, 173.03batch/s]

Training batches on cuda:  36%|███▌      | 258/720 [00:01<00:02, 162.40batch/s]

Training batches on cuda:  38%|███▊      | 275/720 [00:01<00:02, 163.98batch/s]

Training batches on cuda:  41%|████      | 292/720 [00:01<00:02, 164.30batch/s]

Training batches on cuda:  43%|████▎     | 310/720 [00:01<00:02, 168.00batch/s]

Training batches on cuda:  46%|████▌     | 328/720 [00:01<00:02, 170.88batch/s]

Training batches on cuda:  48%|████▊     | 346/720 [00:02<00:02, 166.45batch/s]

Training batches on cuda:  50%|█████     | 363/720 [00:02<00:02, 159.53batch/s]

Training batches on cuda:  53%|█████▎    | 380/720 [00:02<00:02, 159.35batch/s]

Training batches on cuda:  55%|█████▌    | 399/720 [00:02<00:01, 166.42batch/s]

Training batches on cuda:  58%|█████▊    | 418/720 [00:02<00:01, 171.24batch/s]

Training batches on cuda:  61%|██████    | 436/720 [00:02<00:01, 172.19batch/s]

Training batches on cuda:  63%|██████▎   | 454/720 [00:02<00:01, 172.55batch/s]

Training batches on cuda:  66%|██████▌   | 472/720 [00:02<00:01, 170.99batch/s]

Training batches on cuda:  68%|██████▊   | 490/720 [00:02<00:01, 163.04batch/s]

Training batches on cuda:  70%|███████   | 507/720 [00:02<00:01, 157.37batch/s]

Training batches on cuda:  73%|███████▎  | 526/720 [00:03<00:01, 165.06batch/s]

Training batches on cuda:  76%|███████▌  | 545/720 [00:03<00:01, 167.95batch/s]

Training batches on cuda:  78%|███████▊  | 562/720 [00:03<00:00, 166.45batch/s]

Training batches on cuda:  81%|████████  | 580/720 [00:03<00:00, 167.68batch/s]

Training batches on cuda:  83%|████████▎ | 597/720 [00:03<00:00, 166.27batch/s]

Training batches on cuda:  85%|████████▌ | 614/720 [00:03<00:00, 166.45batch/s]

Training batches on cuda:  88%|████████▊ | 631/720 [00:03<00:00, 164.76batch/s]

Training batches on cuda:  90%|█████████ | 648/720 [00:03<00:00, 161.12batch/s]

Training batches on cuda:  92%|█████████▏| 665/720 [00:03<00:00, 159.41batch/s]

Training batches on cuda:  95%|█████████▍| 681/720 [00:04<00:00, 143.96batch/s]

Training batches on cuda:  97%|█████████▋| 697/720 [00:04<00:00, 148.10batch/s]

Training batches on cuda:  99%|█████████▉| 714/720 [00:04<00:00, 153.20batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:45<00:04,  4.53s/epoch, loss=0.00132, prev_loss=0.00128]2021-09-16:17:56:56,661 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:56:57,691 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:57:12,589 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
2021-09-16:17:57:12,714 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  9.34s/epoch, loss=0.00132, prev_loss=0.00128]
Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  6.13s/epoch, loss=0.00132, prev_loss=0.00128]
2021-09-16:17:57:12,715 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp4uuewxq9'
2021-09-16:17:57:12,807 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4uuewxq9' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.38ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.66ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.66ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:57:27,582 INFO     [evaluator.py:657] Evaluation took 14.75s seconds
[I 2021-09-16 17:57:27,586] Trial 21 finished with value: 0.9617823966700032 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.010648135061145285, 'optimizer.lr': 0.01924988969706163, 'negative_sampler.num_negs_per_pos': 7, 'training.batch_size': 432}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:57:27,606 WARNING  [api.py:823] No random seed is specified. Setting to 2028709159.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 6/275 [00:00<00:04, 57.29batch/s]

Training batches on cuda:   5%|▌         | 15/275 [00:00<00:04, 64.28batch/s]

Training batches on cuda:   9%|▊         | 24/275 [00:00<00:03, 69.92batch/s]

Training batches on cuda:  12%|█▏        | 34/275 [00:00<00:03, 75.56batch/s]

Training batches on cuda:  16%|█▌        | 44/275 [00:00<00:02, 80.02batch/s]

Training batches on cuda:  19%|█▉        | 53/275 [00:00<00:02, 82.53batch/s]

Training batches on cuda:  23%|██▎       | 63/275 [00:00<00:02, 86.69batch/s]

Training batches on cuda:  27%|██▋       | 73/275 [00:00<00:02, 88.70batch/s]

Training batches on cuda:  31%|███       | 84/275 [00:00<00:02, 93.63batch/s]

Training batches on cuda:  35%|███▍      | 96/275 [00:01<00:01, 98.17batch/s]

Training batches on cuda:  39%|███▉      | 108/275 [00:01<00:01, 101.90batch/s]

Training batches on cuda:  43%|████▎     | 119/275 [00:01<00:01, 103.12batch/s]

Training batches on cuda:  48%|████▊     | 131/275 [00:01<00:01, 105.16batch/s]

Training batches on cuda:  52%|█████▏    | 142/275 [00:01<00:01, 106.12batch/s]

Training batches on cuda:  56%|█████▌    | 153/275 [00:01<00:01, 105.75batch/s]

Training batches on cuda:  60%|██████    | 165/275 [00:01<00:01, 107.58batch/s]

Training batches on cuda:  64%|██████▍   | 177/275 [00:01<00:00, 108.99batch/s]

Training batches on cuda:  68%|██████▊   | 188/275 [00:01<00:00, 104.63batch/s]

Training batches on cuda:  72%|███████▏  | 199/275 [00:02<00:00, 103.25batch/s]

Training batches on cuda:  76%|███████▋  | 210/275 [00:02<00:00, 101.85batch/s]

Training batches on cuda:  80%|████████  | 221/275 [00:02<00:00, 100.57batch/s]

Training batches on cuda:  84%|████████▍ | 232/275 [00:02<00:00, 100.41batch/s]

Training batches on cuda:  88%|████████▊ | 243/275 [00:02<00:00, 99.70batch/s] 

Training batches on cuda:  92%|█████████▏| 253/275 [00:02<00:00, 99.16batch/s]

Training batches on cuda:  96%|█████████▌| 263/275 [00:02<00:00, 98.45batch/s]

Training batches on cuda:  99%|█████████▉| 273/275 [00:02<00:00, 98.42batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.00117, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:27,  3.03s/epoch, loss=0.00117, prev_loss=nan]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 8/275 [00:00<00:03, 73.11batch/s]

Training batches on cuda:   7%|▋         | 18/275 [00:00<00:03, 78.83batch/s]

Training batches on cuda:  10%|█         | 28/275 [00:00<00:02, 83.39batch/s]

Training batches on cuda:  14%|█▍        | 38/275 [00:00<00:02, 85.88batch/s]

Training batches on cuda:  17%|█▋        | 48/275 [00:00<00:02, 89.29batch/s]

Training batches on cuda:  21%|██        | 58/275 [00:00<00:02, 90.50batch/s]

Training batches on cuda:  24%|██▍       | 67/275 [00:00<00:02, 90.32batch/s]

Training batches on cuda:  28%|██▊       | 77/275 [00:00<00:02, 92.86batch/s]

Training batches on cuda:  32%|███▏      | 87/275 [00:00<00:02, 92.62batch/s]

Training batches on cuda:  35%|███▌      | 97/275 [00:01<00:01, 91.81batch/s]

Training batches on cuda:  39%|███▉      | 107/275 [00:01<00:01, 88.06batch/s]

Training batches on cuda:  43%|████▎     | 117/275 [00:01<00:01, 90.50batch/s]

Training batches on cuda:  46%|████▌     | 127/275 [00:01<00:01, 92.30batch/s]

Training batches on cuda:  50%|████▉     | 137/275 [00:01<00:01, 91.65batch/s]

Training batches on cuda:  53%|█████▎    | 147/275 [00:01<00:01, 91.23batch/s]

Training batches on cuda:  57%|█████▋    | 157/275 [00:01<00:01, 90.38batch/s]

Training batches on cuda:  61%|██████    | 167/275 [00:01<00:01, 88.51batch/s]

Training batches on cuda:  64%|██████▍   | 177/275 [00:01<00:01, 89.48batch/s]

Training batches on cuda:  68%|██████▊   | 187/275 [00:02<00:00, 90.47batch/s]

Training batches on cuda:  72%|███████▏  | 197/275 [00:02<00:00, 87.12batch/s]

Training batches on cuda:  75%|███████▍  | 206/275 [00:02<00:00, 83.92batch/s]

Training batches on cuda:  79%|███████▉  | 217/275 [00:02<00:00, 88.59batch/s]

Training batches on cuda:  83%|████████▎ | 228/275 [00:02<00:00, 93.14batch/s]

Training batches on cuda:  87%|████████▋ | 239/275 [00:02<00:00, 97.59batch/s]

Training batches on cuda:  91%|█████████ | 250/275 [00:02<00:00, 100.29batch/s]

Training batches on cuda:  95%|█████████▍| 261/275 [00:02<00:00, 99.74batch/s] 

Training batches on cuda:  99%|█████████▉| 272/275 [00:02<00:00, 99.25batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:27,  3.03s/epoch, loss=0.00116, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:24,  3.09s/epoch, loss=0.00116, prev_loss=nan]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/275 [00:00<00:05, 48.92batch/s]

Training batches on cuda:   6%|▌         | 16/275 [00:00<00:04, 58.43batch/s]

Training batches on cuda:  10%|▉         | 27/275 [00:00<00:03, 67.15batch/s]

Training batches on cuda:  14%|█▍        | 38/275 [00:00<00:03, 74.93batch/s]

Training batches on cuda:  18%|█▊        | 49/275 [00:00<00:02, 81.92batch/s]

Training batches on cuda:  21%|██▏       | 59/275 [00:00<00:02, 86.55batch/s]

Training batches on cuda:  25%|██▌       | 70/275 [00:00<00:02, 91.70batch/s]

Training batches on cuda:  29%|██▉       | 81/275 [00:00<00:02, 94.76batch/s]

Training batches on cuda:  33%|███▎      | 91/275 [00:00<00:01, 94.53batch/s]

Training batches on cuda:  37%|███▋      | 101/275 [00:01<00:01, 91.29batch/s]

Training batches on cuda:  40%|████      | 111/275 [00:01<00:01, 84.02batch/s]

Training batches on cuda:  44%|████▍     | 121/275 [00:01<00:01, 87.22batch/s]

Training batches on cuda:  48%|████▊     | 132/275 [00:01<00:01, 90.96batch/s]

Training batches on cuda:  52%|█████▏    | 142/275 [00:01<00:01, 90.33batch/s]

Training batches on cuda:  56%|█████▌    | 153/275 [00:01<00:01, 94.67batch/s]

Training batches on cuda:  60%|█████▉    | 164/275 [00:01<00:01, 97.48batch/s]

Training batches on cuda:  64%|██████▎   | 175/275 [00:01<00:01, 99.89batch/s]

Training batches on cuda:  68%|██████▊   | 186/275 [00:01<00:00, 101.77batch/s]

Training batches on cuda:  72%|███████▏  | 198/275 [00:02<00:00, 104.73batch/s]

Training batches on cuda:  76%|███████▌  | 209/275 [00:02<00:00, 106.04batch/s]

Training batches on cuda:  80%|████████  | 220/275 [00:02<00:00, 106.96batch/s]

Training batches on cuda:  84%|████████▍ | 231/275 [00:02<00:00, 106.97batch/s]

Training batches on cuda:  88%|████████▊ | 243/275 [00:02<00:00, 108.89batch/s]

Training batches on cuda:  92%|█████████▏| 254/275 [00:02<00:00, 107.34batch/s]

Training batches on cuda:  96%|█████████▋| 265/275 [00:02<00:00, 98.82batch/s] 

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:24,  3.09s/epoch, loss=0.00116, prev_loss=0.00116]
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:21,  3.08s/epoch, loss=0.00116, prev_loss=0.00116]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/275 [00:00<00:05, 45.63batch/s]

Training batches on cuda:   5%|▌         | 15/275 [00:00<00:04, 54.21batch/s]

Training batches on cuda:   9%|▉         | 25/275 [00:00<00:03, 62.75batch/s]

Training batches on cuda:  13%|█▎        | 35/275 [00:00<00:03, 70.63batch/s]

Training batches on cuda:  17%|█▋        | 46/275 [00:00<00:02, 78.64batch/s]

Training batches on cuda:  21%|██        | 58/275 [00:00<00:02, 85.49batch/s]

Training batches on cuda:  25%|██▌       | 69/275 [00:00<00:02, 91.12batch/s]

Training batches on cuda:  29%|██▊       | 79/275 [00:00<00:02, 93.55batch/s]

Training batches on cuda:  33%|███▎      | 90/275 [00:00<00:01, 97.42batch/s]

Training batches on cuda:  36%|███▋      | 100/275 [00:01<00:01, 97.90batch/s]

Training batches on cuda:  40%|████      | 111/275 [00:01<00:01, 101.13batch/s]

Training batches on cuda:  44%|████▍     | 122/275 [00:01<00:01, 99.30batch/s] 

Training batches on cuda:  48%|████▊     | 133/275 [00:01<00:01, 98.73batch/s]

Training batches on cuda:  52%|█████▏    | 143/275 [00:01<00:01, 98.48batch/s]

Training batches on cuda:  56%|█████▌    | 154/275 [00:01<00:01, 100.17batch/s]

Training batches on cuda:  60%|██████    | 165/275 [00:01<00:01, 102.05batch/s]

Training batches on cuda:  64%|██████▍   | 176/275 [00:01<00:00, 103.47batch/s]

Training batches on cuda:  68%|██████▊   | 187/275 [00:01<00:00, 104.05batch/s]

Training batches on cuda:  72%|███████▏  | 198/275 [00:01<00:00, 105.29batch/s]

Training batches on cuda:  76%|███████▌  | 209/275 [00:02<00:00, 103.78batch/s]

Training batches on cuda:  80%|████████  | 220/275 [00:02<00:00, 101.38batch/s]

Training batches on cuda:  84%|████████▍ | 231/275 [00:02<00:00, 100.63batch/s]

Training batches on cuda:  88%|████████▊ | 242/275 [00:02<00:00, 101.80batch/s]

Training batches on cuda:  92%|█████████▏| 253/275 [00:02<00:00, 102.96batch/s]

Training batches on cuda:  96%|█████████▌| 264/275 [00:02<00:00, 103.34batch/s]

Training batches on cuda: 100%|██████████| 275/275 [00:02<00:00, 103.97batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:21,  3.08s/epoch, loss=0.00116, prev_loss=0.00116]
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:18,  3.03s/epoch, loss=0.00116, prev_loss=0.00116]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 9/275 [00:00<00:02, 89.77batch/s]

Training batches on cuda:   8%|▊         | 21/275 [00:00<00:02, 95.33batch/s]

Training batches on cuda:  12%|█▏        | 33/275 [00:00<00:02, 101.21batch/s]

Training batches on cuda:  16%|█▋        | 45/275 [00:00<00:02, 105.98batch/s]

Training batches on cuda:  21%|██        | 58/275 [00:00<00:01, 109.96batch/s]

Training batches on cuda:  25%|██▌       | 70/275 [00:00<00:01, 112.09batch/s]

Training batches on cuda:  30%|██▉       | 82/275 [00:00<00:01, 112.95batch/s]

Training batches on cuda:  34%|███▍      | 94/275 [00:00<00:01, 113.39batch/s]

Training batches on cuda:  39%|███▊      | 106/275 [00:00<00:01, 115.25batch/s]

Training batches on cuda:  43%|████▎     | 118/275 [00:01<00:01, 116.20batch/s]

Training batches on cuda:  47%|████▋     | 130/275 [00:01<00:01, 116.95batch/s]

Training batches on cuda:  52%|█████▏    | 142/275 [00:01<00:01, 117.59batch/s]

Training batches on cuda:  56%|█████▌    | 154/275 [00:01<00:01, 117.05batch/s]

Training batches on cuda:  61%|██████    | 167/275 [00:01<00:00, 117.99batch/s]

Training batches on cuda:  65%|██████▌   | 179/275 [00:01<00:00, 104.63batch/s]

Training batches on cuda:  69%|██████▉   | 190/275 [00:01<00:00, 104.83batch/s]

Training batches on cuda:  73%|███████▎  | 202/275 [00:01<00:00, 106.42batch/s]

Training batches on cuda:  77%|███████▋  | 213/275 [00:01<00:00, 101.03batch/s]

Training batches on cuda:  81%|████████▏ | 224/275 [00:02<00:00, 99.77batch/s] 

Training batches on cuda:  85%|████████▌ | 235/275 [00:02<00:00, 98.08batch/s]

Training batches on cuda:  89%|████████▉ | 245/275 [00:02<00:00, 94.34batch/s]

Training batches on cuda:  93%|█████████▎| 257/275 [00:02<00:00, 98.59batch/s]

Training batches on cuda:  97%|█████████▋| 268/275 [00:02<00:00, 100.89batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:18,  3.03s/epoch, loss=0.00117, prev_loss=0.00116]
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:14,  2.97s/epoch, loss=0.00117, prev_loss=0.00116]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 8/275 [00:00<00:03, 75.11batch/s]

Training batches on cuda:   5%|▌         | 15/275 [00:00<00:03, 72.40batch/s]

Training batches on cuda:   9%|▉         | 25/275 [00:00<00:03, 78.21batch/s]

Training batches on cuda:  13%|█▎        | 35/275 [00:00<00:02, 83.17batch/s]

Training batches on cuda:  16%|█▌        | 43/275 [00:00<00:02, 80.28batch/s]

Training batches on cuda:  20%|█▉        | 54/275 [00:00<00:02, 85.53batch/s]

Training batches on cuda:  23%|██▎       | 64/275 [00:00<00:02, 87.00batch/s]

Training batches on cuda:  27%|██▋       | 74/275 [00:00<00:02, 89.55batch/s]

Training batches on cuda:  31%|███▏      | 86/275 [00:00<00:01, 95.08batch/s]

Training batches on cuda:  35%|███▌      | 97/275 [00:01<00:01, 97.02batch/s]

Training batches on cuda:  39%|███▉      | 107/275 [00:01<00:01, 95.01batch/s]

Training batches on cuda:  43%|████▎     | 117/275 [00:01<00:01, 95.24batch/s]

Training batches on cuda:  47%|████▋     | 128/275 [00:01<00:01, 97.00batch/s]

Training batches on cuda:  51%|█████     | 139/275 [00:01<00:01, 98.72batch/s]

Training batches on cuda:  54%|█████▍    | 149/275 [00:01<00:01, 97.51batch/s]

Training batches on cuda:  58%|█████▊    | 159/275 [00:01<00:01, 95.84batch/s]

Training batches on cuda:  62%|██████▏   | 170/275 [00:01<00:01, 98.54batch/s]

Training batches on cuda:  66%|██████▌   | 181/275 [00:01<00:00, 100.71batch/s]

Training batches on cuda:  70%|██████▉   | 192/275 [00:02<00:00, 97.78batch/s] 

Training batches on cuda:  74%|███████▍  | 203/275 [00:02<00:00, 100.46batch/s]

Training batches on cuda:  78%|███████▊  | 214/275 [00:02<00:00, 98.46batch/s] 

Training batches on cuda:  81%|████████▏ | 224/275 [00:02<00:00, 98.90batch/s]

Training batches on cuda:  85%|████████▌ | 234/275 [00:02<00:00, 97.89batch/s]

Training batches on cuda:  89%|████████▊ | 244/275 [00:02<00:00, 95.21batch/s]

Training batches on cuda:  92%|█████████▏| 254/275 [00:02<00:00, 94.41batch/s]

Training batches on cuda:  96%|█████████▋| 265/275 [00:02<00:00, 96.40batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:14,  2.97s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:12,  3.00s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 9/275 [00:00<00:03, 83.37batch/s]

Training batches on cuda:   7%|▋         | 19/275 [00:00<00:02, 86.93batch/s]

Training batches on cuda:  11%|█         | 30/275 [00:00<00:02, 91.72batch/s]

Training batches on cuda:  15%|█▍        | 41/275 [00:00<00:02, 95.69batch/s]

Training batches on cuda:  19%|█▉        | 52/275 [00:00<00:02, 98.54batch/s]

Training batches on cuda:  23%|██▎       | 63/275 [00:00<00:02, 101.28batch/s]

Training batches on cuda:  27%|██▋       | 73/275 [00:00<00:02, 100.43batch/s]

Training batches on cuda:  30%|███       | 83/275 [00:00<00:02, 92.75batch/s] 

Training batches on cuda:  33%|███▎      | 92/275 [00:00<00:02, 91.41batch/s]

Training batches on cuda:  37%|███▋      | 103/275 [00:01<00:01, 94.09batch/s]

Training batches on cuda:  41%|████      | 113/275 [00:01<00:01, 95.21batch/s]

Training batches on cuda:  45%|████▍     | 123/275 [00:01<00:01, 93.15batch/s]

Training batches on cuda:  48%|████▊     | 133/275 [00:01<00:01, 93.09batch/s]

Training batches on cuda:  53%|█████▎    | 145/275 [00:01<00:01, 97.74batch/s]

Training batches on cuda:  57%|█████▋    | 157/275 [00:01<00:01, 102.00batch/s]

Training batches on cuda:  61%|██████▏   | 169/275 [00:01<00:01, 103.23batch/s]

Training batches on cuda:  65%|██████▌   | 180/275 [00:01<00:00, 102.74batch/s]

Training batches on cuda:  69%|██████▉   | 191/275 [00:01<00:00, 96.94batch/s] 

Training batches on cuda:  73%|███████▎  | 201/275 [00:02<00:00, 96.92batch/s]

Training batches on cuda:  77%|███████▋  | 211/275 [00:02<00:00, 87.48batch/s]

Training batches on cuda:  80%|████████  | 220/275 [00:02<00:00, 85.74batch/s]

Training batches on cuda:  84%|████████▎ | 230/275 [00:02<00:00, 87.36batch/s]

Training batches on cuda:  87%|████████▋ | 240/275 [00:02<00:00, 89.04batch/s]

Training batches on cuda:  91%|█████████▏| 251/275 [00:02<00:00, 93.68batch/s]

Training batches on cuda:  95%|█████████▌| 262/275 [00:02<00:00, 97.00batch/s]

Training batches on cuda:  99%|█████████▉| 273/275 [00:02<00:00, 99.60batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:21<00:12,  3.00s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  70%|███████   | 7/10 [00:21<00:09,  3.01s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 9/275 [00:00<00:03, 85.68batch/s]

Training batches on cuda:   7%|▋         | 19/275 [00:00<00:02, 88.84batch/s]

Training batches on cuda:  10%|█         | 28/275 [00:00<00:02, 88.64batch/s]

Training batches on cuda:  14%|█▍        | 39/275 [00:00<00:02, 92.61batch/s]

Training batches on cuda:  18%|█▊        | 50/275 [00:00<00:02, 97.12batch/s]

Training batches on cuda:  22%|██▏       | 60/275 [00:00<00:02, 97.76batch/s]

Training batches on cuda:  26%|██▌       | 72/275 [00:00<00:02, 101.40batch/s]

Training batches on cuda:  30%|███       | 83/275 [00:00<00:01, 102.13batch/s]

Training batches on cuda:  34%|███▍      | 94/275 [00:00<00:01, 101.90batch/s]

Training batches on cuda:  38%|███▊      | 105/275 [00:01<00:01, 102.59batch/s]

Training batches on cuda:  42%|████▏     | 116/275 [00:01<00:01, 102.83batch/s]

Training batches on cuda:  46%|████▌     | 127/275 [00:01<00:01, 102.60batch/s]

Training batches on cuda:  50%|█████     | 138/275 [00:01<00:01, 102.56batch/s]

Training batches on cuda:  54%|█████▍    | 149/275 [00:01<00:01, 101.80batch/s]

Training batches on cuda:  58%|█████▊    | 160/275 [00:01<00:01, 99.65batch/s] 

Training batches on cuda:  63%|██████▎   | 172/275 [00:01<00:01, 101.78batch/s]

Training batches on cuda:  67%|██████▋   | 183/275 [00:01<00:00, 98.72batch/s] 

Training batches on cuda:  70%|███████   | 193/275 [00:01<00:00, 98.64batch/s]

Training batches on cuda:  74%|███████▍  | 204/275 [00:02<00:00, 100.72batch/s]

Training batches on cuda:  78%|███████▊  | 215/275 [00:02<00:00, 103.06batch/s]

Training batches on cuda:  83%|████████▎ | 227/275 [00:02<00:00, 105.60batch/s]

Training batches on cuda:  87%|████████▋ | 238/275 [00:02<00:00, 105.45batch/s]

Training batches on cuda:  91%|█████████ | 249/275 [00:02<00:00, 105.23batch/s]

Training batches on cuda:  95%|█████████▍| 260/275 [00:02<00:00, 103.74batch/s]

Training batches on cuda:  99%|█████████▊| 271/275 [00:02<00:00, 103.97batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:24<00:09,  3.01s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  80%|████████  | 8/10 [00:24<00:05,  2.98s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 4/275 [00:00<00:07, 37.02batch/s]

Training batches on cuda:   4%|▍         | 11/275 [00:00<00:06, 42.75batch/s]

Training batches on cuda:   8%|▊         | 21/275 [00:00<00:04, 50.95batch/s]

Training batches on cuda:  11%|█▏        | 31/275 [00:00<00:04, 59.74batch/s]

Training batches on cuda:  15%|█▌        | 42/275 [00:00<00:03, 68.06batch/s]

Training batches on cuda:  19%|█▊        | 51/275 [00:00<00:03, 72.55batch/s]

Training batches on cuda:  22%|██▏       | 61/275 [00:00<00:02, 78.78batch/s]

Training batches on cuda:  26%|██▌       | 72/275 [00:00<00:02, 84.45batch/s]

Training batches on cuda:  30%|███       | 83/275 [00:00<00:02, 89.38batch/s]

Training batches on cuda:  35%|███▍      | 95/275 [00:01<00:01, 95.91batch/s]

Training batches on cuda:  39%|███▉      | 107/275 [00:01<00:01, 101.13batch/s]

Training batches on cuda:  43%|████▎     | 118/275 [00:01<00:01, 94.62batch/s] 

Training batches on cuda:  47%|████▋     | 130/275 [00:01<00:01, 99.98batch/s]

Training batches on cuda:  52%|█████▏    | 142/275 [00:01<00:01, 103.43batch/s]

Training batches on cuda:  56%|█████▌    | 154/275 [00:01<00:01, 107.02batch/s]

Training batches on cuda:  60%|██████    | 166/275 [00:01<00:01, 108.31batch/s]

Training batches on cuda:  64%|██████▍   | 177/275 [00:01<00:00, 108.46batch/s]

Training batches on cuda:  69%|██████▊   | 189/275 [00:01<00:00, 110.09batch/s]

Training batches on cuda:  73%|███████▎  | 201/275 [00:02<00:00, 111.98batch/s]

Training batches on cuda:  77%|███████▋  | 213/275 [00:02<00:00, 112.99batch/s]

Training batches on cuda:  82%|████████▏ | 225/275 [00:02<00:00, 113.81batch/s]

Training batches on cuda:  86%|████████▌ | 237/275 [00:02<00:00, 114.14batch/s]

Training batches on cuda:  91%|█████████ | 249/275 [00:02<00:00, 114.40batch/s]

Training batches on cuda:  95%|█████████▍| 261/275 [00:02<00:00, 113.70batch/s]

Training batches on cuda:  99%|█████████▉| 273/275 [00:02<00:00, 114.42batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:05,  2.98s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  90%|█████████ | 9/10 [00:26<00:02,  2.96s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/275 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 9/275 [00:00<00:03, 88.61batch/s]

Training batches on cuda:   8%|▊         | 21/275 [00:00<00:02, 94.58batch/s]

Training batches on cuda:  12%|█▏        | 33/275 [00:00<00:02, 100.06batch/s]

Training batches on cuda:  16%|█▋        | 45/275 [00:00<00:02, 104.35batch/s]

Training batches on cuda:  21%|██        | 57/275 [00:00<00:02, 107.85batch/s]

Training batches on cuda:  25%|██▌       | 69/275 [00:00<00:01, 111.19batch/s]

Training batches on cuda:  30%|██▉       | 82/275 [00:00<00:01, 114.11batch/s]

Training batches on cuda:  34%|███▍      | 94/275 [00:00<00:01, 115.45batch/s]

Training batches on cuda:  39%|███▊      | 106/275 [00:00<00:01, 115.05batch/s]

Training batches on cuda:  43%|████▎     | 118/275 [00:01<00:01, 113.90batch/s]

Training batches on cuda:  47%|████▋     | 130/275 [00:01<00:01, 113.59batch/s]

Training batches on cuda:  52%|█████▏    | 142/275 [00:01<00:01, 114.26batch/s]

Training batches on cuda:  56%|█████▌    | 154/275 [00:01<00:01, 114.53batch/s]

Training batches on cuda:  60%|██████    | 166/275 [00:01<00:00, 115.33batch/s]

Training batches on cuda:  65%|██████▍   | 178/275 [00:01<00:00, 115.65batch/s]

Training batches on cuda:  69%|██████▉   | 190/275 [00:01<00:00, 115.94batch/s]

Training batches on cuda:  73%|███████▎  | 202/275 [00:01<00:00, 114.12batch/s]

Training batches on cuda:  78%|███████▊  | 214/275 [00:01<00:00, 114.22batch/s]

Training batches on cuda:  82%|████████▏ | 226/275 [00:01<00:00, 115.02batch/s]

Training batches on cuda:  87%|████████▋ | 238/275 [00:02<00:00, 115.16batch/s]

Training batches on cuda:  91%|█████████ | 250/275 [00:02<00:00, 114.26batch/s]

Training batches on cuda:  95%|█████████▌| 262/275 [00:02<00:00, 114.85batch/s]

Training batches on cuda: 100%|█████████▉| 274/275 [00:02<00:00, 113.58batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:29<00:02,  2.96s/epoch, loss=0.00117, prev_loss=0.00117]2021-09-16:17:57:57,603 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:57:58,520 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:58:13,450 INFO     [evaluator.py:657] Evaluation took 14.93s seconds
2021-09-16:17:58:13,603 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  7.65s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  4.56s/epoch, loss=0.00117, prev_loss=0.00117]
2021-09-16:17:58:13,604 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmphmqxb2fx'
2021-09-16:17:58:13,671 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmphmqxb2fx' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.36ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:58:28,472 INFO     [evaluator.py:657] Evaluation took 14.79s seconds
[I 2021-09-16 17:58:28,525] Trial 22 finished with value: 0.007966234629993796 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.09298282924080511, 'optimizer.lr': 0.026656798722465028, 'negative_sampler.num_negs_per_pos': 10, 'training.batch_size': 1132}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:58:28,548 WARNING  [api.py:823] No random seed is specified. Setting to 430102860.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 14/720 [00:00<00:05, 136.73batch/s]

Training batches on cuda:   5%|▍         | 35/720 [00:00<00:04, 151.95batch/s]

Training batches on cuda:   8%|▊         | 55/720 [00:00<00:04, 163.47batch/s]

Training batches on cuda:  10%|█         | 75/720 [00:00<00:03, 172.32batch/s]

Training batches on cuda:  13%|█▎        | 96/720 [00:00<00:03, 180.34batch/s]

Training batches on cuda:  16%|█▌        | 116/720 [00:00<00:03, 185.61batch/s]

Training batches on cuda:  19%|█▉        | 137/720 [00:00<00:03, 190.19batch/s]

Training batches on cuda:  22%|██▏       | 158/720 [00:00<00:02, 193.21batch/s]

Training batches on cuda:  25%|██▍       | 179/720 [00:00<00:02, 196.13batch/s]

Training batches on cuda:  28%|██▊       | 199/720 [00:01<00:02, 191.34batch/s]

Training batches on cuda:  30%|███       | 218/720 [00:01<00:02, 188.35batch/s]

Training batches on cuda:  33%|███▎      | 237/720 [00:01<00:02, 186.42batch/s]

Training batches on cuda:  36%|███▌      | 257/720 [00:01<00:02, 187.92batch/s]

Training batches on cuda:  38%|███▊      | 277/720 [00:01<00:02, 189.26batch/s]

Training batches on cuda:  41%|████▏     | 297/720 [00:01<00:02, 189.88batch/s]

Training batches on cuda:  44%|████▍     | 318/720 [00:01<00:02, 192.64batch/s]

Training batches on cuda:  47%|████▋     | 338/720 [00:01<00:02, 183.34batch/s]

Training batches on cuda:  50%|████▉     | 357/720 [00:01<00:01, 183.40batch/s]

Training batches on cuda:  52%|█████▏    | 377/720 [00:01<00:01, 187.15batch/s]

Training batches on cuda:  55%|█████▌    | 397/720 [00:02<00:01, 188.14batch/s]

Training batches on cuda:  58%|█████▊    | 416/720 [00:02<00:01, 188.08batch/s]

Training batches on cuda:  61%|██████    | 436/720 [00:02<00:01, 188.95batch/s]

Training batches on cuda:  63%|██████▎   | 455/720 [00:02<00:01, 188.80batch/s]

Training batches on cuda:  66%|██████▌   | 474/720 [00:02<00:01, 188.12batch/s]

Training batches on cuda:  68%|██████▊   | 493/720 [00:02<00:01, 184.96batch/s]

Training batches on cuda:  71%|███████   | 512/720 [00:02<00:01, 182.87batch/s]

Training batches on cuda:  74%|███████▍  | 531/720 [00:02<00:01, 180.73batch/s]

Training batches on cuda:  76%|███████▋  | 550/720 [00:02<00:00, 178.64batch/s]

Training batches on cuda:  79%|███████▉  | 568/720 [00:03<00:00, 172.32batch/s]

Training batches on cuda:  81%|████████▏ | 586/720 [00:03<00:00, 166.36batch/s]

Training batches on cuda:  84%|████████▍ | 603/720 [00:03<00:00, 162.84batch/s]

Training batches on cuda:  86%|████████▌ | 620/720 [00:03<00:00, 161.19batch/s]

Training batches on cuda:  88%|████████▊ | 637/720 [00:03<00:00, 159.45batch/s]

Training batches on cuda:  91%|█████████ | 656/720 [00:03<00:00, 166.24batch/s]

Training batches on cuda:  94%|█████████▍| 675/720 [00:03<00:00, 171.75batch/s]

Training batches on cuda:  96%|█████████▋| 694/720 [00:03<00:00, 175.95batch/s]

Training batches on cuda:  99%|█████████▉| 713/720 [00:03<00:00, 179.52batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.00493, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:36,  4.10s/epoch, loss=0.00493, prev_loss=nan]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 15/720 [00:00<00:04, 147.73batch/s]

Training batches on cuda:   5%|▍         | 33/720 [00:00<00:04, 155.56batch/s]

Training batches on cuda:   7%|▋         | 50/720 [00:00<00:04, 158.75batch/s]

Training batches on cuda:   9%|▉         | 68/720 [00:00<00:03, 164.31batch/s]

Training batches on cuda:  12%|█▏        | 88/720 [00:00<00:03, 171.86batch/s]

Training batches on cuda:  14%|█▍        | 103/720 [00:00<00:04, 145.14batch/s]

Training batches on cuda:  16%|█▋        | 118/720 [00:00<00:04, 143.87batch/s]

Training batches on cuda:  18%|█▊        | 133/720 [00:00<00:04, 143.40batch/s]

Training batches on cuda:  21%|██        | 148/720 [00:00<00:04, 142.68batch/s]

Training batches on cuda:  23%|██▎       | 167/720 [00:01<00:03, 152.83batch/s]

Training batches on cuda:  26%|██▌       | 187/720 [00:01<00:03, 162.85batch/s]

Training batches on cuda:  29%|██▉       | 207/720 [00:01<00:03, 170.24batch/s]

Training batches on cuda:  32%|███▏      | 227/720 [00:01<00:02, 176.52batch/s]

Training batches on cuda:  34%|███▍      | 247/720 [00:01<00:02, 182.46batch/s]

Training batches on cuda:  37%|███▋      | 267/720 [00:01<00:02, 186.87batch/s]

Training batches on cuda:  40%|████      | 288/720 [00:01<00:02, 191.67batch/s]

Training batches on cuda:  43%|████▎     | 308/720 [00:01<00:02, 190.67batch/s]

Training batches on cuda:  46%|████▌     | 328/720 [00:01<00:02, 186.12batch/s]

Training batches on cuda:  48%|████▊     | 347/720 [00:02<00:02, 181.19batch/s]

Training batches on cuda:  51%|█████     | 366/720 [00:02<00:02, 169.23batch/s]

Training batches on cuda:  53%|█████▎    | 385/720 [00:02<00:01, 174.92batch/s]

Training batches on cuda:  56%|█████▌    | 404/720 [00:02<00:01, 178.61batch/s]

Training batches on cuda:  59%|█████▉    | 424/720 [00:02<00:01, 182.26batch/s]

Training batches on cuda:  62%|██████▏   | 443/720 [00:02<00:01, 183.97batch/s]

Training batches on cuda:  64%|██████▍   | 462/720 [00:02<00:01, 182.42batch/s]

Training batches on cuda:  67%|██████▋   | 481/720 [00:02<00:01, 183.90batch/s]

Training batches on cuda:  69%|██████▉   | 500/720 [00:02<00:01, 176.37batch/s]

Training batches on cuda:  72%|███████▏  | 518/720 [00:02<00:01, 172.84batch/s]

Training batches on cuda:  74%|███████▍  | 536/720 [00:03<00:01, 170.85batch/s]

Training batches on cuda:  77%|███████▋  | 554/720 [00:03<00:00, 171.57batch/s]

Training batches on cuda:  80%|███████▉  | 574/720 [00:03<00:00, 177.48batch/s]

Training batches on cuda:  82%|████████▎ | 594/720 [00:03<00:00, 181.91batch/s]

Training batches on cuda:  85%|████████▌ | 613/720 [00:03<00:00, 183.90batch/s]

Training batches on cuda:  88%|████████▊ | 632/720 [00:03<00:00, 184.70batch/s]

Training batches on cuda:  90%|█████████ | 651/720 [00:03<00:00, 183.59batch/s]

Training batches on cuda:  93%|█████████▎| 670/720 [00:03<00:00, 174.55batch/s]

Training batches on cuda:  96%|█████████▌| 688/720 [00:03<00:00, 173.73batch/s]

Training batches on cuda:  98%|█████████▊| 706/720 [00:04<00:00, 174.97batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:36,  4.10s/epoch, loss=0.00493, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:33,  4.15s/epoch, loss=0.00493, prev_loss=nan]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 105.78batch/s]

Training batches on cuda:   4%|▍         | 29/720 [00:00<00:05, 120.60batch/s]

Training batches on cuda:   6%|▌         | 43/720 [00:00<00:05, 125.18batch/s]

Training batches on cuda:   8%|▊         | 57/720 [00:00<00:05, 129.19batch/s]

Training batches on cuda:  10%|█         | 73/720 [00:00<00:04, 135.86batch/s]

Training batches on cuda:  12%|█▏        | 89/720 [00:00<00:04, 140.28batch/s]

Training batches on cuda:  14%|█▍        | 103/720 [00:00<00:04, 137.63batch/s]

Training batches on cuda:  17%|█▋        | 119/720 [00:00<00:04, 141.67batch/s]

Training batches on cuda:  19%|█▊        | 134/720 [00:00<00:04, 142.21batch/s]

Training batches on cuda:  21%|██        | 150/720 [00:01<00:03, 144.43batch/s]

Training batches on cuda:  23%|██▎       | 165/720 [00:01<00:03, 144.68batch/s]

Training batches on cuda:  25%|██▌       | 182/720 [00:01<00:03, 151.07batch/s]

Training batches on cuda:  28%|██▊       | 198/720 [00:01<00:04, 127.42batch/s]

Training batches on cuda:  29%|██▉       | 212/720 [00:01<00:03, 127.94batch/s]

Training batches on cuda:  31%|███▏      | 226/720 [00:01<00:03, 127.89batch/s]

Training batches on cuda:  33%|███▎      | 240/720 [00:01<00:03, 130.37batch/s]

Training batches on cuda:  35%|███▌      | 255/720 [00:01<00:03, 133.45batch/s]

Training batches on cuda:  37%|███▋      | 269/720 [00:01<00:03, 128.90batch/s]

Training batches on cuda:  39%|███▉      | 283/720 [00:02<00:03, 131.73batch/s]

Training batches on cuda:  41%|████▏     | 298/720 [00:02<00:03, 134.61batch/s]

Training batches on cuda:  44%|████▍     | 315/720 [00:02<00:02, 141.89batch/s]

Training batches on cuda:  46%|████▌     | 332/720 [00:02<00:02, 147.68batch/s]

Training batches on cuda:  48%|████▊     | 349/720 [00:02<00:02, 151.29batch/s]

Training batches on cuda:  51%|█████     | 367/720 [00:02<00:02, 158.89batch/s]

Training batches on cuda:  54%|█████▍    | 387/720 [00:02<00:01, 167.31batch/s]

Training batches on cuda:  56%|█████▌    | 404/720 [00:02<00:01, 160.16batch/s]

Training batches on cuda:  58%|█████▊    | 421/720 [00:02<00:01, 157.33batch/s]

Training batches on cuda:  61%|██████    | 437/720 [00:03<00:01, 157.67batch/s]

Training batches on cuda:  63%|██████▎   | 456/720 [00:03<00:01, 164.98batch/s]

Training batches on cuda:  66%|██████▌   | 474/720 [00:03<00:01, 168.95batch/s]

Training batches on cuda:  68%|██████▊   | 492/720 [00:03<00:01, 170.03batch/s]

Training batches on cuda:  71%|███████   | 510/720 [00:03<00:01, 172.13batch/s]

Training batches on cuda:  73%|███████▎  | 529/720 [00:03<00:01, 174.53batch/s]

Training batches on cuda:  76%|███████▌  | 547/720 [00:03<00:00, 175.77batch/s]

Training batches on cuda:  78%|███████▊  | 565/720 [00:03<00:00, 176.24batch/s]

Training batches on cuda:  81%|████████  | 583/720 [00:03<00:00, 176.75batch/s]

Training batches on cuda:  83%|████████▎ | 601/720 [00:03<00:00, 176.07batch/s]

Training batches on cuda:  86%|████████▌ | 620/720 [00:04<00:00, 179.12batch/s]

Training batches on cuda:  89%|████████▉ | 639/720 [00:04<00:00, 181.10batch/s]

Training batches on cuda:  91%|█████████▏| 658/720 [00:04<00:00, 174.60batch/s]

Training batches on cuda:  94%|█████████▍| 676/720 [00:04<00:00, 157.83batch/s]

Training batches on cuda:  96%|█████████▋| 694/720 [00:04<00:00, 161.79batch/s]

Training batches on cuda:  99%|█████████▉| 713/720 [00:04<00:00, 168.42batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:33,  4.15s/epoch, loss=0.00493, prev_loss=0.00493]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:30,  4.36s/epoch, loss=0.00493, prev_loss=0.00493]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 14/720 [00:00<00:05, 133.93batch/s]

Training batches on cuda:   4%|▎         | 26/720 [00:00<00:05, 129.09batch/s]

Training batches on cuda:   5%|▌         | 38/720 [00:00<00:05, 125.85batch/s]

Training batches on cuda:   8%|▊         | 55/720 [00:00<00:04, 134.60batch/s]

Training batches on cuda:  10%|▉         | 71/720 [00:00<00:04, 141.32batch/s]

Training batches on cuda:  12%|█▎        | 90/720 [00:00<00:04, 152.67batch/s]

Training batches on cuda:  15%|█▌        | 108/720 [00:00<00:03, 159.13batch/s]

Training batches on cuda:  17%|█▋        | 124/720 [00:00<00:03, 157.49batch/s]

Training batches on cuda:  19%|█▉        | 140/720 [00:00<00:03, 155.72batch/s]

Training batches on cuda:  22%|██▏       | 156/720 [00:01<00:03, 156.98batch/s]

Training batches on cuda:  24%|██▍       | 173/720 [00:01<00:03, 158.89batch/s]

Training batches on cuda:  26%|██▋       | 189/720 [00:01<00:03, 152.39batch/s]

Training batches on cuda:  28%|██▊       | 205/720 [00:01<00:03, 152.50batch/s]

Training batches on cuda:  31%|███       | 223/720 [00:01<00:03, 157.51batch/s]

Training batches on cuda:  33%|███▎      | 239/720 [00:01<00:03, 152.79batch/s]

Training batches on cuda:  36%|███▌      | 257/720 [00:01<00:02, 157.91batch/s]

Training batches on cuda:  38%|███▊      | 274/720 [00:01<00:02, 161.04batch/s]

Training batches on cuda:  40%|████      | 291/720 [00:01<00:02, 162.22batch/s]

Training batches on cuda:  43%|████▎     | 308/720 [00:01<00:02, 162.34batch/s]

Training batches on cuda:  45%|████▌     | 326/720 [00:02<00:02, 165.43batch/s]

Training batches on cuda:  48%|████▊     | 344/720 [00:02<00:02, 167.82batch/s]

Training batches on cuda:  50%|█████     | 361/720 [00:02<00:02, 160.87batch/s]

Training batches on cuda:  52%|█████▎    | 378/720 [00:02<00:02, 161.75batch/s]

Training batches on cuda:  55%|█████▍    | 395/720 [00:02<00:01, 162.58batch/s]

Training batches on cuda:  57%|█████▋    | 413/720 [00:02<00:01, 166.09batch/s]

Training batches on cuda:  60%|██████    | 432/720 [00:02<00:01, 171.49batch/s]

Training batches on cuda:  63%|██████▎   | 451/720 [00:02<00:01, 175.98batch/s]

Training batches on cuda:  65%|██████▌   | 470/720 [00:02<00:01, 177.76batch/s]

Training batches on cuda:  68%|██████▊   | 489/720 [00:03<00:01, 178.79batch/s]

Training batches on cuda:  71%|███████   | 508/720 [00:03<00:01, 180.18batch/s]

Training batches on cuda:  73%|███████▎  | 527/720 [00:03<00:01, 179.14batch/s]

Training batches on cuda:  76%|███████▌  | 545/720 [00:03<00:00, 178.58batch/s]

Training batches on cuda:  78%|███████▊  | 563/720 [00:03<00:00, 173.03batch/s]

Training batches on cuda:  81%|████████  | 581/720 [00:03<00:00, 168.66batch/s]

Training batches on cuda:  83%|████████▎ | 598/720 [00:03<00:00, 168.34batch/s]

Training batches on cuda:  85%|████████▌ | 615/720 [00:03<00:00, 167.22batch/s]

Training batches on cuda:  88%|████████▊ | 632/720 [00:03<00:00, 166.83batch/s]

Training batches on cuda:  90%|█████████ | 649/720 [00:03<00:00, 165.32batch/s]

Training batches on cuda:  92%|█████████▎| 666/720 [00:04<00:00, 165.78batch/s]

Training batches on cuda:  95%|█████████▌| 684/720 [00:04<00:00, 168.65batch/s]

Training batches on cuda:  97%|█████████▋| 701/720 [00:04<00:00, 168.47batch/s]

Training batches on cuda: 100%|█████████▉| 718/720 [00:04<00:00, 162.26batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:30,  4.36s/epoch, loss=0.00492, prev_loss=0.00493]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:26,  4.43s/epoch, loss=0.00492, prev_loss=0.00493]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 12/720 [00:00<00:06, 114.19batch/s]

Training batches on cuda:   4%|▍         | 28/720 [00:00<00:05, 123.44batch/s]

Training batches on cuda:   6%|▌         | 44/720 [00:00<00:05, 131.49batch/s]

Training batches on cuda:   8%|▊         | 60/720 [00:00<00:04, 137.28batch/s]

Training batches on cuda:  11%|█         | 76/720 [00:00<00:04, 143.03batch/s]

Training batches on cuda:  13%|█▎        | 92/720 [00:00<00:04, 146.65batch/s]

Training batches on cuda:  15%|█▌        | 108/720 [00:00<00:04, 149.65batch/s]

Training batches on cuda:  17%|█▋        | 124/720 [00:00<00:03, 152.06batch/s]

Training batches on cuda:  19%|█▉        | 140/720 [00:00<00:03, 153.87batch/s]

Training batches on cuda:  22%|██▏       | 156/720 [00:01<00:03, 152.89batch/s]

Training batches on cuda:  24%|██▍       | 173/720 [00:01<00:03, 156.84batch/s]

Training batches on cuda:  27%|██▋       | 191/720 [00:01<00:03, 162.23batch/s]

Training batches on cuda:  29%|██▉       | 209/720 [00:01<00:03, 165.59batch/s]

Training batches on cuda:  31%|███▏      | 226/720 [00:01<00:03, 163.95batch/s]

Training batches on cuda:  34%|███▍      | 243/720 [00:01<00:02, 160.52batch/s]

Training batches on cuda:  36%|███▌      | 260/720 [00:01<00:02, 158.92batch/s]

Training batches on cuda:  38%|███▊      | 276/720 [00:01<00:02, 157.15batch/s]

Training batches on cuda:  41%|████      | 292/720 [00:01<00:02, 152.43batch/s]

Training batches on cuda:  43%|████▎     | 308/720 [00:01<00:02, 147.50batch/s]

Training batches on cuda:  45%|████▌     | 325/720 [00:02<00:02, 152.19batch/s]

Training batches on cuda:  48%|████▊     | 342/720 [00:02<00:02, 155.69batch/s]

Training batches on cuda:  50%|████▉     | 358/720 [00:02<00:02, 156.12batch/s]

Training batches on cuda:  52%|█████▏    | 375/720 [00:02<00:02, 157.68batch/s]

Training batches on cuda:  54%|█████▍    | 391/720 [00:02<00:02, 153.50batch/s]

Training batches on cuda:  57%|█████▋    | 407/720 [00:02<00:02, 152.21batch/s]

Training batches on cuda:  59%|█████▉    | 423/720 [00:02<00:01, 153.76batch/s]

Training batches on cuda:  61%|██████    | 439/720 [00:02<00:01, 145.66batch/s]

Training batches on cuda:  63%|██████▎   | 454/720 [00:02<00:01, 140.07batch/s]

Training batches on cuda:  65%|██████▌   | 469/720 [00:03<00:01, 137.53batch/s]

Training batches on cuda:  67%|██████▋   | 483/720 [00:03<00:01, 136.77batch/s]

Training batches on cuda:  69%|██████▉   | 498/720 [00:03<00:01, 139.35batch/s]

Training batches on cuda:  72%|███████▏  | 515/720 [00:03<00:01, 145.87batch/s]

Training batches on cuda:  74%|███████▎  | 530/720 [00:03<00:01, 142.52batch/s]

Training batches on cuda:  76%|███████▌  | 546/720 [00:03<00:01, 146.42batch/s]

Training batches on cuda:  78%|███████▊  | 562/720 [00:03<00:01, 148.62batch/s]

Training batches on cuda:  80%|████████  | 578/720 [00:03<00:00, 150.41batch/s]

Training batches on cuda:  82%|████████▎ | 594/720 [00:03<00:00, 150.87batch/s]

Training batches on cuda:  85%|████████▍ | 610/720 [00:04<00:00, 151.54batch/s]

Training batches on cuda:  87%|████████▋ | 628/720 [00:04<00:00, 157.22batch/s]

Training batches on cuda:  90%|████████▉ | 646/720 [00:04<00:00, 162.31batch/s]

Training batches on cuda:  92%|█████████▏| 663/720 [00:04<00:00, 157.27batch/s]

Training batches on cuda:  94%|█████████▍| 679/720 [00:04<00:00, 153.63batch/s]

Training batches on cuda:  97%|█████████▋| 695/720 [00:04<00:00, 151.45batch/s]

Training batches on cuda:  99%|█████████▉| 711/720 [00:04<00:00, 147.97batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:26,  4.43s/epoch, loss=0.00491, prev_loss=0.00492]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:22,  4.58s/epoch, loss=0.00491, prev_loss=0.00492]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 11/720 [00:00<00:06, 109.42batch/s]

Training batches on cuda:   4%|▎         | 26/720 [00:00<00:05, 116.86batch/s]

Training batches on cuda:   5%|▌         | 39/720 [00:00<00:05, 118.85batch/s]

Training batches on cuda:   7%|▋         | 51/720 [00:00<00:05, 117.59batch/s]

Training batches on cuda:   9%|▉         | 64/720 [00:00<00:05, 119.08batch/s]

Training batches on cuda:  11%|█         | 78/720 [00:00<00:05, 124.23batch/s]

Training batches on cuda:  13%|█▎        | 93/720 [00:00<00:04, 128.94batch/s]

Training batches on cuda:  15%|█▍        | 107/720 [00:00<00:04, 130.94batch/s]

Training batches on cuda:  17%|█▋        | 121/720 [00:00<00:04, 132.37batch/s]

Training batches on cuda:  19%|█▉        | 137/720 [00:01<00:04, 137.74batch/s]

Training batches on cuda:  21%|██▏       | 153/720 [00:01<00:03, 142.74batch/s]

Training batches on cuda:  23%|██▎       | 168/720 [00:01<00:03, 142.74batch/s]

Training batches on cuda:  26%|██▌       | 185/720 [00:01<00:03, 148.41batch/s]

Training batches on cuda:  28%|██▊       | 202/720 [00:01<00:03, 153.42batch/s]

Training batches on cuda:  31%|███       | 220/720 [00:01<00:03, 159.91batch/s]

Training batches on cuda:  33%|███▎      | 237/720 [00:01<00:03, 160.48batch/s]

Training batches on cuda:  35%|███▌      | 254/720 [00:01<00:02, 161.19batch/s]

Training batches on cuda:  38%|███▊      | 271/720 [00:01<00:02, 159.17batch/s]

Training batches on cuda:  40%|███▉      | 287/720 [00:01<00:02, 157.11batch/s]

Training batches on cuda:  42%|████▏     | 303/720 [00:02<00:02, 156.55batch/s]

Training batches on cuda:  45%|████▍     | 321/720 [00:02<00:02, 161.03batch/s]

Training batches on cuda:  47%|████▋     | 338/720 [00:02<00:02, 159.76batch/s]

Training batches on cuda:  49%|████▉     | 355/720 [00:02<00:02, 159.71batch/s]

Training batches on cuda:  52%|█████▏    | 373/720 [00:02<00:02, 163.42batch/s]

Training batches on cuda:  54%|█████▍    | 391/720 [00:02<00:01, 167.58batch/s]

Training batches on cuda:  57%|█████▋    | 410/720 [00:02<00:01, 172.76batch/s]

Training batches on cuda:  60%|█████▉    | 429/720 [00:02<00:01, 175.11batch/s]

Training batches on cuda:  62%|██████▏   | 448/720 [00:02<00:01, 176.88batch/s]

Training batches on cuda:  65%|██████▍   | 466/720 [00:03<00:01, 177.59batch/s]

Training batches on cuda:  67%|██████▋   | 485/720 [00:03<00:01, 179.41batch/s]

Training batches on cuda:  70%|██████▉   | 503/720 [00:03<00:01, 178.94batch/s]

Training batches on cuda:  72%|███████▏  | 521/720 [00:03<00:01, 178.01batch/s]

Training batches on cuda:  75%|███████▍  | 539/720 [00:03<00:01, 174.80batch/s]

Training batches on cuda:  77%|███████▋  | 557/720 [00:03<00:00, 173.28batch/s]

Training batches on cuda:  80%|███████▉  | 575/720 [00:03<00:00, 168.58batch/s]

Training batches on cuda:  82%|████████▏ | 593/720 [00:03<00:00, 169.51batch/s]

Training batches on cuda:  85%|████████▍ | 610/720 [00:03<00:00, 162.42batch/s]

Training batches on cuda:  87%|████████▋ | 628/720 [00:03<00:00, 166.93batch/s]

Training batches on cuda:  90%|████████▉ | 645/720 [00:04<00:00, 164.50batch/s]

Training batches on cuda:  92%|█████████▏| 662/720 [00:04<00:00, 160.18batch/s]

Training batches on cuda:  94%|█████████▍| 679/720 [00:04<00:00, 161.52batch/s]

Training batches on cuda:  97%|█████████▋| 696/720 [00:04<00:00, 163.51batch/s]

Training batches on cuda:  99%|█████████▉| 713/720 [00:04<00:00, 162.17batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:27<00:22,  4.58s/epoch, loss=0.00488, prev_loss=0.00491]
Training epochs on cuda:  60%|██████    | 6/10 [00:27<00:18,  4.63s/epoch, loss=0.00488, prev_loss=0.00491]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 8/720 [00:00<00:09, 77.34batch/s]

Training batches on cuda:   3%|▎         | 21/720 [00:00<00:08, 86.58batch/s]

Training batches on cuda:   5%|▌         | 38/720 [00:00<00:06, 100.43batch/s]

Training batches on cuda:   8%|▊         | 55/720 [00:00<00:05, 113.87batch/s]

Training batches on cuda:  10%|█         | 73/720 [00:00<00:05, 127.78batch/s]

Training batches on cuda:  12%|█▎        | 90/720 [00:00<00:04, 136.60batch/s]

Training batches on cuda:  15%|█▍        | 106/720 [00:00<00:04, 140.90batch/s]

Training batches on cuda:  17%|█▋        | 124/720 [00:00<00:03, 150.25batch/s]

Training batches on cuda:  20%|█▉        | 142/720 [00:00<00:03, 157.06batch/s]

Training batches on cuda:  22%|██▎       | 162/720 [00:01<00:03, 167.83batch/s]

Training batches on cuda:  25%|██▌       | 182/720 [00:01<00:03, 176.19batch/s]

Training batches on cuda:  28%|██▊       | 203/720 [00:01<00:02, 183.46batch/s]

Training batches on cuda:  31%|███       | 224/720 [00:01<00:02, 188.34batch/s]

Training batches on cuda:  34%|███▍      | 245/720 [00:01<00:02, 191.91batch/s]

Training batches on cuda:  37%|███▋      | 266/720 [00:01<00:02, 194.74batch/s]

Training batches on cuda:  40%|███▉      | 287/720 [00:01<00:02, 198.55batch/s]

Training batches on cuda:  43%|████▎     | 308/720 [00:01<00:02, 200.73batch/s]

Training batches on cuda:  46%|████▌     | 329/720 [00:01<00:01, 202.32batch/s]

Training batches on cuda:  49%|████▊     | 350/720 [00:01<00:01, 203.81batch/s]

Training batches on cuda:  52%|█████▏    | 371/720 [00:02<00:01, 204.42batch/s]

Training batches on cuda:  54%|█████▍    | 392/720 [00:02<00:01, 196.38batch/s]

Training batches on cuda:  57%|█████▋    | 412/720 [00:02<00:01, 167.19batch/s]

Training batches on cuda:  60%|█████▉    | 430/720 [00:02<00:01, 166.57batch/s]

Training batches on cuda:  62%|██████▏   | 448/720 [00:02<00:01, 168.61batch/s]

Training batches on cuda:  65%|██████▍   | 466/720 [00:02<00:01, 168.91batch/s]

Training batches on cuda:  67%|██████▋   | 485/720 [00:02<00:01, 173.49batch/s]

Training batches on cuda:  70%|███████   | 504/720 [00:02<00:01, 176.85batch/s]

Training batches on cuda:  72%|███████▎  | 522/720 [00:02<00:01, 174.72batch/s]

Training batches on cuda:  75%|███████▌  | 540/720 [00:03<00:01, 165.97batch/s]

Training batches on cuda:  77%|███████▋  | 557/720 [00:03<00:01, 157.53batch/s]

Training batches on cuda:  80%|███████▉  | 573/720 [00:03<00:00, 154.61batch/s]

Training batches on cuda:  82%|████████▏ | 589/720 [00:03<00:00, 152.20batch/s]

Training batches on cuda:  84%|████████▍ | 605/720 [00:03<00:00, 150.40batch/s]

Training batches on cuda:  86%|████████▋ | 621/720 [00:03<00:00, 149.58batch/s]

Training batches on cuda:  89%|████████▊ | 638/720 [00:03<00:00, 154.16batch/s]

Training batches on cuda:  91%|█████████ | 654/720 [00:03<00:00, 154.33batch/s]

Training batches on cuda:  93%|█████████▎| 671/720 [00:03<00:00, 158.64batch/s]

Training batches on cuda:  96%|█████████▌| 689/720 [00:04<00:00, 162.03batch/s]

Training batches on cuda:  98%|█████████▊| 706/720 [00:04<00:00, 162.18batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:31<00:18,  4.63s/epoch, loss=0.00483, prev_loss=0.00488]
Training epochs on cuda:  70%|███████   | 7/10 [00:31<00:13,  4.59s/epoch, loss=0.00483, prev_loss=0.00488]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 8/720 [00:00<00:09, 78.21batch/s]

Training batches on cuda:   4%|▎         | 26/720 [00:00<00:07, 93.95batch/s]

Training batches on cuda:   6%|▋         | 45/720 [00:00<00:06, 110.19batch/s]

Training batches on cuda:   9%|▉         | 63/720 [00:00<00:05, 124.47batch/s]

Training batches on cuda:  11%|█▏        | 81/720 [00:00<00:04, 135.81batch/s]

Training batches on cuda:  13%|█▎        | 97/720 [00:00<00:04, 140.99batch/s]

Training batches on cuda:  16%|█▌        | 114/720 [00:00<00:04, 147.52batch/s]

Training batches on cuda:  18%|█▊        | 132/720 [00:00<00:03, 155.36batch/s]

Training batches on cuda:  21%|██        | 150/720 [00:00<00:03, 161.59batch/s]

Training batches on cuda:  23%|██▎       | 168/720 [00:01<00:03, 165.16batch/s]

Training batches on cuda:  26%|██▌       | 186/720 [00:01<00:03, 169.31batch/s]

Training batches on cuda:  28%|██▊       | 205/720 [00:01<00:02, 173.21batch/s]

Training batches on cuda:  31%|███       | 223/720 [00:01<00:02, 174.89batch/s]

Training batches on cuda:  33%|███▎      | 241/720 [00:01<00:02, 175.57batch/s]

Training batches on cuda:  36%|███▌      | 259/720 [00:01<00:02, 176.10batch/s]

Training batches on cuda:  38%|███▊      | 277/720 [00:01<00:02, 173.43batch/s]

Training batches on cuda:  41%|████      | 295/720 [00:01<00:02, 166.77batch/s]

Training batches on cuda:  43%|████▎     | 313/720 [00:01<00:02, 170.37batch/s]

Training batches on cuda:  46%|████▌     | 332/720 [00:01<00:02, 173.72batch/s]

Training batches on cuda:  49%|████▊     | 350/720 [00:02<00:02, 171.92batch/s]

Training batches on cuda:  51%|█████▏    | 369/720 [00:02<00:02, 174.35batch/s]

Training batches on cuda:  54%|█████▍    | 387/720 [00:02<00:01, 174.16batch/s]

Training batches on cuda:  56%|█████▋    | 405/720 [00:02<00:01, 173.33batch/s]

Training batches on cuda:  59%|█████▉    | 423/720 [00:02<00:01, 174.49batch/s]

Training batches on cuda:  61%|██████▏   | 441/720 [00:02<00:01, 174.18batch/s]

Training batches on cuda:  64%|██████▍   | 459/720 [00:02<00:01, 173.73batch/s]

Training batches on cuda:  66%|██████▋   | 477/720 [00:02<00:01, 145.34batch/s]

Training batches on cuda:  68%|██████▊   | 493/720 [00:02<00:01, 148.01batch/s]

Training batches on cuda:  71%|███████   | 510/720 [00:03<00:01, 151.98batch/s]

Training batches on cuda:  73%|███████▎  | 527/720 [00:03<00:01, 154.11batch/s]

Training batches on cuda:  76%|███████▌  | 544/720 [00:03<00:01, 156.81batch/s]

Training batches on cuda:  78%|███████▊  | 561/720 [00:03<00:00, 159.08batch/s]

Training batches on cuda:  80%|████████  | 578/720 [00:03<00:00, 156.72batch/s]

Training batches on cuda:  83%|████████▎ | 596/720 [00:03<00:00, 161.58batch/s]

Training batches on cuda:  85%|████████▌ | 613/720 [00:03<00:00, 160.13batch/s]

Training batches on cuda:  88%|████████▊ | 630/720 [00:03<00:00, 159.40batch/s]

Training batches on cuda:  90%|████████▉ | 647/720 [00:03<00:00, 158.95batch/s]

Training batches on cuda:  92%|█████████▏| 663/720 [00:04<00:00, 158.64batch/s]

Training batches on cuda:  95%|█████████▍| 681/720 [00:04<00:00, 162.85batch/s]

Training batches on cuda:  97%|█████████▋| 699/720 [00:04<00:00, 167.33batch/s]

Training batches on cuda: 100%|█████████▉| 717/720 [00:04<00:00, 170.48batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:36<00:13,  4.59s/epoch, loss=0.00472, prev_loss=0.00483]
Training epochs on cuda:  80%|████████  | 8/10 [00:36<00:09,  4.60s/epoch, loss=0.00472, prev_loss=0.00483]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 13/720 [00:00<00:05, 127.61batch/s]

Training batches on cuda:   4%|▍         | 31/720 [00:00<00:04, 139.73batch/s]

Training batches on cuda:   7%|▋         | 49/720 [00:00<00:04, 149.48batch/s]

Training batches on cuda:   9%|▉         | 68/720 [00:00<00:04, 156.43batch/s]

Training batches on cuda:  12%|█▏        | 84/720 [00:00<00:04, 156.37batch/s]

Training batches on cuda:  14%|█▍        | 101/720 [00:00<00:03, 157.72batch/s]

Training batches on cuda:  16%|█▋        | 118/720 [00:00<00:03, 161.20batch/s]

Training batches on cuda:  19%|█▉        | 138/720 [00:00<00:03, 169.30batch/s]

Training batches on cuda:  22%|██▏       | 157/720 [00:00<00:03, 172.91batch/s]

Training batches on cuda:  25%|██▍       | 177/720 [00:01<00:03, 178.71batch/s]

Training batches on cuda:  27%|██▋       | 195/720 [00:01<00:03, 174.85batch/s]

Training batches on cuda:  30%|██▉       | 213/720 [00:01<00:02, 173.11batch/s]

Training batches on cuda:  32%|███▏      | 231/720 [00:01<00:02, 171.73batch/s]

Training batches on cuda:  35%|███▍      | 249/720 [00:01<00:02, 166.43batch/s]

Training batches on cuda:  37%|███▋      | 266/720 [00:01<00:02, 164.92batch/s]

Training batches on cuda:  39%|███▉      | 283/720 [00:01<00:02, 159.64batch/s]

Training batches on cuda:  42%|████▏     | 301/720 [00:01<00:02, 162.87batch/s]

Training batches on cuda:  44%|████▍     | 319/720 [00:01<00:02, 167.52batch/s]

Training batches on cuda:  47%|████▋     | 337/720 [00:01<00:02, 168.76batch/s]

Training batches on cuda:  49%|████▉     | 354/720 [00:02<00:02, 167.12batch/s]

Training batches on cuda:  52%|█████▏    | 371/720 [00:02<00:02, 160.12batch/s]

Training batches on cuda:  54%|█████▍    | 388/720 [00:02<00:02, 158.48batch/s]

Training batches on cuda:  56%|█████▌    | 404/720 [00:02<00:02, 155.81batch/s]

Training batches on cuda:  58%|█████▊    | 420/720 [00:02<00:02, 149.98batch/s]

Training batches on cuda:  61%|██████    | 436/720 [00:02<00:01, 149.36batch/s]

Training batches on cuda:  63%|██████▎   | 453/720 [00:02<00:01, 154.40batch/s]

Training batches on cuda:  65%|██████▌   | 470/720 [00:02<00:01, 156.34batch/s]

Training batches on cuda:  68%|██████▊   | 490/720 [00:02<00:01, 164.95batch/s]

Training batches on cuda:  71%|███████   | 509/720 [00:03<00:01, 169.99batch/s]

Training batches on cuda:  73%|███████▎  | 527/720 [00:03<00:01, 166.75batch/s]

Training batches on cuda:  76%|███████▌  | 544/720 [00:03<00:01, 164.67batch/s]

Training batches on cuda:  78%|███████▊  | 561/720 [00:03<00:00, 162.85batch/s]

Training batches on cuda:  80%|████████  | 578/720 [00:03<00:00, 163.85batch/s]

Training batches on cuda:  83%|████████▎ | 595/720 [00:03<00:00, 163.50batch/s]

Training batches on cuda:  85%|████████▌ | 612/720 [00:03<00:00, 164.77batch/s]

Training batches on cuda:  87%|████████▋ | 629/720 [00:03<00:00, 164.25batch/s]

Training batches on cuda:  90%|████████▉ | 646/720 [00:03<00:00, 165.09batch/s]

Training batches on cuda:  92%|█████████▏| 663/720 [00:04<00:00, 161.51batch/s]

Training batches on cuda:  94%|█████████▍| 680/720 [00:04<00:00, 163.11batch/s]

Training batches on cuda:  97%|█████████▋| 697/720 [00:04<00:00, 161.81batch/s]

Training batches on cuda:  99%|█████████▉| 714/720 [00:04<00:00, 164.09batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:41<00:09,  4.60s/epoch, loss=0.00464, prev_loss=0.00472]
Training epochs on cuda:  90%|█████████ | 9/10 [00:41<00:04,  4.59s/epoch, loss=0.00464, prev_loss=0.00472]

Training batches on cuda:   0%|          | 0/720 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 12/720 [00:00<00:06, 114.56batch/s]

Training batches on cuda:   4%|▍         | 29/720 [00:00<00:05, 126.37batch/s]

Training batches on cuda:   6%|▋         | 46/720 [00:00<00:04, 136.79batch/s]

Training batches on cuda:   9%|▉         | 63/720 [00:00<00:04, 145.07batch/s]

Training batches on cuda:  11%|█         | 80/720 [00:00<00:04, 151.20batch/s]

Training batches on cuda:  13%|█▎        | 96/720 [00:00<00:04, 151.98batch/s]

Training batches on cuda:  16%|█▌        | 114/720 [00:00<00:03, 157.19batch/s]

Training batches on cuda:  18%|█▊        | 132/720 [00:00<00:03, 161.33batch/s]

Training batches on cuda:  21%|██        | 149/720 [00:00<00:03, 163.15batch/s]

Training batches on cuda:  23%|██▎       | 166/720 [00:01<00:03, 164.99batch/s]

Training batches on cuda:  25%|██▌       | 183/720 [00:01<00:03, 164.29batch/s]

Training batches on cuda:  28%|██▊       | 201/720 [00:01<00:03, 166.50batch/s]

Training batches on cuda:  30%|███       | 218/720 [00:01<00:03, 166.41batch/s]

Training batches on cuda:  33%|███▎      | 235/720 [00:01<00:02, 162.94batch/s]

Training batches on cuda:  35%|███▌      | 253/720 [00:01<00:02, 166.08batch/s]

Training batches on cuda:  38%|███▊      | 272/720 [00:01<00:02, 171.65batch/s]

Training batches on cuda:  41%|████      | 292/720 [00:01<00:02, 178.80batch/s]

Training batches on cuda:  43%|████▎     | 313/720 [00:01<00:02, 185.77batch/s]

Training batches on cuda:  46%|████▋     | 333/720 [00:01<00:02, 188.91batch/s]

Training batches on cuda:  49%|████▉     | 354/720 [00:02<00:01, 194.20batch/s]

Training batches on cuda:  52%|█████▏    | 374/720 [00:02<00:01, 195.28batch/s]

Training batches on cuda:  55%|█████▍    | 395/720 [00:02<00:01, 197.48batch/s]

Training batches on cuda:  58%|█████▊    | 415/720 [00:02<00:01, 196.67batch/s]

Training batches on cuda:  61%|██████    | 436/720 [00:02<00:01, 198.89batch/s]

Training batches on cuda:  63%|██████▎   | 456/720 [00:02<00:01, 195.66batch/s]

Training batches on cuda:  66%|██████▌   | 476/720 [00:02<00:01, 192.93batch/s]

Training batches on cuda:  69%|██████▉   | 496/720 [00:02<00:01, 191.13batch/s]

Training batches on cuda:  72%|███████▏  | 516/720 [00:02<00:01, 190.43batch/s]

Training batches on cuda:  74%|███████▍  | 536/720 [00:03<00:00, 186.03batch/s]

Training batches on cuda:  77%|███████▋  | 555/720 [00:03<00:00, 184.69batch/s]

Training batches on cuda:  80%|███████▉  | 574/720 [00:03<00:00, 182.30batch/s]

Training batches on cuda:  82%|████████▏ | 593/720 [00:03<00:00, 174.42batch/s]

Training batches on cuda:  85%|████████▌ | 612/720 [00:03<00:00, 177.86batch/s]

Training batches on cuda:  88%|████████▊ | 632/720 [00:03<00:00, 182.73batch/s]

Training batches on cuda:  90%|█████████ | 651/720 [00:03<00:00, 183.91batch/s]

Training batches on cuda:  93%|█████████▎| 670/720 [00:03<00:00, 185.51batch/s]

Training batches on cuda:  96%|█████████▌| 689/720 [00:03<00:00, 186.75batch/s]

Training batches on cuda:  98%|█████████▊| 708/720 [00:03<00:00, 187.62batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:45<00:04,  4.59s/epoch, loss=0.00459, prev_loss=0.00464]2021-09-16:17:59:14,350 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:17:59:15,489 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:17:59:30,475 INFO     [evaluator.py:657] Evaluation took 14.99s seconds
2021-09-16:17:59:30,607 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  9.36s/epoch, loss=0.00459, prev_loss=0.00464]
Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  6.17s/epoch, loss=0.00459, prev_loss=0.00464]
2021-09-16:17:59:30,608 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpuhrjpz00'
2021-09-16:17:59:30,673 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpuhrjpz00' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.58ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.60ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:17:59:45,474 INFO     [evaluator.py:657] Evaluation took 14.79s seconds
[I 2021-09-16 17:59:45,479] Trial 23 finished with value: 0.2776876790387671 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.03506062431775608, 'optimizer.lr': 0.00913279261305711, 'negative_sampler.num_negs_per_pos': 6, 'training.batch_size': 432}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:17:59:45,499 WARNING  [api.py:823] No random seed is specified. Setting to 3297058733.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 6/253 [00:00<00:04, 56.39batch/s]

Training batches on cuda:   6%|▌         | 14/253 [00:00<00:03, 61.41batch/s]

Training batches on cuda:   9%|▊         | 22/253 [00:00<00:03, 66.01batch/s]

Training batches on cuda:  12%|█▏        | 31/253 [00:00<00:03, 71.36batch/s]

Training batches on cuda:  16%|█▌        | 41/253 [00:00<00:02, 76.46batch/s]

Training batches on cuda:  20%|██        | 51/253 [00:00<00:02, 82.03batch/s]

Training batches on cuda:  24%|██▍       | 61/253 [00:00<00:02, 86.48batch/s]

Training batches on cuda:  28%|██▊       | 72/253 [00:00<00:02, 90.24batch/s]

Training batches on cuda:  32%|███▏      | 82/253 [00:00<00:01, 92.85batch/s]

Training batches on cuda:  36%|███▋      | 92/253 [00:01<00:01, 94.00batch/s]

Training batches on cuda:  40%|████      | 102/253 [00:01<00:01, 95.27batch/s]

Training batches on cuda:  45%|████▍     | 113/253 [00:01<00:01, 96.83batch/s]

Training batches on cuda:  49%|████▊     | 123/253 [00:01<00:01, 97.67batch/s]

Training batches on cuda:  53%|█████▎    | 133/253 [00:01<00:01, 98.35batch/s]

Training batches on cuda:  57%|█████▋    | 143/253 [00:01<00:01, 97.93batch/s]

Training batches on cuda:  61%|██████    | 154/253 [00:01<00:00, 99.07batch/s]

Training batches on cuda:  65%|██████▌   | 165/253 [00:01<00:00, 99.59batch/s]

Training batches on cuda:  70%|██████▉   | 176/253 [00:01<00:00, 100.36batch/s]

Training batches on cuda:  74%|███████▍  | 187/253 [00:01<00:00, 100.99batch/s]

Training batches on cuda:  78%|███████▊  | 198/253 [00:02<00:00, 100.47batch/s]

Training batches on cuda:  83%|████████▎ | 209/253 [00:02<00:00, 100.59batch/s]

Training batches on cuda:  87%|████████▋ | 220/253 [00:02<00:00, 100.23batch/s]

Training batches on cuda:  91%|█████████▏| 231/253 [00:02<00:00, 100.07batch/s]

Training batches on cuda:  96%|█████████▌| 242/253 [00:02<00:00, 99.87batch/s] 

Training batches on cuda: 100%|██████████| 253/253 [00:02<00:00, 100.83batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.00108, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:25,  2.81s/epoch, loss=0.00108, prev_loss=nan]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 8/253 [00:00<00:03, 76.28batch/s]

Training batches on cuda:   7%|▋         | 18/253 [00:00<00:02, 81.79batch/s]

Training batches on cuda:  11%|█         | 28/253 [00:00<00:02, 85.86batch/s]

Training batches on cuda:  15%|█▌        | 38/253 [00:00<00:02, 89.29batch/s]

Training batches on cuda:  19%|█▉        | 48/253 [00:00<00:02, 91.66batch/s]

Training batches on cuda:  23%|██▎       | 59/253 [00:00<00:02, 94.03batch/s]

Training batches on cuda:  27%|██▋       | 69/253 [00:00<00:01, 95.68batch/s]

Training batches on cuda:  31%|███       | 79/253 [00:00<00:01, 96.30batch/s]

Training batches on cuda:  36%|███▌      | 90/253 [00:00<00:01, 97.46batch/s]

Training batches on cuda:  40%|███▉      | 100/253 [00:01<00:01, 93.50batch/s]

Training batches on cuda:  43%|████▎     | 110/253 [00:01<00:01, 91.44batch/s]

Training batches on cuda:  47%|████▋     | 120/253 [00:01<00:01, 90.43batch/s]

Training batches on cuda:  51%|█████     | 129/253 [00:01<00:01, 89.24batch/s]

Training batches on cuda:  55%|█████▍    | 138/253 [00:01<00:01, 86.24batch/s]

Training batches on cuda:  58%|█████▊    | 147/253 [00:01<00:01, 81.78batch/s]

Training batches on cuda:  62%|██████▏   | 156/253 [00:01<00:01, 80.67batch/s]

Training batches on cuda:  65%|██████▌   | 165/253 [00:01<00:01, 82.85batch/s]

Training batches on cuda:  69%|██████▉   | 175/253 [00:01<00:00, 85.50batch/s]

Training batches on cuda:  73%|███████▎  | 184/253 [00:02<00:00, 85.09batch/s]

Training batches on cuda:  76%|███████▋  | 193/253 [00:02<00:00, 86.30batch/s]

Training batches on cuda:  80%|███████▉  | 202/253 [00:02<00:00, 87.12batch/s]

Training batches on cuda:  83%|████████▎ | 211/253 [00:02<00:00, 87.10batch/s]

Training batches on cuda:  87%|████████▋ | 220/253 [00:02<00:00, 86.24batch/s]

Training batches on cuda:  91%|█████████ | 230/253 [00:02<00:00, 87.69batch/s]

Training batches on cuda:  94%|█████████▍| 239/253 [00:02<00:00, 88.33batch/s]

Training batches on cuda:  98%|█████████▊| 248/253 [00:02<00:00, 88.61batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:25,  2.81s/epoch, loss=0.00107, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:23,  2.90s/epoch, loss=0.00107, prev_loss=nan]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/253 [00:00<00:03, 66.47batch/s]

Training batches on cuda:   6%|▋         | 16/253 [00:00<00:03, 71.01batch/s]

Training batches on cuda:  10%|▉         | 25/253 [00:00<00:03, 75.31batch/s]

Training batches on cuda:  13%|█▎        | 34/253 [00:00<00:02, 77.17batch/s]

Training batches on cuda:  17%|█▋        | 43/253 [00:00<00:02, 78.69batch/s]

Training batches on cuda:  21%|██        | 52/253 [00:00<00:02, 80.50batch/s]

Training batches on cuda:  24%|██▍       | 61/253 [00:00<00:02, 81.22batch/s]

Training batches on cuda:  28%|██▊       | 70/253 [00:00<00:02, 82.26batch/s]

Training batches on cuda:  32%|███▏      | 80/253 [00:00<00:02, 84.55batch/s]

Training batches on cuda:  35%|███▌      | 89/253 [00:01<00:01, 85.75batch/s]

Training batches on cuda:  39%|███▊      | 98/253 [00:01<00:01, 83.80batch/s]

Training batches on cuda:  42%|████▏     | 107/253 [00:01<00:01, 83.57batch/s]

Training batches on cuda:  46%|████▌     | 116/253 [00:01<00:01, 82.82batch/s]

Training batches on cuda:  49%|████▉     | 125/253 [00:01<00:01, 76.32batch/s]

Training batches on cuda:  53%|█████▎    | 133/253 [00:01<00:01, 75.19batch/s]

Training batches on cuda:  56%|█████▌    | 142/253 [00:01<00:01, 78.03batch/s]

Training batches on cuda:  59%|█████▉    | 150/253 [00:01<00:01, 76.91batch/s]

Training batches on cuda:  62%|██████▏   | 158/253 [00:01<00:01, 76.96batch/s]

Training batches on cuda:  66%|██████▌   | 167/253 [00:02<00:01, 78.15batch/s]

Training batches on cuda:  70%|██████▉   | 176/253 [00:02<00:00, 80.60batch/s]

Training batches on cuda:  74%|███████▎  | 186/253 [00:02<00:00, 83.38batch/s]

Training batches on cuda:  77%|███████▋  | 195/253 [00:02<00:00, 85.06batch/s]

Training batches on cuda:  81%|████████  | 204/253 [00:02<00:00, 85.97batch/s]

Training batches on cuda:  85%|████████▍ | 214/253 [00:02<00:00, 87.94batch/s]

Training batches on cuda:  88%|████████▊ | 223/253 [00:02<00:00, 88.41batch/s]

Training batches on cuda:  92%|█████████▏| 233/253 [00:02<00:00, 88.91batch/s]

Training batches on cuda:  96%|█████████▌| 243/253 [00:02<00:00, 89.83batch/s]

Training batches on cuda: 100%|██████████| 253/253 [00:03<00:00, 90.71batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:23,  2.90s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:20,  3.00s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/253 [00:00<00:05, 49.52batch/s]

Training batches on cuda:   6%|▌         | 15/253 [00:00<00:04, 57.31batch/s]

Training batches on cuda:   9%|▉         | 24/253 [00:00<00:03, 64.02batch/s]

Training batches on cuda:  12%|█▏        | 30/253 [00:00<00:03, 58.75batch/s]

Training batches on cuda:  15%|█▌        | 38/253 [00:00<00:03, 62.91batch/s]

Training batches on cuda:  19%|█▊        | 47/253 [00:00<00:03, 68.26batch/s]

Training batches on cuda:  23%|██▎       | 57/253 [00:00<00:02, 74.08batch/s]

Training batches on cuda:  26%|██▌       | 66/253 [00:00<00:02, 76.14batch/s]

Training batches on cuda:  30%|██▉       | 75/253 [00:00<00:02, 77.74batch/s]

Training batches on cuda:  34%|███▎      | 85/253 [00:01<00:02, 80.42batch/s]

Training batches on cuda:  37%|███▋      | 94/253 [00:01<00:02, 77.43batch/s]

Training batches on cuda:  41%|████      | 103/253 [00:01<00:01, 78.53batch/s]

Training batches on cuda:  44%|████▍     | 111/253 [00:01<00:01, 76.18batch/s]

Training batches on cuda:  47%|████▋     | 119/253 [00:01<00:01, 74.42batch/s]

Training batches on cuda:  51%|█████     | 128/253 [00:01<00:01, 76.37batch/s]

Training batches on cuda:  54%|█████▍    | 137/253 [00:01<00:01, 79.57batch/s]

Training batches on cuda:  58%|█████▊    | 146/253 [00:01<00:01, 81.39batch/s]

Training batches on cuda:  61%|██████▏   | 155/253 [00:01<00:01, 83.28batch/s]

Training batches on cuda:  65%|██████▍   | 164/253 [00:02<00:01, 82.28batch/s]

Training batches on cuda:  68%|██████▊   | 173/253 [00:02<00:00, 81.50batch/s]

Training batches on cuda:  72%|███████▏  | 182/253 [00:02<00:00, 81.31batch/s]

Training batches on cuda:  75%|███████▌  | 191/253 [00:02<00:00, 79.83batch/s]

Training batches on cuda:  79%|███████▉  | 200/253 [00:02<00:00, 78.45batch/s]

Training batches on cuda:  83%|████████▎ | 209/253 [00:02<00:00, 80.55batch/s]

Training batches on cuda:  86%|████████▌ | 218/253 [00:02<00:00, 82.32batch/s]

Training batches on cuda:  90%|█████████ | 228/253 [00:02<00:00, 85.04batch/s]

Training batches on cuda:  94%|█████████▎| 237/253 [00:02<00:00, 86.13batch/s]

Training batches on cuda:  97%|█████████▋| 246/253 [00:03<00:00, 79.21batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:20,  3.00s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:18,  3.11s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 6/253 [00:00<00:04, 59.15batch/s]

Training batches on cuda:   6%|▌         | 14/253 [00:00<00:03, 63.17batch/s]

Training batches on cuda:   9%|▊         | 22/253 [00:00<00:03, 67.09batch/s]

Training batches on cuda:  12%|█▏        | 30/253 [00:00<00:03, 69.80batch/s]

Training batches on cuda:  14%|█▍        | 36/253 [00:00<00:03, 65.11batch/s]

Training batches on cuda:  18%|█▊        | 45/253 [00:00<00:03, 69.18batch/s]

Training batches on cuda:  21%|██        | 53/253 [00:00<00:02, 70.78batch/s]

Training batches on cuda:  24%|██▍       | 61/253 [00:00<00:02, 72.55batch/s]

Training batches on cuda:  28%|██▊       | 70/253 [00:00<00:02, 74.90batch/s]

Training batches on cuda:  31%|███       | 78/253 [00:01<00:02, 74.51batch/s]

Training batches on cuda:  34%|███▍      | 86/253 [00:01<00:02, 75.99batch/s]

Training batches on cuda:  38%|███▊      | 95/253 [00:01<00:02, 78.08batch/s]

Training batches on cuda:  41%|████      | 103/253 [00:01<00:01, 76.87batch/s]

Training batches on cuda:  44%|████▍     | 111/253 [00:01<00:01, 77.45batch/s]

Training batches on cuda:  47%|████▋     | 119/253 [00:01<00:01, 77.36batch/s]

Training batches on cuda:  50%|█████     | 127/253 [00:01<00:01, 77.13batch/s]

Training batches on cuda:  54%|█████▍    | 136/253 [00:01<00:01, 78.84batch/s]

Training batches on cuda:  57%|█████▋    | 145/253 [00:01<00:01, 79.87batch/s]

Training batches on cuda:  61%|██████    | 154/253 [00:02<00:01, 80.54batch/s]

Training batches on cuda:  64%|██████▍   | 163/253 [00:02<00:01, 79.93batch/s]

Training batches on cuda:  68%|██████▊   | 172/253 [00:02<00:01, 79.94batch/s]

Training batches on cuda:  71%|███████   | 180/253 [00:02<00:00, 79.77batch/s]

Training batches on cuda:  74%|███████▍  | 188/253 [00:02<00:00, 77.98batch/s]

Training batches on cuda:  78%|███████▊  | 197/253 [00:02<00:00, 79.42batch/s]

Training batches on cuda:  81%|████████▏ | 206/253 [00:02<00:00, 81.08batch/s]

Training batches on cuda:  85%|████████▍ | 215/253 [00:02<00:00, 82.32batch/s]

Training batches on cuda:  89%|████████▊ | 224/253 [00:02<00:00, 81.59batch/s]

Training batches on cuda:  92%|█████████▏| 233/253 [00:02<00:00, 80.46batch/s]

Training batches on cuda:  96%|█████████▌| 242/253 [00:03<00:00, 82.51batch/s]

Training batches on cuda:  99%|█████████▉| 251/253 [00:03<00:00, 84.26batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:18,  3.11s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:16,  3.22s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 6/253 [00:00<00:04, 57.75batch/s]

Training batches on cuda:   6%|▌         | 15/253 [00:00<00:03, 64.12batch/s]

Training batches on cuda:  10%|▉         | 25/253 [00:00<00:03, 69.55batch/s]

Training batches on cuda:  13%|█▎        | 32/253 [00:00<00:03, 67.93batch/s]

Training batches on cuda:  15%|█▌        | 39/253 [00:00<00:03, 67.29batch/s]

Training batches on cuda:  18%|█▊        | 46/253 [00:00<00:03, 65.71batch/s]

Training batches on cuda:  21%|██        | 53/253 [00:00<00:03, 65.08batch/s]

Training batches on cuda:  24%|██▍       | 61/253 [00:00<00:02, 66.98batch/s]

Training batches on cuda:  28%|██▊       | 70/253 [00:00<00:02, 71.36batch/s]

Training batches on cuda:  31%|███       | 78/253 [00:01<00:02, 73.57batch/s]

Training batches on cuda:  34%|███▍      | 87/253 [00:01<00:02, 75.57batch/s]

Training batches on cuda:  38%|███▊      | 96/253 [00:01<00:02, 78.00batch/s]

Training batches on cuda:  41%|████      | 104/253 [00:01<00:01, 75.77batch/s]

Training batches on cuda:  45%|████▍     | 113/253 [00:01<00:01, 77.91batch/s]

Training batches on cuda:  48%|████▊     | 122/253 [00:01<00:01, 80.59batch/s]

Training batches on cuda:  52%|█████▏    | 131/253 [00:01<00:01, 82.86batch/s]

Training batches on cuda:  55%|█████▌    | 140/253 [00:01<00:01, 84.87batch/s]

Training batches on cuda:  59%|█████▉    | 149/253 [00:01<00:01, 86.13batch/s]

Training batches on cuda:  63%|██████▎   | 159/253 [00:02<00:01, 87.25batch/s]

Training batches on cuda:  66%|██████▋   | 168/253 [00:02<00:00, 87.55batch/s]

Training batches on cuda:  70%|███████   | 178/253 [00:02<00:00, 89.09batch/s]

Training batches on cuda:  74%|███████▍  | 187/253 [00:02<00:00, 87.78batch/s]

Training batches on cuda:  77%|███████▋  | 196/253 [00:02<00:00, 87.53batch/s]

Training batches on cuda:  81%|████████  | 205/253 [00:02<00:00, 87.24batch/s]

Training batches on cuda:  85%|████████▍ | 214/253 [00:02<00:00, 86.62batch/s]

Training batches on cuda:  88%|████████▊ | 223/253 [00:02<00:00, 86.29batch/s]

Training batches on cuda:  92%|█████████▏| 232/253 [00:02<00:00, 86.61batch/s]

Training batches on cuda:  95%|█████████▌| 241/253 [00:02<00:00, 87.36batch/s]

Training batches on cuda:  99%|█████████▉| 250/253 [00:03<00:00, 86.49batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:19<00:16,  3.22s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  60%|██████    | 6/10 [00:19<00:12,  3.25s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 7/253 [00:00<00:03, 68.19batch/s]

Training batches on cuda:   6%|▌         | 15/253 [00:00<00:03, 69.39batch/s]

Training batches on cuda:   9%|▉         | 23/253 [00:00<00:03, 71.76batch/s]

Training batches on cuda:  12%|█▏        | 31/253 [00:00<00:03, 72.99batch/s]

Training batches on cuda:  15%|█▌        | 38/253 [00:00<00:03, 69.86batch/s]

Training batches on cuda:  18%|█▊        | 46/253 [00:00<00:02, 71.33batch/s]

Training batches on cuda:  22%|██▏       | 55/253 [00:00<00:02, 74.62batch/s]

Training batches on cuda:  25%|██▍       | 62/253 [00:00<00:02, 72.09batch/s]

Training batches on cuda:  28%|██▊       | 70/253 [00:00<00:02, 73.50batch/s]

Training batches on cuda:  31%|███       | 78/253 [00:01<00:02, 75.25batch/s]

Training batches on cuda:  34%|███▍      | 86/253 [00:01<00:02, 75.23batch/s]

Training batches on cuda:  38%|███▊      | 95/253 [00:01<00:02, 77.76batch/s]

Training batches on cuda:  41%|████      | 104/253 [00:01<00:01, 80.65batch/s]

Training batches on cuda:  45%|████▍     | 113/253 [00:01<00:01, 82.51batch/s]

Training batches on cuda:  48%|████▊     | 122/253 [00:01<00:01, 82.90batch/s]

Training batches on cuda:  52%|█████▏    | 131/253 [00:01<00:01, 82.65batch/s]

Training batches on cuda:  55%|█████▌    | 140/253 [00:01<00:01, 82.37batch/s]

Training batches on cuda:  59%|█████▉    | 149/253 [00:01<00:01, 81.78batch/s]

Training batches on cuda:  62%|██████▏   | 158/253 [00:02<00:01, 79.43batch/s]

Training batches on cuda:  66%|██████▌   | 166/253 [00:02<00:01, 78.61batch/s]

Training batches on cuda:  69%|██████▉   | 175/253 [00:02<00:00, 79.21batch/s]

Training batches on cuda:  72%|███████▏  | 183/253 [00:02<00:00, 75.44batch/s]

Training batches on cuda:  76%|███████▌  | 192/253 [00:02<00:00, 77.92batch/s]

Training batches on cuda:  79%|███████▉  | 200/253 [00:02<00:00, 77.21batch/s]

Training batches on cuda:  82%|████████▏ | 208/253 [00:02<00:00, 66.63batch/s]

Training batches on cuda:  86%|████████▌ | 217/253 [00:02<00:00, 71.09batch/s]

Training batches on cuda:  89%|████████▉ | 226/253 [00:02<00:00, 73.84batch/s]

Training batches on cuda:  92%|█████████▏| 234/253 [00:03<00:00, 71.07batch/s]

Training batches on cuda:  96%|█████████▌| 242/253 [00:03<00:00, 70.73batch/s]

Training batches on cuda:  99%|█████████▉| 251/253 [00:03<00:00, 73.92batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:22<00:12,  3.25s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  70%|███████   | 7/10 [00:22<00:09,  3.33s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/253 [00:00<00:05, 44.96batch/s]

Training batches on cuda:   5%|▍         | 12/253 [00:00<00:04, 49.51batch/s]

Training batches on cuda:   8%|▊         | 20/253 [00:00<00:04, 55.50batch/s]

Training batches on cuda:  12%|█▏        | 30/253 [00:00<00:03, 62.75batch/s]

Training batches on cuda:  15%|█▌        | 39/253 [00:00<00:03, 68.77batch/s]

Training batches on cuda:  19%|█▉        | 49/253 [00:00<00:02, 74.73batch/s]

Training batches on cuda:  23%|██▎       | 58/253 [00:00<00:02, 78.70batch/s]

Training batches on cuda:  26%|██▋       | 67/253 [00:00<00:02, 81.00batch/s]

Training batches on cuda:  30%|███       | 76/253 [00:00<00:02, 82.99batch/s]

Training batches on cuda:  34%|███▍      | 86/253 [00:01<00:01, 85.37batch/s]

Training batches on cuda:  38%|███▊      | 95/253 [00:01<00:01, 86.63batch/s]

Training batches on cuda:  42%|████▏     | 105/253 [00:01<00:01, 87.05batch/s]

Training batches on cuda:  45%|████▌     | 114/253 [00:01<00:01, 87.42batch/s]

Training batches on cuda:  49%|████▉     | 124/253 [00:01<00:01, 88.80batch/s]

Training batches on cuda:  53%|█████▎    | 133/253 [00:01<00:01, 88.59batch/s]

Training batches on cuda:  57%|█████▋    | 143/253 [00:01<00:01, 90.14batch/s]

Training batches on cuda:  60%|██████    | 153/253 [00:01<00:01, 89.24batch/s]

Training batches on cuda:  64%|██████▍   | 162/253 [00:01<00:01, 78.49batch/s]

Training batches on cuda:  68%|██████▊   | 172/253 [00:02<00:00, 82.12batch/s]

Training batches on cuda:  72%|███████▏  | 181/253 [00:02<00:00, 80.40batch/s]

Training batches on cuda:  75%|███████▌  | 190/253 [00:02<00:00, 80.94batch/s]

Training batches on cuda:  79%|███████▉  | 200/253 [00:02<00:00, 83.58batch/s]

Training batches on cuda:  83%|████████▎ | 210/253 [00:02<00:00, 85.41batch/s]

Training batches on cuda:  87%|████████▋ | 219/253 [00:02<00:00, 83.90batch/s]

Training batches on cuda:  90%|█████████ | 228/253 [00:02<00:00, 83.59batch/s]

Training batches on cuda:  94%|█████████▎| 237/253 [00:02<00:00, 81.42batch/s]

Training batches on cuda:  97%|█████████▋| 246/253 [00:02<00:00, 81.99batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:26<00:09,  3.33s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:06,  3.29s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/253 [00:00<00:08, 29.03batch/s]

Training batches on cuda:   4%|▍         | 11/253 [00:00<00:06, 35.47batch/s]

Training batches on cuda:   8%|▊         | 19/253 [00:00<00:05, 42.33batch/s]

Training batches on cuda:  11%|█▏        | 29/253 [00:00<00:04, 50.48batch/s]

Training batches on cuda:  15%|█▌        | 38/253 [00:00<00:03, 58.03batch/s]

Training batches on cuda:  19%|█▊        | 47/253 [00:00<00:03, 63.50batch/s]

Training batches on cuda:  22%|██▏       | 56/253 [00:00<00:02, 68.97batch/s]

Training batches on cuda:  25%|██▌       | 64/253 [00:00<00:02, 71.07batch/s]

Training batches on cuda:  28%|██▊       | 72/253 [00:00<00:02, 70.84batch/s]

Training batches on cuda:  32%|███▏      | 80/253 [00:01<00:02, 72.10batch/s]

Training batches on cuda:  35%|███▍      | 88/253 [00:01<00:02, 73.36batch/s]

Training batches on cuda:  38%|███▊      | 97/253 [00:01<00:02, 75.98batch/s]

Training batches on cuda:  42%|████▏     | 107/253 [00:01<00:01, 80.35batch/s]

Training batches on cuda:  46%|████▌     | 116/253 [00:01<00:01, 78.74batch/s]

Training batches on cuda:  49%|████▉     | 125/253 [00:01<00:01, 78.99batch/s]

Training batches on cuda:  53%|█████▎    | 133/253 [00:01<00:01, 76.36batch/s]

Training batches on cuda:  56%|█████▌    | 141/253 [00:01<00:01, 74.70batch/s]

Training batches on cuda:  59%|█████▉    | 150/253 [00:01<00:01, 76.64batch/s]

Training batches on cuda:  63%|██████▎   | 159/253 [00:02<00:01, 78.52batch/s]

Training batches on cuda:  66%|██████▌   | 167/253 [00:02<00:01, 75.22batch/s]

Training batches on cuda:  70%|██████▉   | 176/253 [00:02<00:01, 76.89batch/s]

Training batches on cuda:  73%|███████▎  | 185/253 [00:02<00:00, 79.26batch/s]

Training batches on cuda:  77%|███████▋  | 194/253 [00:02<00:00, 81.24batch/s]

Training batches on cuda:  80%|████████  | 203/253 [00:02<00:00, 82.11batch/s]

Training batches on cuda:  84%|████████▍ | 212/253 [00:02<00:00, 81.46batch/s]

Training batches on cuda:  87%|████████▋ | 221/253 [00:02<00:00, 81.58batch/s]

Training batches on cuda:  91%|█████████ | 230/253 [00:02<00:00, 80.46batch/s]

Training batches on cuda:  94%|█████████▍| 239/253 [00:03<00:00, 80.70batch/s]

Training batches on cuda:  98%|█████████▊| 248/253 [00:03<00:00, 78.72batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:29<00:06,  3.29s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  90%|█████████ | 9/10 [00:29<00:03,  3.37s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 6/253 [00:00<00:04, 54.01batch/s]

Training batches on cuda:   6%|▌         | 14/253 [00:00<00:04, 59.26batch/s]

Training batches on cuda:   9%|▉         | 23/253 [00:00<00:03, 65.65batch/s]

Training batches on cuda:  13%|█▎        | 32/253 [00:00<00:03, 70.95batch/s]

Training batches on cuda:  16%|█▌        | 41/253 [00:00<00:02, 74.37batch/s]

Training batches on cuda:  20%|█▉        | 50/253 [00:00<00:02, 76.62batch/s]

Training batches on cuda:  23%|██▎       | 59/253 [00:00<00:02, 77.58batch/s]

Training batches on cuda:  27%|██▋       | 68/253 [00:00<00:02, 79.85batch/s]

Training batches on cuda:  30%|███       | 77/253 [00:00<00:02, 80.56batch/s]

Training batches on cuda:  34%|███▍      | 87/253 [00:01<00:01, 83.45batch/s]

Training batches on cuda:  38%|███▊      | 96/253 [00:01<00:01, 84.70batch/s]

Training batches on cuda:  42%|████▏     | 105/253 [00:01<00:01, 83.36batch/s]

Training batches on cuda:  45%|████▌     | 114/253 [00:01<00:01, 82.63batch/s]

Training batches on cuda:  49%|████▊     | 123/253 [00:01<00:01, 81.35batch/s]

Training batches on cuda:  52%|█████▏    | 132/253 [00:01<00:01, 80.25batch/s]

Training batches on cuda:  56%|█████▌    | 141/253 [00:01<00:01, 79.77batch/s]

Training batches on cuda:  59%|█████▉    | 150/253 [00:01<00:01, 80.26batch/s]

Training batches on cuda:  63%|██████▎   | 160/253 [00:01<00:01, 83.07batch/s]

Training batches on cuda:  67%|██████▋   | 170/253 [00:02<00:00, 85.34batch/s]

Training batches on cuda:  71%|███████   | 180/253 [00:02<00:00, 87.16batch/s]

Training batches on cuda:  75%|███████▍  | 189/253 [00:02<00:00, 86.04batch/s]

Training batches on cuda:  78%|███████▊  | 198/253 [00:02<00:00, 83.59batch/s]

Training batches on cuda:  82%|████████▏ | 207/253 [00:02<00:00, 82.24batch/s]

Training batches on cuda:  85%|████████▌ | 216/253 [00:02<00:00, 80.99batch/s]

Training batches on cuda:  89%|████████▉ | 225/253 [00:02<00:00, 82.64batch/s]

Training batches on cuda:  92%|█████████▏| 234/253 [00:02<00:00, 83.01batch/s]

Training batches on cuda:  96%|█████████▌| 243/253 [00:02<00:00, 84.87batch/s]

Training batches on cuda: 100%|█████████▉| 252/253 [00:03<00:00, 86.25batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:32<00:03,  3.37s/epoch, loss=0.00107, prev_loss=0.00107]2021-09-16:18:00:18,753 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:00:19,770 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:00:34,738 INFO     [evaluator.py:657] Evaluation took 14.97s seconds
2021-09-16:18:00:34,869 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00,  8.17s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00,  4.89s/epoch, loss=0.00107, prev_loss=0.00107]
2021-09-16:18:00:34,870 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpj4al1fpr'
2021-09-16:18:00:34,941 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpj4al1fpr' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.36ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.55ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.57ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:03, 2.59ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.61ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:00:49,807 INFO     [evaluator.py:657] Evaluation took 14.85s seconds
[I 2021-09-16 18:00:49,812] Trial 24 finished with value: -0.0007994004661826537 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.11756600630603276, 'optimizer.lr': 0.02150006430160812, 'negative_sampler.num_negs_per_pos': 16, 'training.batch_size': 1232}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:00:49,835 WARNING  [api.py:823] No random seed is specified. Setting to 3928741925.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 9/1339 [00:00<00:15, 86.07batch/s]

Training batches on cuda:   2%|▏         | 27/1339 [00:00<00:12, 101.14batch/s]

Training batches on cuda:   3%|▎         | 44/1339 [00:00<00:11, 114.39batch/s]

Training batches on cuda:   5%|▍         | 61/1339 [00:00<00:10, 126.45batch/s]

Training batches on cuda:   6%|▌         | 79/1339 [00:00<00:09, 138.02batch/s]

Training batches on cuda:   7%|▋         | 97/1339 [00:00<00:08, 147.28batch/s]

Training batches on cuda:   9%|▊         | 115/1339 [00:00<00:07, 154.46batch/s]

Training batches on cuda:  10%|▉         | 133/1339 [00:00<00:07, 159.00batch/s]

Training batches on cuda:  11%|█         | 150/1339 [00:00<00:07, 161.19batch/s]

Training batches on cuda:  12%|█▏        | 167/1339 [00:01<00:08, 142.76batch/s]

Training batches on cuda:  14%|█▍        | 185/1339 [00:01<00:07, 151.32batch/s]

Training batches on cuda:  15%|█▌        | 203/1339 [00:01<00:07, 158.29batch/s]

Training batches on cuda:  17%|█▋        | 221/1339 [00:01<00:06, 163.16batch/s]

Training batches on cuda:  18%|█▊        | 240/1339 [00:01<00:06, 167.82batch/s]

Training batches on cuda:  19%|█▉        | 258/1339 [00:01<00:06, 170.77batch/s]

Training batches on cuda:  21%|██        | 276/1339 [00:01<00:06, 172.76batch/s]

Training batches on cuda:  22%|██▏       | 294/1339 [00:01<00:06, 173.60batch/s]

Training batches on cuda:  23%|██▎       | 314/1339 [00:01<00:05, 180.62batch/s]

Training batches on cuda:  25%|██▍       | 333/1339 [00:01<00:05, 183.33batch/s]

Training batches on cuda:  26%|██▋       | 352/1339 [00:02<00:05, 184.23batch/s]

Training batches on cuda:  28%|██▊       | 371/1339 [00:02<00:05, 182.13batch/s]

Training batches on cuda:  29%|██▉       | 391/1339 [00:02<00:05, 186.33batch/s]

Training batches on cuda:  31%|███       | 412/1339 [00:02<00:04, 191.39batch/s]

Training batches on cuda:  32%|███▏      | 433/1339 [00:02<00:04, 194.61batch/s]

Training batches on cuda:  34%|███▍      | 454/1339 [00:02<00:04, 197.90batch/s]

Training batches on cuda:  35%|███▌      | 474/1339 [00:02<00:04, 194.21batch/s]

Training batches on cuda:  37%|███▋      | 494/1339 [00:02<00:04, 190.81batch/s]

Training batches on cuda:  38%|███▊      | 514/1339 [00:02<00:04, 189.91batch/s]

Training batches on cuda:  40%|███▉      | 534/1339 [00:03<00:04, 185.33batch/s]

Training batches on cuda:  41%|████▏     | 553/1339 [00:03<00:04, 181.62batch/s]

Training batches on cuda:  43%|████▎     | 572/1339 [00:03<00:04, 183.01batch/s]

Training batches on cuda:  44%|████▍     | 591/1339 [00:03<00:04, 180.08batch/s]

Training batches on cuda:  46%|████▌     | 610/1339 [00:03<00:04, 172.16batch/s]

Training batches on cuda:  47%|████▋     | 628/1339 [00:03<00:04, 172.41batch/s]

Training batches on cuda:  48%|████▊     | 646/1339 [00:03<00:04, 162.22batch/s]

Training batches on cuda:  50%|████▉     | 663/1339 [00:03<00:04, 158.41batch/s]

Training batches on cuda:  51%|█████     | 679/1339 [00:03<00:04, 153.73batch/s]

Training batches on cuda:  52%|█████▏    | 695/1339 [00:04<00:04, 149.40batch/s]

Training batches on cuda:  53%|█████▎    | 711/1339 [00:04<00:04, 142.23batch/s]

Training batches on cuda:  54%|█████▍    | 726/1339 [00:04<00:04, 137.34batch/s]

Training batches on cuda:  55%|█████▌    | 740/1339 [00:04<00:04, 133.34batch/s]

Training batches on cuda:  56%|█████▋    | 754/1339 [00:04<00:04, 129.99batch/s]

Training batches on cuda:  58%|█████▊    | 770/1339 [00:04<00:04, 135.76batch/s]

Training batches on cuda:  59%|█████▊    | 784/1339 [00:04<00:04, 132.89batch/s]

Training batches on cuda:  60%|█████▉    | 798/1339 [00:04<00:04, 133.20batch/s]

Training batches on cuda:  61%|██████    | 814/1339 [00:04<00:03, 137.84batch/s]

Training batches on cuda:  62%|██████▏   | 833/1339 [00:05<00:03, 149.70batch/s]

Training batches on cuda:  64%|██████▎   | 852/1339 [00:05<00:03, 158.01batch/s]

Training batches on cuda:  65%|██████▍   | 869/1339 [00:05<00:02, 160.62batch/s]

Training batches on cuda:  66%|██████▋   | 888/1339 [00:05<00:02, 166.61batch/s]

Training batches on cuda:  68%|██████▊   | 909/1339 [00:05<00:02, 176.42batch/s]

Training batches on cuda:  69%|██████▉   | 930/1339 [00:05<00:02, 183.81batch/s]

Training batches on cuda:  71%|███████   | 951/1339 [00:05<00:02, 189.88batch/s]

Training batches on cuda:  73%|███████▎  | 973/1339 [00:05<00:01, 195.97batch/s]

Training batches on cuda:  74%|███████▍  | 994/1339 [00:05<00:01, 199.77batch/s]

Training batches on cuda:  76%|███████▌  | 1015/1339 [00:06<00:01, 193.34batch/s]

Training batches on cuda:  77%|███████▋  | 1035/1339 [00:06<00:01, 187.57batch/s]

Training batches on cuda:  79%|███████▊  | 1054/1339 [00:06<00:01, 182.06batch/s]

Training batches on cuda:  80%|████████  | 1073/1339 [00:06<00:01, 179.64batch/s]

Training batches on cuda:  82%|████████▏ | 1092/1339 [00:06<00:01, 179.05batch/s]

Training batches on cuda:  83%|████████▎ | 1111/1339 [00:06<00:01, 180.01batch/s]

Training batches on cuda:  84%|████████▍ | 1130/1339 [00:06<00:01, 177.20batch/s]

Training batches on cuda:  86%|████████▌ | 1150/1339 [00:06<00:01, 181.11batch/s]

Training batches on cuda:  87%|████████▋ | 1170/1339 [00:06<00:00, 186.04batch/s]

Training batches on cuda:  89%|████████▉ | 1190/1339 [00:06<00:00, 189.32batch/s]

Training batches on cuda:  90%|█████████ | 1211/1339 [00:07<00:00, 193.47batch/s]

Training batches on cuda:  92%|█████████▏| 1231/1339 [00:07<00:00, 182.04batch/s]

Training batches on cuda:  93%|█████████▎| 1250/1339 [00:07<00:00, 182.65batch/s]

Training batches on cuda:  95%|█████████▍| 1269/1339 [00:07<00:00, 183.39batch/s]

Training batches on cuda:  96%|█████████▌| 1288/1339 [00:07<00:00, 182.83batch/s]

Training batches on cuda:  98%|█████████▊| 1307/1339 [00:07<00:00, 181.65batch/s]

Training batches on cuda:  99%|█████████▉| 1326/1339 [00:07<00:00, 173.52batch/s]

                                                                                 
Training epochs on cuda:   0%|          | 0/10 [00:08<?, ?epoch/s, loss=0.0132, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:08<01:12,  8.03s/epoch, loss=0.0132, prev_loss=nan]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 8/1339 [00:00<00:16, 79.57batch/s]

Training batches on cuda:   2%|▏         | 27/1339 [00:00<00:13, 96.07batch/s]

Training batches on cuda:   3%|▎         | 41/1339 [00:00<00:12, 104.93batch/s]

Training batches on cuda:   4%|▍         | 57/1339 [00:00<00:11, 116.46batch/s]

Training batches on cuda:   5%|▌         | 73/1339 [00:00<00:10, 125.12batch/s]

Training batches on cuda:   7%|▋         | 89/1339 [00:00<00:09, 133.76batch/s]

Training batches on cuda:   8%|▊         | 107/1339 [00:00<00:08, 144.89batch/s]

Training batches on cuda:   9%|▉         | 122/1339 [00:00<00:09, 132.08batch/s]

Training batches on cuda:  10%|█         | 137/1339 [00:00<00:08, 135.45batch/s]

Training batches on cuda:  12%|█▏        | 155/1339 [00:01<00:08, 145.46batch/s]

Training batches on cuda:  13%|█▎        | 174/1339 [00:01<00:07, 149.92batch/s]

Training batches on cuda:  14%|█▍        | 192/1339 [00:01<00:07, 157.18batch/s]

Training batches on cuda:  16%|█▌        | 212/1339 [00:01<00:06, 166.74batch/s]

Training batches on cuda:  17%|█▋        | 230/1339 [00:01<00:06, 167.35batch/s]

Training batches on cuda:  19%|█▊        | 248/1339 [00:01<00:06, 164.76batch/s]

Training batches on cuda:  20%|█▉        | 265/1339 [00:01<00:06, 154.07batch/s]

Training batches on cuda:  21%|██        | 281/1339 [00:01<00:06, 151.84batch/s]

Training batches on cuda:  22%|██▏       | 300/1339 [00:01<00:06, 160.48batch/s]

Training batches on cuda:  24%|██▎       | 318/1339 [00:02<00:06, 164.30batch/s]

Training batches on cuda:  25%|██▌       | 336/1339 [00:02<00:06, 166.96batch/s]

Training batches on cuda:  26%|██▋       | 353/1339 [00:02<00:06, 160.08batch/s]

Training batches on cuda:  28%|██▊       | 370/1339 [00:02<00:06, 158.67batch/s]

Training batches on cuda:  29%|██▉       | 386/1339 [00:02<00:06, 155.89batch/s]

Training batches on cuda:  30%|███       | 405/1339 [00:02<00:05, 162.52batch/s]

Training batches on cuda:  32%|███▏      | 424/1339 [00:02<00:05, 168.37batch/s]

Training batches on cuda:  33%|███▎      | 442/1339 [00:02<00:05, 171.25batch/s]

Training batches on cuda:  34%|███▍      | 460/1339 [00:02<00:05, 170.88batch/s]

Training batches on cuda:  36%|███▌      | 478/1339 [00:03<00:04, 172.54batch/s]

Training batches on cuda:  37%|███▋      | 496/1339 [00:03<00:05, 155.70batch/s]

Training batches on cuda:  38%|███▊      | 514/1339 [00:03<00:05, 160.83batch/s]

Training batches on cuda:  40%|███▉      | 531/1339 [00:03<00:05, 153.86batch/s]

Training batches on cuda:  41%|████      | 551/1339 [00:03<00:04, 163.48batch/s]

Training batches on cuda:  43%|████▎     | 571/1339 [00:03<00:04, 171.16batch/s]

Training batches on cuda:  44%|████▍     | 589/1339 [00:03<00:04, 168.06batch/s]

Training batches on cuda:  45%|████▌     | 608/1339 [00:03<00:04, 171.77batch/s]

Training batches on cuda:  47%|████▋     | 629/1339 [00:03<00:03, 181.66batch/s]

Training batches on cuda:  49%|████▊     | 650/1339 [00:03<00:03, 187.35batch/s]

Training batches on cuda:  50%|█████     | 672/1339 [00:04<00:03, 195.18batch/s]

Training batches on cuda:  52%|█████▏    | 693/1339 [00:04<00:03, 198.45batch/s]

Training batches on cuda:  53%|█████▎    | 715/1339 [00:04<00:03, 202.53batch/s]

Training batches on cuda:  55%|█████▍    | 736/1339 [00:04<00:03, 199.49batch/s]

Training batches on cuda:  57%|█████▋    | 758/1339 [00:04<00:02, 202.90batch/s]

Training batches on cuda:  58%|█████▊    | 780/1339 [00:04<00:02, 205.58batch/s]

Training batches on cuda:  60%|█████▉    | 802/1339 [00:04<00:02, 207.44batch/s]

Training batches on cuda:  62%|██████▏   | 824/1339 [00:04<00:02, 210.76batch/s]

Training batches on cuda:  63%|██████▎   | 846/1339 [00:04<00:02, 201.78batch/s]

Training batches on cuda:  65%|██████▍   | 867/1339 [00:05<00:02, 195.40batch/s]

Training batches on cuda:  66%|██████▋   | 888/1339 [00:05<00:02, 196.92batch/s]

Training batches on cuda:  68%|██████▊   | 908/1339 [00:05<00:02, 181.73batch/s]

Training batches on cuda:  69%|██████▉   | 927/1339 [00:05<00:02, 181.84batch/s]

Training batches on cuda:  71%|███████   | 946/1339 [00:05<00:02, 154.14batch/s]

Training batches on cuda:  72%|███████▏  | 963/1339 [00:05<00:02, 154.62batch/s]

Training batches on cuda:  73%|███████▎  | 982/1339 [00:05<00:02, 162.70batch/s]

Training batches on cuda:  75%|███████▍  | 1002/1339 [00:05<00:01, 171.57batch/s]

Training batches on cuda:  76%|███████▋  | 1021/1339 [00:05<00:01, 175.49batch/s]

Training batches on cuda:  78%|███████▊  | 1042/1339 [00:06<00:01, 183.74batch/s]

Training batches on cuda:  79%|███████▉  | 1063/1339 [00:06<00:01, 190.64batch/s]

Training batches on cuda:  81%|████████  | 1084/1339 [00:06<00:01, 195.94batch/s]

Training batches on cuda:  83%|████████▎ | 1105/1339 [00:06<00:01, 198.78batch/s]

Training batches on cuda:  84%|████████▍ | 1126/1339 [00:06<00:01, 200.54batch/s]

Training batches on cuda:  86%|████████▌ | 1148/1339 [00:06<00:00, 203.58batch/s]

Training batches on cuda:  87%|████████▋ | 1169/1339 [00:06<00:00, 204.71batch/s]

Training batches on cuda:  89%|████████▉ | 1190/1339 [00:06<00:00, 204.57batch/s]

Training batches on cuda:  90%|█████████ | 1211/1339 [00:06<00:00, 205.82batch/s]

Training batches on cuda:  92%|█████████▏| 1232/1339 [00:06<00:00, 204.63batch/s]

Training batches on cuda:  94%|█████████▎| 1253/1339 [00:07<00:00, 195.00batch/s]

Training batches on cuda:  95%|█████████▌| 1273/1339 [00:07<00:00, 167.91batch/s]

Training batches on cuda:  96%|█████████▋| 1292/1339 [00:07<00:00, 171.80batch/s]

Training batches on cuda:  98%|█████████▊| 1310/1339 [00:07<00:00, 173.64batch/s]

Training batches on cuda:  99%|█████████▉| 1328/1339 [00:07<00:00, 154.48batch/s]

                                                                                 
Training epochs on cuda:  10%|█         | 1/10 [00:16<01:12,  8.03s/epoch, loss=0.0132, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:16<01:04,  8.02s/epoch, loss=0.0132, prev_loss=nan]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 6/1339 [00:00<00:22, 59.77batch/s]

Training batches on cuda:   2%|▏         | 26/1339 [00:00<00:17, 75.25batch/s]

Training batches on cuda:   3%|▎         | 43/1339 [00:00<00:14, 90.26batch/s]

Training batches on cuda:   4%|▍         | 57/1339 [00:00<00:12, 100.23batch/s]

Training batches on cuda:   5%|▌         | 71/1339 [00:00<00:11, 107.96batch/s]

Training batches on cuda:   6%|▋         | 86/1339 [00:00<00:10, 117.71batch/s]

Training batches on cuda:   8%|▊         | 102/1339 [00:00<00:09, 127.65batch/s]

Training batches on cuda:   9%|▊         | 116/1339 [00:00<00:09, 129.69batch/s]

Training batches on cuda:  10%|▉         | 130/1339 [00:00<00:09, 130.41batch/s]

Training batches on cuda:  11%|█         | 144/1339 [00:01<00:09, 130.97batch/s]

Training batches on cuda:  12%|█▏        | 162/1339 [00:01<00:08, 139.95batch/s]

Training batches on cuda:  13%|█▎        | 177/1339 [00:01<00:08, 141.14batch/s]

Training batches on cuda:  14%|█▍        | 194/1339 [00:01<00:07, 148.26batch/s]

Training batches on cuda:  16%|█▌        | 211/1339 [00:01<00:07, 153.00batch/s]

Training batches on cuda:  17%|█▋        | 229/1339 [00:01<00:06, 159.44batch/s]

Training batches on cuda:  19%|█▊        | 248/1339 [00:01<00:06, 165.75batch/s]

Training batches on cuda:  20%|█▉        | 265/1339 [00:01<00:06, 160.64batch/s]

Training batches on cuda:  21%|██        | 282/1339 [00:01<00:07, 142.94batch/s]

Training batches on cuda:  22%|██▏       | 301/1339 [00:02<00:06, 154.17batch/s]

Training batches on cuda:  24%|██▎       | 318/1339 [00:02<00:07, 140.51batch/s]

Training batches on cuda:  25%|██▌       | 336/1339 [00:02<00:06, 147.82batch/s]

Training batches on cuda:  26%|██▋       | 353/1339 [00:02<00:06, 152.07batch/s]

Training batches on cuda:  28%|██▊       | 371/1339 [00:02<00:06, 155.95batch/s]

Training batches on cuda:  29%|██▉       | 390/1339 [00:02<00:05, 162.85batch/s]

Training batches on cuda:  31%|███       | 409/1339 [00:02<00:05, 169.78batch/s]

Training batches on cuda:  32%|███▏      | 428/1339 [00:02<00:05, 174.77batch/s]

Training batches on cuda:  33%|███▎      | 446/1339 [00:02<00:05, 175.09batch/s]

Training batches on cuda:  35%|███▍      | 464/1339 [00:03<00:05, 171.17batch/s]

Training batches on cuda:  36%|███▌      | 482/1339 [00:03<00:05, 164.31batch/s]

Training batches on cuda:  37%|███▋      | 500/1339 [00:03<00:04, 168.50batch/s]

Training batches on cuda:  39%|███▊      | 517/1339 [00:03<00:05, 164.03batch/s]

Training batches on cuda:  40%|███▉      | 534/1339 [00:03<00:04, 165.20batch/s]

Training batches on cuda:  41%|████▏     | 554/1339 [00:03<00:04, 174.01batch/s]

Training batches on cuda:  43%|████▎     | 572/1339 [00:03<00:04, 160.52batch/s]

Training batches on cuda:  44%|████▍     | 589/1339 [00:03<00:04, 157.47batch/s]

Training batches on cuda:  45%|████▌     | 605/1339 [00:03<00:04, 157.23batch/s]

Training batches on cuda:  46%|████▋     | 621/1339 [00:04<00:04, 149.38batch/s]

Training batches on cuda:  48%|████▊     | 637/1339 [00:04<00:04, 151.38batch/s]

Training batches on cuda:  49%|████▉     | 655/1339 [00:04<00:04, 157.65batch/s]

Training batches on cuda:  50%|█████     | 674/1339 [00:04<00:04, 163.54batch/s]

Training batches on cuda:  52%|█████▏    | 691/1339 [00:04<00:04, 153.32batch/s]

Training batches on cuda:  53%|█████▎    | 707/1339 [00:04<00:04, 150.18batch/s]

Training batches on cuda:  54%|█████▍    | 724/1339 [00:04<00:03, 153.76batch/s]

Training batches on cuda:  55%|█████▌    | 740/1339 [00:04<00:03, 151.36batch/s]

Training batches on cuda:  56%|█████▋    | 756/1339 [00:04<00:04, 135.09batch/s]

Training batches on cuda:  58%|█████▊    | 771/1339 [00:05<00:04, 135.92batch/s]

Training batches on cuda:  59%|█████▉    | 787/1339 [00:05<00:03, 139.04batch/s]

Training batches on cuda:  60%|█████▉    | 802/1339 [00:05<00:04, 131.35batch/s]

Training batches on cuda:  61%|██████    | 818/1339 [00:05<00:03, 138.28batch/s]

Training batches on cuda:  62%|██████▏   | 833/1339 [00:05<00:03, 133.86batch/s]

Training batches on cuda:  64%|██████▎   | 853/1339 [00:05<00:03, 148.29batch/s]

Training batches on cuda:  65%|██████▍   | 869/1339 [00:05<00:03, 145.33batch/s]

Training batches on cuda:  66%|██████▌   | 886/1339 [00:05<00:03, 146.93batch/s]

Training batches on cuda:  67%|██████▋   | 903/1339 [00:05<00:02, 153.07batch/s]

Training batches on cuda:  69%|██████▊   | 919/1339 [00:06<00:02, 151.08batch/s]

Training batches on cuda:  70%|██████▉   | 935/1339 [00:06<00:02, 146.98batch/s]

Training batches on cuda:  71%|███████   | 950/1339 [00:06<00:02, 143.82batch/s]

Training batches on cuda:  72%|███████▏  | 969/1339 [00:06<00:02, 154.40batch/s]

Training batches on cuda:  74%|███████▎  | 987/1339 [00:06<00:02, 159.91batch/s]

Training batches on cuda:  75%|███████▍  | 1004/1339 [00:06<00:02, 161.88batch/s]

Training batches on cuda:  76%|███████▋  | 1021/1339 [00:06<00:01, 159.89batch/s]

Training batches on cuda:  78%|███████▊  | 1038/1339 [00:06<00:02, 149.70batch/s]

Training batches on cuda:  79%|███████▊  | 1054/1339 [00:06<00:01, 147.40batch/s]

Training batches on cuda:  80%|███████▉  | 1069/1339 [00:07<00:01, 135.41batch/s]

Training batches on cuda:  81%|████████  | 1087/1339 [00:07<00:01, 145.53batch/s]

Training batches on cuda:  82%|████████▏ | 1104/1339 [00:07<00:01, 150.53batch/s]

Training batches on cuda:  84%|████████▎ | 1120/1339 [00:07<00:01, 145.25batch/s]

Training batches on cuda:  85%|████████▍ | 1135/1339 [00:07<00:01, 146.41batch/s]

Training batches on cuda:  86%|████████▌ | 1151/1339 [00:07<00:01, 148.57batch/s]

Training batches on cuda:  87%|████████▋ | 1170/1339 [00:07<00:01, 153.82batch/s]

Training batches on cuda:  89%|████████▉ | 1189/1339 [00:07<00:00, 161.81batch/s]

Training batches on cuda:  90%|█████████ | 1206/1339 [00:07<00:00, 152.15batch/s]

Training batches on cuda:  91%|█████████▏| 1225/1339 [00:08<00:00, 160.25batch/s]

Training batches on cuda:  93%|█████████▎| 1242/1339 [00:08<00:00, 157.33batch/s]

Training batches on cuda:  94%|█████████▍| 1259/1339 [00:08<00:00, 155.37batch/s]

Training batches on cuda:  95%|█████████▌| 1276/1339 [00:08<00:00, 157.87batch/s]

Training batches on cuda:  96%|█████████▋| 1292/1339 [00:08<00:00, 154.88batch/s]

Training batches on cuda:  98%|█████████▊| 1308/1339 [00:08<00:00, 156.15batch/s]

Training batches on cuda:  99%|█████████▉| 1324/1339 [00:08<00:00, 146.39batch/s]

Training batches on cuda: 100%|██████████| 1339/1339 [00:08<00:00, 143.15batch/s]

                                                                                 
Training epochs on cuda:  20%|██        | 2/10 [00:25<01:04,  8.02s/epoch, loss=0.0132, prev_loss=0.0132]
Training epochs on cuda:  30%|███       | 3/10 [00:25<00:58,  8.36s/epoch, loss=0.0132, prev_loss=0.0132]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 8/1339 [00:00<00:17, 76.65batch/s]

Training batches on cuda:   2%|▏         | 26/1339 [00:00<00:14, 92.60batch/s]

Training batches on cuda:   3%|▎         | 45/1339 [00:00<00:11, 109.01batch/s]

Training batches on cuda:   5%|▍         | 61/1339 [00:00<00:10, 119.70batch/s]

Training batches on cuda:   6%|▌         | 81/1339 [00:00<00:09, 134.27batch/s]

Training batches on cuda:   7%|▋         | 96/1339 [00:00<00:09, 134.68batch/s]

Training batches on cuda:   9%|▊         | 114/1339 [00:00<00:08, 144.56batch/s]

Training batches on cuda:  10%|▉         | 130/1339 [00:00<00:08, 148.53batch/s]

Training batches on cuda:  11%|█         | 146/1339 [00:00<00:08, 146.22batch/s]

Training batches on cuda:  12%|█▏        | 161/1339 [00:01<00:08, 138.96batch/s]

Training batches on cuda:  13%|█▎        | 176/1339 [00:01<00:08, 139.34batch/s]

Training batches on cuda:  14%|█▍        | 191/1339 [00:01<00:08, 138.12batch/s]

Training batches on cuda:  15%|█▌        | 206/1339 [00:01<00:08, 139.03batch/s]

Training batches on cuda:  17%|█▋        | 221/1339 [00:01<00:07, 140.68batch/s]

Training batches on cuda:  18%|█▊        | 236/1339 [00:01<00:07, 141.50batch/s]

Training batches on cuda:  19%|█▉        | 252/1339 [00:01<00:07, 145.99batch/s]

Training batches on cuda:  20%|██        | 271/1339 [00:01<00:06, 155.77batch/s]

Training batches on cuda:  21%|██▏       | 287/1339 [00:01<00:07, 147.56batch/s]

Training batches on cuda:  23%|██▎       | 303/1339 [00:02<00:06, 150.41batch/s]

Training batches on cuda:  24%|██▍       | 319/1339 [00:02<00:06, 152.43batch/s]

Training batches on cuda:  25%|██▌       | 340/1339 [00:02<00:06, 164.86batch/s]

Training batches on cuda:  27%|██▋       | 359/1339 [00:02<00:05, 169.34batch/s]

Training batches on cuda:  28%|██▊       | 377/1339 [00:02<00:05, 170.89batch/s]

Training batches on cuda:  29%|██▉       | 395/1339 [00:02<00:05, 168.99batch/s]

Training batches on cuda:  31%|███       | 417/1339 [00:02<00:05, 179.72batch/s]

Training batches on cuda:  33%|███▎      | 436/1339 [00:02<00:05, 169.35batch/s]

Training batches on cuda:  34%|███▍      | 454/1339 [00:02<00:05, 158.84batch/s]

Training batches on cuda:  35%|███▌      | 471/1339 [00:03<00:05, 157.72batch/s]

Training batches on cuda:  36%|███▋      | 488/1339 [00:03<00:05, 154.92batch/s]

Training batches on cuda:  38%|███▊      | 504/1339 [00:03<00:05, 140.99batch/s]

Training batches on cuda:  39%|███▉      | 519/1339 [00:03<00:05, 142.36batch/s]

Training batches on cuda:  40%|███▉      | 535/1339 [00:03<00:05, 144.84batch/s]

Training batches on cuda:  41%|████      | 550/1339 [00:03<00:05, 142.79batch/s]

Training batches on cuda:  42%|████▏     | 565/1339 [00:03<00:05, 143.91batch/s]

Training batches on cuda:  43%|████▎     | 580/1339 [00:03<00:05, 143.87batch/s]

Training batches on cuda:  45%|████▍     | 601/1339 [00:03<00:04, 157.46batch/s]

Training batches on cuda:  46%|████▌     | 618/1339 [00:04<00:04, 159.09batch/s]

Training batches on cuda:  47%|████▋     | 635/1339 [00:04<00:04, 155.71batch/s]

Training batches on cuda:  49%|████▊     | 652/1339 [00:04<00:04, 159.04batch/s]

Training batches on cuda:  50%|████▉     | 669/1339 [00:04<00:04, 150.88batch/s]

Training batches on cuda:  51%|█████     | 685/1339 [00:04<00:04, 151.43batch/s]

Training batches on cuda:  52%|█████▏    | 701/1339 [00:04<00:04, 152.61batch/s]

Training batches on cuda:  54%|█████▎    | 718/1339 [00:04<00:04, 150.72batch/s]

Training batches on cuda:  55%|█████▍    | 734/1339 [00:04<00:04, 148.75batch/s]

Training batches on cuda:  56%|█████▌    | 752/1339 [00:04<00:03, 156.56batch/s]

Training batches on cuda:  58%|█████▊    | 770/1339 [00:04<00:03, 157.12batch/s]

Training batches on cuda:  59%|█████▊    | 786/1339 [00:05<00:03, 142.48batch/s]

Training batches on cuda:  60%|██████    | 805/1339 [00:05<00:03, 152.09batch/s]

Training batches on cuda:  62%|██████▏   | 824/1339 [00:05<00:03, 161.16batch/s]

Training batches on cuda:  63%|██████▎   | 845/1339 [00:05<00:02, 172.81batch/s]

Training batches on cuda:  65%|██████▍   | 866/1339 [00:05<00:02, 180.82batch/s]

Training batches on cuda:  66%|██████▌   | 885/1339 [00:05<00:02, 174.65batch/s]

Training batches on cuda:  67%|██████▋   | 903/1339 [00:05<00:02, 174.23batch/s]

Training batches on cuda:  69%|██████▉   | 921/1339 [00:05<00:02, 172.95batch/s]

Training batches on cuda:  70%|███████   | 939/1339 [00:06<00:02, 154.60batch/s]

Training batches on cuda:  71%|███████▏  | 957/1339 [00:06<00:02, 159.53batch/s]

Training batches on cuda:  73%|███████▎  | 974/1339 [00:06<00:02, 155.28batch/s]

Training batches on cuda:  74%|███████▍  | 992/1339 [00:06<00:02, 160.56batch/s]

Training batches on cuda:  75%|███████▌  | 1009/1339 [00:06<00:02, 160.13batch/s]

Training batches on cuda:  77%|███████▋  | 1028/1339 [00:06<00:01, 166.88batch/s]

Training batches on cuda:  78%|███████▊  | 1045/1339 [00:06<00:01, 157.92batch/s]

Training batches on cuda:  79%|███████▉  | 1062/1339 [00:06<00:01, 151.38batch/s]

Training batches on cuda:  81%|████████  | 1078/1339 [00:06<00:01, 134.22batch/s]

Training batches on cuda:  82%|████████▏ | 1094/1339 [00:07<00:01, 138.34batch/s]

Training batches on cuda:  83%|████████▎ | 1109/1339 [00:07<00:01, 140.70batch/s]

Training batches on cuda:  84%|████████▍ | 1124/1339 [00:07<00:01, 132.05batch/s]

Training batches on cuda:  85%|████████▍ | 1138/1339 [00:07<00:01, 132.00batch/s]

Training batches on cuda:  86%|████████▌ | 1152/1339 [00:07<00:01, 128.00batch/s]

Training batches on cuda:  87%|████████▋ | 1166/1339 [00:07<00:01, 131.33batch/s]

Training batches on cuda:  88%|████████▊ | 1182/1339 [00:07<00:01, 137.11batch/s]

Training batches on cuda:  89%|████████▉ | 1196/1339 [00:07<00:01, 136.07batch/s]

Training batches on cuda:  91%|█████████ | 1213/1339 [00:07<00:00, 143.60batch/s]

Training batches on cuda:  92%|█████████▏| 1228/1339 [00:08<00:00, 140.70batch/s]

Training batches on cuda:  93%|█████████▎| 1243/1339 [00:08<00:00, 132.51batch/s]

Training batches on cuda:  94%|█████████▍| 1261/1339 [00:08<00:00, 142.33batch/s]

Training batches on cuda:  95%|█████████▌| 1277/1339 [00:08<00:00, 144.88batch/s]

Training batches on cuda:  97%|█████████▋| 1294/1339 [00:08<00:00, 151.35batch/s]

Training batches on cuda:  98%|█████████▊| 1310/1339 [00:08<00:00, 148.75batch/s]

Training batches on cuda:  99%|█████████▉| 1327/1339 [00:08<00:00, 152.97batch/s]

                                                                                 
Training epochs on cuda:  30%|███       | 3/10 [00:34<00:58,  8.36s/epoch, loss=0.0132, prev_loss=0.0132]
Training epochs on cuda:  40%|████      | 4/10 [00:34<00:51,  8.56s/epoch, loss=0.0132, prev_loss=0.0132]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 9/1339 [00:00<00:15, 84.84batch/s]

Training batches on cuda:   2%|▏         | 24/1339 [00:00<00:13, 96.73batch/s]

Training batches on cuda:   3%|▎         | 42/1339 [00:00<00:11, 111.91batch/s]

Training batches on cuda:   5%|▍         | 61/1339 [00:00<00:10, 126.89batch/s]

Training batches on cuda:   6%|▌         | 80/1339 [00:00<00:08, 140.66batch/s]

Training batches on cuda:   7%|▋         | 95/1339 [00:00<00:09, 134.65batch/s]

Training batches on cuda:   8%|▊         | 111/1339 [00:00<00:08, 140.18batch/s]

Training batches on cuda:  10%|▉         | 128/1339 [00:00<00:08, 146.19batch/s]

Training batches on cuda:  11%|█         | 146/1339 [00:00<00:07, 154.63batch/s]

Training batches on cuda:  12%|█▏        | 162/1339 [00:01<00:07, 155.17batch/s]

Training batches on cuda:  13%|█▎        | 178/1339 [00:01<00:08, 145.12batch/s]

Training batches on cuda:  14%|█▍        | 194/1339 [00:01<00:07, 148.76batch/s]

Training batches on cuda:  16%|█▌        | 210/1339 [00:01<00:07, 150.31batch/s]

Training batches on cuda:  17%|█▋        | 226/1339 [00:01<00:07, 146.24batch/s]

Training batches on cuda:  18%|█▊        | 244/1339 [00:01<00:07, 153.19batch/s]

Training batches on cuda:  19%|█▉        | 260/1339 [00:01<00:07, 147.17batch/s]

Training batches on cuda:  21%|██        | 281/1339 [00:01<00:06, 160.03batch/s]

Training batches on cuda:  22%|██▏       | 301/1339 [00:01<00:06, 168.92batch/s]

Training batches on cuda:  24%|██▍       | 319/1339 [00:02<00:06, 154.52batch/s]

Training batches on cuda:  25%|██▌       | 337/1339 [00:02<00:06, 159.71batch/s]

Training batches on cuda:  26%|██▋       | 354/1339 [00:02<00:06, 159.04batch/s]

Training batches on cuda:  28%|██▊       | 371/1339 [00:02<00:05, 162.06batch/s]

Training batches on cuda:  29%|██▉       | 388/1339 [00:02<00:05, 161.18batch/s]

Training batches on cuda:  30%|███       | 405/1339 [00:02<00:05, 155.83batch/s]

Training batches on cuda:  31%|███▏      | 421/1339 [00:02<00:06, 148.84batch/s]

Training batches on cuda:  33%|███▎      | 439/1339 [00:02<00:05, 153.99batch/s]

Training batches on cuda:  34%|███▍      | 458/1339 [00:02<00:05, 162.10batch/s]

Training batches on cuda:  35%|███▌      | 475/1339 [00:03<00:05, 158.02batch/s]

Training batches on cuda:  37%|███▋      | 491/1339 [00:03<00:05, 154.44batch/s]

Training batches on cuda:  38%|███▊      | 510/1339 [00:03<00:05, 163.10batch/s]

Training batches on cuda:  40%|███▉      | 531/1339 [00:03<00:04, 173.58batch/s]

Training batches on cuda:  41%|████      | 549/1339 [00:03<00:04, 167.25batch/s]

Training batches on cuda:  42%|████▏     | 567/1339 [00:03<00:04, 167.07batch/s]

Training batches on cuda:  44%|████▎     | 584/1339 [00:03<00:04, 157.48batch/s]

Training batches on cuda:  45%|████▌     | 604/1339 [00:03<00:04, 167.11batch/s]

Training batches on cuda:  46%|████▋     | 622/1339 [00:03<00:04, 156.40batch/s]

Training batches on cuda:  48%|████▊     | 641/1339 [00:04<00:04, 164.65batch/s]

Training batches on cuda:  49%|████▉     | 658/1339 [00:04<00:04, 139.10batch/s]

Training batches on cuda:  50%|█████     | 673/1339 [00:04<00:04, 137.88batch/s]

Training batches on cuda:  51%|█████▏    | 689/1339 [00:04<00:04, 143.40batch/s]

Training batches on cuda:  53%|█████▎    | 707/1339 [00:04<00:04, 151.60batch/s]

Training batches on cuda:  54%|█████▍    | 724/1339 [00:04<00:03, 154.76batch/s]

Training batches on cuda:  55%|█████▌    | 742/1339 [00:04<00:03, 161.43batch/s]

Training batches on cuda:  57%|█████▋    | 759/1339 [00:04<00:03, 163.31batch/s]

Training batches on cuda:  58%|█████▊    | 780/1339 [00:04<00:03, 172.82batch/s]

Training batches on cuda:  60%|█████▉    | 798/1339 [00:05<00:03, 163.56batch/s]

Training batches on cuda:  61%|██████    | 815/1339 [00:05<00:03, 152.52batch/s]

Training batches on cuda:  62%|██████▏   | 831/1339 [00:05<00:03, 152.50batch/s]

Training batches on cuda:  63%|██████▎   | 847/1339 [00:05<00:03, 146.77batch/s]

Training batches on cuda:  64%|██████▍   | 862/1339 [00:05<00:03, 147.46batch/s]

Training batches on cuda:  66%|██████▌   | 878/1339 [00:05<00:03, 148.44batch/s]

Training batches on cuda:  67%|██████▋   | 893/1339 [00:05<00:03, 138.65batch/s]

Training batches on cuda:  68%|██████▊   | 908/1339 [00:05<00:03, 141.78batch/s]

Training batches on cuda:  69%|██████▉   | 923/1339 [00:05<00:03, 136.40batch/s]

Training batches on cuda:  70%|███████   | 938/1339 [00:06<00:02, 139.10batch/s]

Training batches on cuda:  71%|███████▏  | 955/1339 [00:06<00:02, 146.01batch/s]

Training batches on cuda:  72%|███████▏  | 970/1339 [00:06<00:02, 145.23batch/s]

Training batches on cuda:  74%|███████▎  | 986/1339 [00:06<00:02, 149.17batch/s]

Training batches on cuda:  75%|███████▍  | 1002/1339 [00:06<00:02, 142.68batch/s]

Training batches on cuda:  76%|███████▌  | 1017/1339 [00:06<00:02, 143.01batch/s]

Training batches on cuda:  77%|███████▋  | 1032/1339 [00:06<00:02, 139.58batch/s]

Training batches on cuda:  78%|███████▊  | 1050/1339 [00:06<00:01, 148.91batch/s]

Training batches on cuda:  80%|███████▉  | 1068/1339 [00:06<00:01, 155.66batch/s]

Training batches on cuda:  81%|████████  | 1085/1339 [00:07<00:01, 159.09batch/s]

Training batches on cuda:  82%|████████▏ | 1102/1339 [00:07<00:01, 151.35batch/s]

Training batches on cuda:  84%|████████▎ | 1119/1339 [00:07<00:01, 153.22batch/s]

Training batches on cuda:  85%|████████▍ | 1135/1339 [00:07<00:01, 155.07batch/s]

Training batches on cuda:  86%|████████▌ | 1154/1339 [00:07<00:01, 162.67batch/s]

Training batches on cuda:  88%|████████▊ | 1175/1339 [00:07<00:00, 173.78batch/s]

Training batches on cuda:  89%|████████▉ | 1194/1339 [00:07<00:00, 177.98batch/s]

Training batches on cuda:  91%|█████████ | 1215/1339 [00:07<00:00, 186.15batch/s]

Training batches on cuda:  92%|█████████▏| 1235/1339 [00:07<00:00, 188.45batch/s]

Training batches on cuda:  94%|█████████▎| 1255/1339 [00:07<00:00, 190.84batch/s]

Training batches on cuda:  95%|█████████▌| 1275/1339 [00:08<00:00, 192.28batch/s]

Training batches on cuda:  97%|█████████▋| 1295/1339 [00:08<00:00, 178.29batch/s]

Training batches on cuda:  98%|█████████▊| 1314/1339 [00:08<00:00, 178.00batch/s]

Training batches on cuda: 100%|█████████▉| 1333/1339 [00:08<00:00, 179.28batch/s]

                                                                                 
Training epochs on cuda:  40%|████      | 4/10 [00:42<00:51,  8.56s/epoch, loss=0.0132, prev_loss=0.0132]
Training epochs on cuda:  50%|█████     | 5/10 [00:42<00:43,  8.62s/epoch, loss=0.0132, prev_loss=0.0132]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 15/1339 [00:00<00:09, 143.36batch/s]

Training batches on cuda:   3%|▎         | 35/1339 [00:00<00:08, 156.13batch/s]

Training batches on cuda:   4%|▍         | 54/1339 [00:00<00:07, 163.30batch/s]

Training batches on cuda:   6%|▌         | 75/1339 [00:00<00:07, 174.12batch/s]

Training batches on cuda:   7%|▋         | 96/1339 [00:00<00:06, 181.74batch/s]

Training batches on cuda:   9%|▊         | 117/1339 [00:00<00:06, 187.48batch/s]

Training batches on cuda:  10%|█         | 138/1339 [00:00<00:06, 192.42batch/s]

Training batches on cuda:  12%|█▏        | 157/1339 [00:00<00:06, 186.51batch/s]

Training batches on cuda:  13%|█▎        | 176/1339 [00:00<00:06, 171.57batch/s]

Training batches on cuda:  14%|█▍        | 193/1339 [00:01<00:06, 170.98batch/s]

Training batches on cuda:  16%|█▌        | 212/1339 [00:01<00:06, 174.09batch/s]

Training batches on cuda:  17%|█▋        | 231/1339 [00:01<00:06, 178.53batch/s]

Training batches on cuda:  19%|█▊        | 250/1339 [00:01<00:06, 180.00batch/s]

Training batches on cuda:  20%|██        | 269/1339 [00:01<00:05, 182.37batch/s]

Training batches on cuda:  22%|██▏       | 288/1339 [00:01<00:05, 176.70batch/s]

Training batches on cuda:  23%|██▎       | 306/1339 [00:01<00:06, 171.06batch/s]

Training batches on cuda:  24%|██▍       | 324/1339 [00:01<00:06, 167.36batch/s]

Training batches on cuda:  26%|██▌       | 342/1339 [00:01<00:05, 169.71batch/s]

Training batches on cuda:  27%|██▋       | 360/1339 [00:02<00:05, 170.39batch/s]

Training batches on cuda:  28%|██▊       | 378/1339 [00:02<00:05, 169.77batch/s]

Training batches on cuda:  30%|██▉       | 396/1339 [00:02<00:05, 169.13batch/s]

Training batches on cuda:  31%|███       | 413/1339 [00:02<00:05, 165.86batch/s]

Training batches on cuda:  32%|███▏      | 430/1339 [00:02<00:05, 166.19batch/s]

Training batches on cuda:  33%|███▎      | 447/1339 [00:02<00:05, 165.75batch/s]

Training batches on cuda:  35%|███▍      | 465/1339 [00:02<00:05, 167.27batch/s]

Training batches on cuda:  36%|███▌      | 482/1339 [00:02<00:05, 167.80batch/s]

Training batches on cuda:  37%|███▋      | 499/1339 [00:02<00:05, 167.43batch/s]

Training batches on cuda:  39%|███▊      | 517/1339 [00:02<00:04, 170.87batch/s]

Training batches on cuda:  40%|███▉      | 535/1339 [00:03<00:04, 166.67batch/s]

Training batches on cuda:  41%|████      | 552/1339 [00:03<00:04, 167.35batch/s]

Training batches on cuda:  43%|████▎     | 572/1339 [00:03<00:04, 174.94batch/s]

Training batches on cuda:  44%|████▍     | 590/1339 [00:03<00:04, 170.72batch/s]

Training batches on cuda:  45%|████▌     | 608/1339 [00:03<00:04, 153.39batch/s]

Training batches on cuda:  47%|████▋     | 627/1339 [00:03<00:04, 159.90batch/s]

Training batches on cuda:  48%|████▊     | 644/1339 [00:03<00:04, 157.65batch/s]

Training batches on cuda:  50%|████▉     | 663/1339 [00:03<00:04, 164.50batch/s]

Training batches on cuda:  51%|█████     | 680/1339 [00:03<00:04, 158.30batch/s]

Training batches on cuda:  52%|█████▏    | 698/1339 [00:04<00:03, 163.63batch/s]

Training batches on cuda:  54%|█████▎    | 717/1339 [00:04<00:03, 168.58batch/s]

Training batches on cuda:  55%|█████▌    | 737/1339 [00:04<00:03, 176.43batch/s]

Training batches on cuda:  56%|█████▋    | 756/1339 [00:04<00:03, 178.92batch/s]

Training batches on cuda:  58%|█████▊    | 776/1339 [00:04<00:03, 181.70batch/s]

Training batches on cuda:  59%|█████▉    | 796/1339 [00:04<00:02, 186.45batch/s]

Training batches on cuda:  61%|██████    | 815/1339 [00:04<00:02, 186.53batch/s]

Training batches on cuda:  62%|██████▏   | 834/1339 [00:04<00:02, 185.87batch/s]

Training batches on cuda:  64%|██████▍   | 854/1339 [00:04<00:02, 187.57batch/s]

Training batches on cuda:  65%|██████▌   | 873/1339 [00:05<00:02, 179.57batch/s]

Training batches on cuda:  67%|██████▋   | 894/1339 [00:05<00:02, 185.80batch/s]

Training batches on cuda:  68%|██████▊   | 913/1339 [00:05<00:02, 185.14batch/s]

Training batches on cuda:  70%|██████▉   | 932/1339 [00:05<00:02, 183.53batch/s]

Training batches on cuda:  71%|███████   | 951/1339 [00:05<00:02, 176.46batch/s]

Training batches on cuda:  72%|███████▏  | 969/1339 [00:05<00:02, 174.59batch/s]

Training batches on cuda:  74%|███████▎  | 987/1339 [00:05<00:02, 170.92batch/s]

Training batches on cuda:  75%|███████▌  | 1005/1339 [00:05<00:02, 166.60batch/s]

Training batches on cuda:  76%|███████▋  | 1022/1339 [00:05<00:01, 163.02batch/s]

Training batches on cuda:  78%|███████▊  | 1039/1339 [00:06<00:01, 155.58batch/s]

Training batches on cuda:  79%|███████▉  | 1059/1339 [00:06<00:01, 166.27batch/s]

Training batches on cuda:  81%|████████  | 1079/1339 [00:06<00:01, 174.98batch/s]

Training batches on cuda:  82%|████████▏ | 1097/1339 [00:06<00:01, 165.03batch/s]

Training batches on cuda:  83%|████████▎ | 1114/1339 [00:06<00:01, 158.27batch/s]

Training batches on cuda:  85%|████████▍ | 1135/1339 [00:06<00:01, 169.51batch/s]

Training batches on cuda:  86%|████████▌ | 1154/1339 [00:06<00:01, 174.65batch/s]

Training batches on cuda:  88%|████████▊ | 1174/1339 [00:06<00:00, 181.41batch/s]

Training batches on cuda:  89%|████████▉ | 1193/1339 [00:06<00:00, 179.39batch/s]

Training batches on cuda:  91%|█████████ | 1212/1339 [00:06<00:00, 172.98batch/s]

Training batches on cuda:  92%|█████████▏| 1230/1339 [00:07<00:00, 170.27batch/s]

Training batches on cuda:  93%|█████████▎| 1248/1339 [00:07<00:00, 167.80batch/s]

Training batches on cuda:  94%|█████████▍| 1265/1339 [00:07<00:00, 164.60batch/s]

Training batches on cuda:  96%|█████████▌| 1282/1339 [00:07<00:00, 159.35batch/s]

Training batches on cuda:  97%|█████████▋| 1299/1339 [00:07<00:00, 160.08batch/s]

Training batches on cuda:  98%|█████████▊| 1316/1339 [00:07<00:00, 157.86batch/s]

Training batches on cuda:  99%|█████████▉| 1332/1339 [00:07<00:00, 157.61batch/s]

                                                                                 
Training epochs on cuda:  50%|█████     | 5/10 [00:51<00:43,  8.62s/epoch, loss=0.0132, prev_loss=0.0132]
Training epochs on cuda:  60%|██████    | 6/10 [00:51<00:33,  8.45s/epoch, loss=0.0132, prev_loss=0.0132]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 13/1339 [00:00<00:10, 123.52batch/s]

Training batches on cuda:   2%|▏         | 32/1339 [00:00<00:09, 136.94batch/s]

Training batches on cuda:   4%|▍         | 51/1339 [00:00<00:08, 148.43batch/s]

Training batches on cuda:   5%|▌         | 70/1339 [00:00<00:08, 157.36batch/s]

Training batches on cuda:   6%|▋         | 84/1339 [00:00<00:09, 137.08batch/s]

Training batches on cuda:   8%|▊         | 102/1339 [00:00<00:08, 147.40batch/s]

Training batches on cuda:   9%|▉         | 120/1339 [00:00<00:07, 154.26batch/s]

Training batches on cuda:  10%|█         | 138/1339 [00:00<00:07, 159.90batch/s]

Training batches on cuda:  12%|█▏        | 156/1339 [00:00<00:07, 164.62batch/s]

Training batches on cuda:  13%|█▎        | 174/1339 [00:01<00:06, 168.80batch/s]

Training batches on cuda:  14%|█▍        | 192/1339 [00:01<00:06, 171.76batch/s]

Training batches on cuda:  16%|█▌        | 211/1339 [00:01<00:06, 174.91batch/s]

Training batches on cuda:  17%|█▋        | 231/1339 [00:01<00:06, 179.72batch/s]

Training batches on cuda:  19%|█▊        | 251/1339 [00:01<00:05, 182.88batch/s]

Training batches on cuda:  20%|██        | 270/1339 [00:01<00:06, 174.77batch/s]

Training batches on cuda:  22%|██▏       | 291/1339 [00:01<00:05, 183.18batch/s]

Training batches on cuda:  23%|██▎       | 310/1339 [00:01<00:05, 181.41batch/s]

Training batches on cuda:  25%|██▍       | 329/1339 [00:01<00:05, 172.46batch/s]

Training batches on cuda:  26%|██▌       | 347/1339 [00:02<00:05, 174.52batch/s]

Training batches on cuda:  27%|██▋       | 365/1339 [00:02<00:05, 174.14batch/s]

Training batches on cuda:  29%|██▊       | 383/1339 [00:02<00:05, 173.16batch/s]

Training batches on cuda:  30%|██▉       | 401/1339 [00:02<00:05, 168.27batch/s]

Training batches on cuda:  31%|███▏      | 419/1339 [00:02<00:05, 169.54batch/s]

Training batches on cuda:  33%|███▎      | 437/1339 [00:02<00:05, 171.80batch/s]

Training batches on cuda:  34%|███▍      | 455/1339 [00:02<00:05, 166.39batch/s]

Training batches on cuda:  35%|███▌      | 473/1339 [00:02<00:05, 170.00batch/s]

Training batches on cuda:  37%|███▋      | 491/1339 [00:02<00:04, 172.27batch/s]

Training batches on cuda:  38%|███▊      | 509/1339 [00:02<00:04, 171.60batch/s]

Training batches on cuda:  39%|███▉      | 528/1339 [00:03<00:04, 174.93batch/s]

Training batches on cuda:  41%|████      | 546/1339 [00:03<00:04, 176.07batch/s]

Training batches on cuda:  42%|████▏     | 564/1339 [00:03<00:04, 176.49batch/s]

Training batches on cuda:  43%|████▎     | 582/1339 [00:03<00:04, 176.86batch/s]

Training batches on cuda:  45%|████▌     | 603/1339 [00:03<00:04, 183.94batch/s]

Training batches on cuda:  47%|████▋     | 624/1339 [00:03<00:03, 189.64batch/s]

Training batches on cuda:  48%|████▊     | 645/1339 [00:03<00:03, 191.76batch/s]

Training batches on cuda:  50%|████▉     | 665/1339 [00:03<00:03, 190.61batch/s]

Training batches on cuda:  51%|█████     | 685/1339 [00:03<00:03, 190.79batch/s]

Training batches on cuda:  53%|█████▎    | 705/1339 [00:04<00:03, 187.91batch/s]

Training batches on cuda:  54%|█████▍    | 724/1339 [00:04<00:03, 184.89batch/s]

Training batches on cuda:  55%|█████▌    | 743/1339 [00:04<00:03, 179.44batch/s]

Training batches on cuda:  57%|█████▋    | 762/1339 [00:04<00:03, 178.25batch/s]

Training batches on cuda:  58%|█████▊    | 780/1339 [00:04<00:03, 176.01batch/s]

Training batches on cuda:  60%|█████▉    | 798/1339 [00:04<00:03, 172.55batch/s]

Training batches on cuda:  61%|██████    | 816/1339 [00:04<00:03, 169.32batch/s]

Training batches on cuda:  62%|██████▏   | 833/1339 [00:04<00:03, 164.44batch/s]

Training batches on cuda:  64%|██████▎   | 851/1339 [00:04<00:02, 168.50batch/s]

Training batches on cuda:  65%|██████▌   | 871/1339 [00:04<00:02, 174.79batch/s]

Training batches on cuda:  67%|██████▋   | 891/1339 [00:05<00:02, 180.31batch/s]

Training batches on cuda:  68%|██████▊   | 911/1339 [00:05<00:02, 184.91batch/s]

Training batches on cuda:  70%|██████▉   | 931/1339 [00:05<00:02, 189.00batch/s]

Training batches on cuda:  71%|███████   | 952/1339 [00:05<00:02, 192.28batch/s]

Training batches on cuda:  73%|███████▎  | 972/1339 [00:05<00:01, 187.32batch/s]

Training batches on cuda:  74%|███████▍  | 991/1339 [00:05<00:02, 165.02batch/s]

Training batches on cuda:  75%|███████▌  | 1009/1339 [00:05<00:01, 168.73batch/s]

Training batches on cuda:  77%|███████▋  | 1027/1339 [00:05<00:01, 168.19batch/s]

Training batches on cuda:  78%|███████▊  | 1045/1339 [00:05<00:01, 169.36batch/s]

Training batches on cuda:  79%|███████▉  | 1063/1339 [00:06<00:01, 168.88batch/s]

Training batches on cuda:  81%|████████  | 1082/1339 [00:06<00:01, 172.75batch/s]

Training batches on cuda:  82%|████████▏ | 1100/1339 [00:06<00:01, 174.21batch/s]

Training batches on cuda:  83%|████████▎ | 1118/1339 [00:06<00:01, 174.34batch/s]

Training batches on cuda:  85%|████████▍ | 1136/1339 [00:06<00:01, 173.27batch/s]

Training batches on cuda:  86%|████████▌ | 1154/1339 [00:06<00:01, 170.93batch/s]

Training batches on cuda:  88%|████████▊ | 1172/1339 [00:06<00:00, 168.81batch/s]

Training batches on cuda:  89%|████████▉ | 1189/1339 [00:06<00:00, 168.59batch/s]

Training batches on cuda:  90%|█████████ | 1206/1339 [00:06<00:00, 162.57batch/s]

Training batches on cuda:  92%|█████████▏| 1226/1339 [00:07<00:00, 171.72batch/s]

Training batches on cuda:  93%|█████████▎| 1247/1339 [00:07<00:00, 181.27batch/s]

Training batches on cuda:  95%|█████████▍| 1269/1339 [00:07<00:00, 187.63batch/s]

Training batches on cuda:  96%|█████████▋| 1291/1339 [00:07<00:00, 195.58batch/s]

Training batches on cuda:  98%|█████████▊| 1314/1339 [00:07<00:00, 202.64batch/s]

Training batches on cuda: 100%|█████████▉| 1336/1339 [00:07<00:00, 206.26batch/s]

                                                                                 
Training epochs on cuda:  60%|██████    | 6/10 [00:58<00:33,  8.45s/epoch, loss=0.0132, prev_loss=0.0132]
Training epochs on cuda:  70%|███████   | 7/10 [00:58<00:24,  8.24s/epoch, loss=0.0132, prev_loss=0.0132]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 6/1339 [00:00<00:22, 59.91batch/s]

Training batches on cuda:   2%|▏         | 21/1339 [00:00<00:18, 72.55batch/s]

Training batches on cuda:   3%|▎         | 38/1339 [00:00<00:14, 86.87batch/s]

Training batches on cuda:   4%|▍         | 58/1339 [00:00<00:12, 104.42batch/s]

Training batches on cuda:   6%|▌         | 77/1339 [00:00<00:10, 120.27batch/s]

Training batches on cuda:   7%|▋         | 95/1339 [00:00<00:09, 132.73batch/s]

Training batches on cuda:   9%|▊         | 116/1339 [00:00<00:08, 148.59batch/s]

Training batches on cuda:  10%|█         | 136/1339 [00:00<00:07, 159.40batch/s]

Training batches on cuda:  12%|█▏        | 156/1339 [00:00<00:07, 168.58batch/s]

Training batches on cuda:  13%|█▎        | 175/1339 [00:01<00:06, 173.27batch/s]

Training batches on cuda:  14%|█▍        | 194/1339 [00:01<00:07, 163.12batch/s]

Training batches on cuda:  16%|█▌        | 212/1339 [00:01<00:06, 164.19batch/s]

Training batches on cuda:  17%|█▋        | 229/1339 [00:01<00:06, 165.38batch/s]

Training batches on cuda:  18%|█▊        | 247/1339 [00:01<00:06, 167.02batch/s]

Training batches on cuda:  20%|█▉        | 264/1339 [00:01<00:06, 167.71batch/s]

Training batches on cuda:  21%|██        | 281/1339 [00:01<00:06, 165.29batch/s]

Training batches on cuda:  22%|██▏       | 298/1339 [00:01<00:06, 162.17batch/s]

Training batches on cuda:  24%|██▎       | 315/1339 [00:01<00:06, 162.43batch/s]

Training batches on cuda:  25%|██▍       | 333/1339 [00:01<00:06, 165.17batch/s]

Training batches on cuda:  26%|██▌       | 350/1339 [00:02<00:06, 164.33batch/s]

Training batches on cuda:  27%|██▋       | 367/1339 [00:02<00:06, 153.89batch/s]

Training batches on cuda:  29%|██▊       | 383/1339 [00:02<00:06, 150.08batch/s]

Training batches on cuda:  30%|██▉       | 401/1339 [00:02<00:05, 157.70batch/s]

Training batches on cuda:  31%|███▏      | 420/1339 [00:02<00:05, 163.84batch/s]

Training batches on cuda:  33%|███▎      | 440/1339 [00:02<00:05, 171.64batch/s]

Training batches on cuda:  34%|███▍      | 458/1339 [00:02<00:05, 170.46batch/s]

Training batches on cuda:  36%|███▌      | 476/1339 [00:02<00:05, 153.05batch/s]

Training batches on cuda:  37%|███▋      | 492/1339 [00:03<00:05, 155.04batch/s]

Training batches on cuda:  38%|███▊      | 509/1339 [00:03<00:05, 158.31batch/s]

Training batches on cuda:  39%|███▉      | 527/1339 [00:03<00:04, 163.90batch/s]

Training batches on cuda:  41%|████      | 545/1339 [00:03<00:04, 167.08batch/s]

Training batches on cuda:  42%|████▏     | 562/1339 [00:03<00:04, 165.21batch/s]

Training batches on cuda:  43%|████▎     | 579/1339 [00:03<00:04, 162.35batch/s]

Training batches on cuda:  45%|████▍     | 596/1339 [00:03<00:04, 153.79batch/s]

Training batches on cuda:  46%|████▌     | 612/1339 [00:03<00:04, 148.17batch/s]

Training batches on cuda:  47%|████▋     | 630/1339 [00:03<00:04, 154.90batch/s]

Training batches on cuda:  48%|████▊     | 648/1339 [00:03<00:04, 159.65batch/s]

Training batches on cuda:  50%|████▉     | 667/1339 [00:04<00:04, 165.44batch/s]

Training batches on cuda:  51%|█████     | 686/1339 [00:04<00:03, 171.71batch/s]

Training batches on cuda:  53%|█████▎    | 704/1339 [00:04<00:03, 161.05batch/s]

Training batches on cuda:  54%|█████▍    | 723/1339 [00:04<00:03, 167.25batch/s]

Training batches on cuda:  55%|█████▌    | 742/1339 [00:04<00:03, 170.71batch/s]

Training batches on cuda:  57%|█████▋    | 760/1339 [00:04<00:03, 172.22batch/s]

Training batches on cuda:  58%|█████▊    | 778/1339 [00:04<00:03, 172.00batch/s]

Training batches on cuda:  59%|█████▉    | 796/1339 [00:04<00:03, 173.35batch/s]

Training batches on cuda:  61%|██████    | 814/1339 [00:04<00:03, 167.33batch/s]

Training batches on cuda:  62%|██████▏   | 831/1339 [00:05<00:03, 161.93batch/s]

Training batches on cuda:  63%|██████▎   | 848/1339 [00:05<00:03, 161.92batch/s]

Training batches on cuda:  65%|██████▍   | 867/1339 [00:05<00:02, 167.29batch/s]

Training batches on cuda:  66%|██████▌   | 887/1339 [00:05<00:02, 175.31batch/s]

Training batches on cuda:  68%|██████▊   | 909/1339 [00:05<00:02, 184.49batch/s]

Training batches on cuda:  69%|██████▉   | 929/1339 [00:05<00:02, 187.08batch/s]

Training batches on cuda:  71%|███████   | 948/1339 [00:05<00:02, 186.80batch/s]

Training batches on cuda:  72%|███████▏  | 968/1339 [00:05<00:01, 190.39batch/s]

Training batches on cuda:  74%|███████▍  | 988/1339 [00:05<00:01, 187.18batch/s]

Training batches on cuda:  75%|███████▌  | 1007/1339 [00:06<00:01, 183.10batch/s]

Training batches on cuda:  77%|███████▋  | 1026/1339 [00:06<00:01, 177.52batch/s]

Training batches on cuda:  78%|███████▊  | 1044/1339 [00:06<00:01, 173.99batch/s]

Training batches on cuda:  79%|███████▉  | 1062/1339 [00:06<00:01, 166.68batch/s]

Training batches on cuda:  81%|████████  | 1079/1339 [00:06<00:01, 161.13batch/s]

Training batches on cuda:  82%|████████▏ | 1098/1339 [00:06<00:01, 168.47batch/s]

Training batches on cuda:  83%|████████▎ | 1118/1339 [00:06<00:01, 176.00batch/s]

Training batches on cuda:  85%|████████▍ | 1138/1339 [00:06<00:01, 182.39batch/s]

Training batches on cuda:  87%|████████▋ | 1160/1339 [00:06<00:00, 191.17batch/s]

Training batches on cuda:  88%|████████▊ | 1181/1339 [00:06<00:00, 195.19batch/s]

Training batches on cuda:  90%|████████▉ | 1203/1339 [00:07<00:00, 201.53batch/s]

Training batches on cuda:  91%|█████████▏| 1224/1339 [00:07<00:00, 200.38batch/s]

Training batches on cuda:  93%|█████████▎| 1245/1339 [00:07<00:00, 195.13batch/s]

Training batches on cuda:  94%|█████████▍| 1265/1339 [00:07<00:00, 191.04batch/s]

Training batches on cuda:  96%|█████████▌| 1285/1339 [00:07<00:00, 186.02batch/s]

Training batches on cuda:  97%|█████████▋| 1304/1339 [00:07<00:00, 185.16batch/s]

Training batches on cuda:  99%|█████████▉| 1324/1339 [00:07<00:00, 189.36batch/s]

                                                                                 
Training epochs on cuda:  70%|███████   | 7/10 [01:06<00:24,  8.24s/epoch, loss=0.0132, prev_loss=0.0132]
Training epochs on cuda:  80%|████████  | 8/10 [01:06<00:16,  8.19s/epoch, loss=0.0132, prev_loss=0.0132]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 13/1339 [00:00<00:10, 125.74batch/s]

Training batches on cuda:   3%|▎         | 34/1339 [00:00<00:09, 142.04batch/s]

Training batches on cuda:   4%|▍         | 55/1339 [00:00<00:08, 156.51batch/s]

Training batches on cuda:   6%|▌         | 75/1339 [00:00<00:07, 166.68batch/s]

Training batches on cuda:   7%|▋         | 96/1339 [00:00<00:07, 177.30batch/s]

Training batches on cuda:   9%|▊         | 117/1339 [00:00<00:06, 185.94batch/s]

Training batches on cuda:  10%|█         | 138/1339 [00:00<00:06, 191.33batch/s]

Training batches on cuda:  12%|█▏        | 157/1339 [00:00<00:06, 181.03batch/s]

Training batches on cuda:  13%|█▎        | 175/1339 [00:00<00:06, 174.34batch/s]

Training batches on cuda:  14%|█▍        | 194/1339 [00:01<00:06, 176.27batch/s]

Training batches on cuda:  16%|█▌        | 212/1339 [00:01<00:06, 174.97batch/s]

Training batches on cuda:  17%|█▋        | 230/1339 [00:01<00:06, 174.33batch/s]

Training batches on cuda:  19%|█▊        | 249/1339 [00:01<00:06, 177.91batch/s]

Training batches on cuda:  20%|██        | 268/1339 [00:01<00:05, 178.90batch/s]

Training batches on cuda:  21%|██▏       | 286/1339 [00:01<00:05, 176.84batch/s]

Training batches on cuda:  23%|██▎       | 304/1339 [00:01<00:05, 177.25batch/s]

Training batches on cuda:  24%|██▍       | 322/1339 [00:01<00:05, 170.87batch/s]

Training batches on cuda:  25%|██▌       | 341/1339 [00:01<00:05, 174.74batch/s]

Training batches on cuda:  27%|██▋       | 360/1339 [00:01<00:05, 177.89batch/s]

Training batches on cuda:  28%|██▊       | 379/1339 [00:02<00:05, 179.69batch/s]

Training batches on cuda:  30%|██▉       | 400/1339 [00:02<00:05, 186.63batch/s]

Training batches on cuda:  32%|███▏      | 422/1339 [00:02<00:04, 193.52batch/s]

Training batches on cuda:  33%|███▎      | 442/1339 [00:02<00:04, 189.73batch/s]

Training batches on cuda:  35%|███▍      | 462/1339 [00:02<00:04, 188.48batch/s]

Training batches on cuda:  36%|███▌      | 482/1339 [00:02<00:04, 190.17batch/s]

Training batches on cuda:  37%|███▋      | 502/1339 [00:02<00:04, 192.50batch/s]

Training batches on cuda:  39%|███▉      | 522/1339 [00:02<00:04, 192.49batch/s]

Training batches on cuda:  40%|████      | 542/1339 [00:02<00:04, 189.56batch/s]

Training batches on cuda:  42%|████▏     | 561/1339 [00:03<00:04, 187.53batch/s]

Training batches on cuda:  43%|████▎     | 580/1339 [00:03<00:04, 180.91batch/s]

Training batches on cuda:  45%|████▍     | 599/1339 [00:03<00:04, 183.32batch/s]

Training batches on cuda:  46%|████▌     | 618/1339 [00:03<00:04, 177.38batch/s]

Training batches on cuda:  47%|████▋     | 636/1339 [00:03<00:04, 173.01batch/s]

Training batches on cuda:  49%|████▉     | 654/1339 [00:03<00:03, 172.00batch/s]

Training batches on cuda:  50%|█████     | 672/1339 [00:03<00:03, 172.95batch/s]

Training batches on cuda:  52%|█████▏    | 690/1339 [00:03<00:03, 167.41batch/s]

Training batches on cuda:  53%|█████▎    | 707/1339 [00:03<00:03, 167.19batch/s]

Training batches on cuda:  54%|█████▍    | 725/1339 [00:04<00:03, 169.48batch/s]

Training batches on cuda:  56%|█████▌    | 744/1339 [00:04<00:03, 174.96batch/s]

Training batches on cuda:  57%|█████▋    | 762/1339 [00:04<00:03, 176.26batch/s]

Training batches on cuda:  58%|█████▊    | 780/1339 [00:04<00:03, 171.68batch/s]

Training batches on cuda:  60%|█████▉    | 798/1339 [00:04<00:03, 168.26batch/s]

Training batches on cuda:  61%|██████    | 815/1339 [00:04<00:03, 166.89batch/s]

Training batches on cuda:  62%|██████▏   | 832/1339 [00:04<00:03, 161.52batch/s]

Training batches on cuda:  63%|██████▎   | 850/1339 [00:04<00:02, 164.46batch/s]

Training batches on cuda:  65%|██████▍   | 867/1339 [00:04<00:03, 154.83batch/s]

Training batches on cuda:  66%|██████▌   | 883/1339 [00:04<00:03, 151.07batch/s]

Training batches on cuda:  67%|██████▋   | 899/1339 [00:05<00:02, 147.23batch/s]

Training batches on cuda:  68%|██████▊   | 914/1339 [00:05<00:02, 147.92batch/s]

Training batches on cuda:  69%|██████▉   | 930/1339 [00:05<00:02, 151.09batch/s]

Training batches on cuda:  71%|███████   | 946/1339 [00:05<00:02, 148.43batch/s]

Training batches on cuda:  72%|███████▏  | 968/1339 [00:05<00:02, 162.82batch/s]

Training batches on cuda:  74%|███████▎  | 986/1339 [00:05<00:02, 165.74batch/s]

Training batches on cuda:  75%|███████▍  | 1003/1339 [00:05<00:02, 157.62batch/s]

Training batches on cuda:  76%|███████▋  | 1023/1339 [00:05<00:01, 166.01batch/s]

Training batches on cuda:  78%|███████▊  | 1040/1339 [00:05<00:01, 164.51batch/s]

Training batches on cuda:  79%|███████▉  | 1057/1339 [00:06<00:01, 163.42batch/s]

Training batches on cuda:  80%|████████  | 1074/1339 [00:06<00:01, 163.28batch/s]

Training batches on cuda:  82%|████████▏ | 1092/1339 [00:06<00:01, 166.23batch/s]

Training batches on cuda:  83%|████████▎ | 1109/1339 [00:06<00:01, 167.04batch/s]

Training batches on cuda:  84%|████████▍ | 1127/1339 [00:06<00:01, 168.22batch/s]

Training batches on cuda:  86%|████████▌ | 1146/1339 [00:06<00:01, 171.68batch/s]

Training batches on cuda:  87%|████████▋ | 1164/1339 [00:06<00:01, 170.56batch/s]

Training batches on cuda:  88%|████████▊ | 1182/1339 [00:06<00:00, 167.48batch/s]

Training batches on cuda:  90%|████████▉ | 1203/1339 [00:06<00:00, 178.28batch/s]

Training batches on cuda:  91%|█████████▏| 1222/1339 [00:07<00:00, 178.83batch/s]

Training batches on cuda:  93%|█████████▎| 1241/1339 [00:07<00:00, 181.04batch/s]

Training batches on cuda:  94%|█████████▍| 1260/1339 [00:07<00:00, 176.35batch/s]

Training batches on cuda:  95%|█████████▌| 1278/1339 [00:07<00:00, 174.40batch/s]

Training batches on cuda:  97%|█████████▋| 1296/1339 [00:07<00:00, 173.49batch/s]

Training batches on cuda:  98%|█████████▊| 1314/1339 [00:07<00:00, 170.45batch/s]

Training batches on cuda:  99%|█████████▉| 1332/1339 [00:07<00:00, 172.55batch/s]

                                                                                 
Training epochs on cuda:  80%|████████  | 8/10 [01:14<00:16,  8.19s/epoch, loss=0.0132, prev_loss=0.0132]
Training epochs on cuda:  90%|█████████ | 9/10 [01:14<00:08,  8.09s/epoch, loss=0.0132, prev_loss=0.0132]

Training batches on cuda:   0%|          | 0/1339 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 12/1339 [00:00<00:11, 119.70batch/s]

Training batches on cuda:   2%|▏         | 31/1339 [00:00<00:09, 134.48batch/s]

Training batches on cuda:   4%|▎         | 50/1339 [00:00<00:08, 146.71batch/s]

Training batches on cuda:   5%|▌         | 69/1339 [00:00<00:08, 156.37batch/s]

Training batches on cuda:   6%|▋         | 84/1339 [00:00<00:08, 153.19batch/s]

Training batches on cuda:   8%|▊         | 101/1339 [00:00<00:07, 155.98batch/s]

Training batches on cuda:   9%|▉         | 118/1339 [00:00<00:07, 159.68batch/s]

Training batches on cuda:  10%|█         | 136/1339 [00:00<00:07, 163.79batch/s]

Training batches on cuda:  11%|█▏        | 153/1339 [00:00<00:07, 164.79batch/s]

Training batches on cuda:  13%|█▎        | 170/1339 [00:01<00:07, 165.51batch/s]

Training batches on cuda:  14%|█▍        | 187/1339 [00:01<00:06, 166.25batch/s]

Training batches on cuda:  15%|█▌        | 207/1339 [00:01<00:06, 173.80batch/s]

Training batches on cuda:  17%|█▋        | 225/1339 [00:01<00:06, 173.20batch/s]

Training batches on cuda:  18%|█▊        | 243/1339 [00:01<00:06, 158.88batch/s]

Training batches on cuda:  19%|█▉        | 260/1339 [00:01<00:07, 153.49batch/s]

Training batches on cuda:  21%|██        | 276/1339 [00:01<00:06, 153.93batch/s]

Training batches on cuda:  22%|██▏       | 292/1339 [00:01<00:07, 149.43batch/s]

Training batches on cuda:  23%|██▎       | 309/1339 [00:01<00:06, 154.05batch/s]

Training batches on cuda:  24%|██▍       | 325/1339 [00:02<00:07, 134.31batch/s]

Training batches on cuda:  25%|██▌       | 341/1339 [00:02<00:07, 141.04batch/s]

Training batches on cuda:  27%|██▋       | 359/1339 [00:02<00:06, 150.36batch/s]

Training batches on cuda:  28%|██▊       | 376/1339 [00:02<00:06, 155.72batch/s]

Training batches on cuda:  29%|██▉       | 394/1339 [00:02<00:05, 161.02batch/s]

Training batches on cuda:  31%|███       | 412/1339 [00:02<00:05, 164.79batch/s]

Training batches on cuda:  32%|███▏      | 429/1339 [00:02<00:05, 161.97batch/s]

Training batches on cuda:  33%|███▎      | 446/1339 [00:02<00:05, 163.38batch/s]

Training batches on cuda:  35%|███▍      | 463/1339 [00:02<00:05, 164.92batch/s]

Training batches on cuda:  36%|███▌      | 482/1339 [00:02<00:05, 170.54batch/s]

Training batches on cuda:  37%|███▋      | 500/1339 [00:03<00:04, 173.13batch/s]

Training batches on cuda:  39%|███▊      | 518/1339 [00:03<00:04, 171.47batch/s]

Training batches on cuda:  40%|████      | 536/1339 [00:03<00:04, 170.91batch/s]

Training batches on cuda:  41%|████▏     | 554/1339 [00:03<00:04, 173.28batch/s]

Training batches on cuda:  43%|████▎     | 573/1339 [00:03<00:04, 176.24batch/s]

Training batches on cuda:  44%|████▍     | 593/1339 [00:03<00:04, 181.54batch/s]

Training batches on cuda:  46%|████▌     | 615/1339 [00:03<00:03, 189.62batch/s]

Training batches on cuda:  48%|████▊     | 637/1339 [00:03<00:03, 195.77batch/s]

Training batches on cuda:  49%|████▉     | 659/1339 [00:03<00:03, 198.43batch/s]

Training batches on cuda:  51%|█████     | 680/1339 [00:04<00:03, 201.23batch/s]

Training batches on cuda:  52%|█████▏    | 701/1339 [00:04<00:03, 195.19batch/s]

Training batches on cuda:  54%|█████▍    | 721/1339 [00:04<00:03, 181.52batch/s]

Training batches on cuda:  55%|█████▌    | 740/1339 [00:04<00:03, 183.63batch/s]

Training batches on cuda:  57%|█████▋    | 759/1339 [00:04<00:03, 183.54batch/s]

Training batches on cuda:  58%|█████▊    | 778/1339 [00:04<00:03, 181.62batch/s]

Training batches on cuda:  60%|█████▉    | 797/1339 [00:04<00:02, 181.98batch/s]

Training batches on cuda:  61%|██████    | 816/1339 [00:04<00:02, 183.34batch/s]

Training batches on cuda:  62%|██████▏   | 836/1339 [00:04<00:02, 187.86batch/s]

Training batches on cuda:  64%|██████▍   | 857/1339 [00:04<00:02, 193.90batch/s]

Training batches on cuda:  65%|██████▌   | 877/1339 [00:05<00:02, 193.98batch/s]

Training batches on cuda:  67%|██████▋   | 897/1339 [00:05<00:02, 185.12batch/s]

Training batches on cuda:  68%|██████▊   | 917/1339 [00:05<00:02, 187.32batch/s]

Training batches on cuda:  70%|███████   | 938/1339 [00:05<00:02, 191.69batch/s]

Training batches on cuda:  72%|███████▏  | 958/1339 [00:05<00:02, 185.84batch/s]

Training batches on cuda:  73%|███████▎  | 978/1339 [00:05<00:01, 188.44batch/s]

Training batches on cuda:  75%|███████▍  | 999/1339 [00:05<00:01, 192.31batch/s]

Training batches on cuda:  76%|███████▌  | 1019/1339 [00:05<00:01, 194.42batch/s]

Training batches on cuda:  78%|███████▊  | 1040/1339 [00:05<00:01, 196.65batch/s]

Training batches on cuda:  79%|███████▉  | 1060/1339 [00:06<00:01, 196.74batch/s]

Training batches on cuda:  81%|████████  | 1081/1339 [00:06<00:01, 198.61batch/s]

Training batches on cuda:  82%|████████▏ | 1101/1339 [00:06<00:01, 197.34batch/s]

Training batches on cuda:  84%|████████▎ | 1121/1339 [00:06<00:01, 190.54batch/s]

Training batches on cuda:  85%|████████▌ | 1141/1339 [00:06<00:01, 191.98batch/s]

Training batches on cuda:  87%|████████▋ | 1161/1339 [00:06<00:00, 186.26batch/s]

Training batches on cuda:  88%|████████▊ | 1180/1339 [00:06<00:00, 187.23batch/s]

Training batches on cuda:  90%|████████▉ | 1200/1339 [00:06<00:00, 189.30batch/s]

Training batches on cuda:  91%|█████████ | 1221/1339 [00:06<00:00, 194.73batch/s]

Training batches on cuda:  93%|█████████▎| 1241/1339 [00:06<00:00, 190.65batch/s]

Training batches on cuda:  94%|█████████▍| 1261/1339 [00:07<00:00, 166.85batch/s]

Training batches on cuda:  96%|█████████▌| 1280/1339 [00:07<00:00, 171.93batch/s]

Training batches on cuda:  97%|█████████▋| 1298/1339 [00:07<00:00, 168.15batch/s]

Training batches on cuda:  98%|█████████▊| 1316/1339 [00:07<00:00, 169.37batch/s]

Training batches on cuda: 100%|█████████▉| 1334/1339 [00:07<00:00, 170.31batch/s]

                                                                                 
Training epochs on cuda:  90%|█████████ | 9/10 [01:22<00:08,  8.09s/epoch, loss=0.0132, prev_loss=0.0132]2021-09-16:18:02:12,812 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:02:13,816 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:02:28,756 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:18:02:28,887 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00, 12.84s/epoch, loss=0.0132, prev_loss=0.0132]
Training epochs on cuda: 100%|██████████| 10/10 [01:38<00:00,  9.86s/epoch, loss=0.0132, prev_loss=0.0132]
2021-09-16:18:02:28,887 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpbrzg67nk'
2021-09-16:18:02:28,976 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbrzg67nk' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.66ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.66ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.66ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.66ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:18:02:43,753 INFO     [evaluator.py:657] Evaluation took 14.76s seconds
[I 2021-09-16 18:02:43,758] Trial 25 finished with value: -0.0020107761750951525 and parameters: {'loss.margin': 3, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.25951438766898416, 'optimizer.lr': 0.03986891478966269, 'negative_sampler.num_negs_per_pos': 10, 'training.batch_size': 232}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:02:43,778 WARNING  [api.py:823] No random seed is specified. Setting to 245712849.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 9/374 [00:00<00:04, 88.50batch/s]

Training batches on cuda:   6%|▋         | 24/374 [00:00<00:03, 100.15batch/s]

Training batches on cuda:  11%|█         | 40/374 [00:00<00:02, 111.65batch/s]

Training batches on cuda:  14%|█▍        | 53/374 [00:00<00:02, 115.11batch/s]

Training batches on cuda:  18%|█▊        | 66/374 [00:00<00:02, 117.78batch/s]

Training batches on cuda:  21%|██        | 79/374 [00:00<00:02, 120.73batch/s]

Training batches on cuda:  24%|██▍       | 91/374 [00:00<00:02, 120.44batch/s]

Training batches on cuda:  28%|██▊       | 103/374 [00:00<00:02, 118.52batch/s]

Training batches on cuda:  31%|███       | 116/374 [00:00<00:02, 120.32batch/s]

Training batches on cuda:  34%|███▍      | 129/374 [00:01<00:02, 121.25batch/s]

Training batches on cuda:  38%|███▊      | 143/374 [00:01<00:01, 125.48batch/s]

Training batches on cuda:  42%|████▏     | 158/374 [00:01<00:01, 131.80batch/s]

Training batches on cuda:  47%|████▋     | 174/374 [00:01<00:01, 137.41batch/s]

Training batches on cuda:  50%|█████     | 188/374 [00:01<00:01, 134.42batch/s]

Training batches on cuda:  54%|█████▍    | 202/374 [00:01<00:01, 130.21batch/s]

Training batches on cuda:  58%|█████▊    | 216/374 [00:01<00:01, 126.47batch/s]

Training batches on cuda:  61%|██████    | 229/374 [00:01<00:01, 124.96batch/s]

Training batches on cuda:  65%|██████▍   | 242/374 [00:01<00:01, 125.14batch/s]

Training batches on cuda:  68%|██████▊   | 255/374 [00:02<00:00, 122.60batch/s]

Training batches on cuda:  72%|███████▏  | 268/374 [00:02<00:00, 117.90batch/s]

Training batches on cuda:  75%|███████▍  | 280/374 [00:02<00:00, 115.58batch/s]

Training batches on cuda:  78%|███████▊  | 292/374 [00:02<00:00, 113.27batch/s]

Training batches on cuda:  82%|████████▏ | 306/374 [00:02<00:00, 118.10batch/s]

Training batches on cuda:  85%|████████▌ | 318/374 [00:02<00:00, 113.30batch/s]

Training batches on cuda:  88%|████████▊ | 330/374 [00:02<00:00, 110.27batch/s]

Training batches on cuda:  92%|█████████▏| 344/374 [00:02<00:00, 116.56batch/s]

Training batches on cuda:  96%|█████████▌| 358/374 [00:02<00:00, 121.12batch/s]

Training batches on cuda:  99%|█████████▉| 372/374 [00:03<00:00, 123.91batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.00242, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:28,  3.20s/epoch, loss=0.00242, prev_loss=nan]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 10/374 [00:00<00:03, 94.50batch/s]

Training batches on cuda:   6%|▌         | 23/374 [00:00<00:03, 101.86batch/s]

Training batches on cuda:   8%|▊         | 31/374 [00:00<00:03, 93.00batch/s] 

Training batches on cuda:  12%|█▏        | 45/374 [00:00<00:03, 103.01batch/s]

Training batches on cuda:  16%|█▌        | 60/374 [00:00<00:02, 113.23batch/s]

Training batches on cuda:  20%|█▉        | 74/374 [00:00<00:02, 119.43batch/s]

Training batches on cuda:  24%|██▎       | 88/374 [00:00<00:02, 123.08batch/s]

Training batches on cuda:  28%|██▊       | 104/374 [00:00<00:02, 129.58batch/s]

Training batches on cuda:  31%|███▏      | 117/374 [00:00<00:01, 129.10batch/s]

Training batches on cuda:  35%|███▍      | 130/374 [00:01<00:01, 127.43batch/s]

Training batches on cuda:  38%|███▊      | 143/374 [00:01<00:01, 124.60batch/s]

Training batches on cuda:  42%|████▏     | 156/374 [00:01<00:01, 118.49batch/s]

Training batches on cuda:  45%|████▍     | 168/374 [00:01<00:01, 118.89batch/s]

Training batches on cuda:  48%|████▊     | 180/374 [00:01<00:01, 115.55batch/s]

Training batches on cuda:  52%|█████▏    | 193/374 [00:01<00:01, 119.28batch/s]

Training batches on cuda:  55%|█████▌    | 206/374 [00:01<00:01, 121.50batch/s]

Training batches on cuda:  59%|█████▉    | 221/374 [00:01<00:01, 128.28batch/s]

Training batches on cuda:  63%|██████▎   | 235/374 [00:01<00:01, 128.20batch/s]

Training batches on cuda:  67%|██████▋   | 249/374 [00:01<00:00, 131.04batch/s]

Training batches on cuda:  70%|███████   | 263/374 [00:02<00:00, 133.04batch/s]

Training batches on cuda:  74%|███████▍  | 278/374 [00:02<00:00, 136.37batch/s]

Training batches on cuda:  79%|███████▊  | 294/374 [00:02<00:00, 141.03batch/s]

Training batches on cuda:  83%|████████▎ | 310/374 [00:02<00:00, 144.33batch/s]

Training batches on cuda:  87%|████████▋ | 325/374 [00:02<00:00, 145.81batch/s]

Training batches on cuda:  91%|█████████ | 341/374 [00:02<00:00, 148.08batch/s]

Training batches on cuda:  95%|█████████▌| 357/374 [00:02<00:00, 148.83batch/s]

Training batches on cuda:  99%|█████████▉| 372/374 [00:02<00:00, 146.54batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:28,  3.20s/epoch, loss=0.00241, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:25,  3.15s/epoch, loss=0.00241, prev_loss=nan]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 11/374 [00:00<00:03, 106.56batch/s]

Training batches on cuda:   6%|▋         | 24/374 [00:00<00:03, 112.62batch/s]

Training batches on cuda:  10%|█         | 39/374 [00:00<00:02, 119.95batch/s]

Training batches on cuda:  14%|█▍        | 53/374 [00:00<00:02, 125.10batch/s]

Training batches on cuda:  17%|█▋        | 65/374 [00:00<00:02, 122.98batch/s]

Training batches on cuda:  20%|██        | 76/374 [00:00<00:02, 117.09batch/s]

Training batches on cuda:  24%|██▎       | 88/374 [00:00<00:02, 117.13batch/s]

Training batches on cuda:  27%|██▋       | 102/374 [00:00<00:02, 122.26batch/s]

Training batches on cuda:  31%|███▏      | 117/374 [00:00<00:02, 127.74batch/s]

Training batches on cuda:  35%|███▌      | 132/374 [00:01<00:01, 131.25batch/s]

Training batches on cuda:  39%|███▉      | 145/374 [00:01<00:01, 129.49batch/s]

Training batches on cuda:  42%|████▏     | 158/374 [00:01<00:01, 123.06batch/s]

Training batches on cuda:  46%|████▌     | 171/374 [00:01<00:01, 119.85batch/s]

Training batches on cuda:  49%|████▉     | 183/374 [00:01<00:01, 119.12batch/s]

Training batches on cuda:  52%|█████▏    | 196/374 [00:01<00:01, 121.24batch/s]

Training batches on cuda:  56%|█████▌    | 209/374 [00:01<00:01, 121.75batch/s]

Training batches on cuda:  59%|█████▉    | 222/374 [00:01<00:01, 119.51batch/s]

Training batches on cuda:  63%|██████▎   | 235/374 [00:01<00:01, 121.56batch/s]

Training batches on cuda:  66%|██████▋   | 248/374 [00:02<00:01, 121.14batch/s]

Training batches on cuda:  70%|██████▉   | 261/374 [00:02<00:00, 120.49batch/s]

Training batches on cuda:  73%|███████▎  | 274/374 [00:02<00:00, 119.61batch/s]

Training batches on cuda:  76%|███████▋  | 286/374 [00:02<00:00, 105.94batch/s]

Training batches on cuda:  80%|████████  | 300/374 [00:02<00:00, 113.47batch/s]

Training batches on cuda:  84%|████████▍ | 314/374 [00:02<00:00, 119.14batch/s]

Training batches on cuda:  88%|████████▊ | 329/374 [00:02<00:00, 125.27batch/s]

Training batches on cuda:  92%|█████████▏| 343/374 [00:02<00:00, 128.73batch/s]

Training batches on cuda:  96%|█████████▌| 358/374 [00:02<00:00, 132.80batch/s]

Training batches on cuda:  99%|█████████▉| 372/374 [00:02<00:00, 133.94batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:25,  3.15s/epoch, loss=0.00241, prev_loss=0.00241]
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:22,  3.18s/epoch, loss=0.00241, prev_loss=0.00241]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/374 [00:00<00:04, 76.19batch/s]

Training batches on cuda:   6%|▋         | 24/374 [00:00<00:03, 89.82batch/s]

Training batches on cuda:  10%|█         | 39/374 [00:00<00:03, 102.11batch/s]

Training batches on cuda:  15%|█▍        | 55/374 [00:00<00:02, 113.11batch/s]

Training batches on cuda:  18%|█▊        | 69/374 [00:00<00:02, 118.25batch/s]

Training batches on cuda:  22%|██▏       | 83/374 [00:00<00:02, 122.20batch/s]

Training batches on cuda:  26%|██▌       | 96/374 [00:00<00:02, 123.35batch/s]

Training batches on cuda:  29%|██▉       | 110/374 [00:00<00:02, 126.64batch/s]

Training batches on cuda:  33%|███▎      | 123/374 [00:00<00:01, 127.26batch/s]

Training batches on cuda:  37%|███▋      | 137/374 [00:01<00:01, 128.70batch/s]

Training batches on cuda:  41%|████      | 152/374 [00:01<00:01, 132.06batch/s]

Training batches on cuda:  45%|████▍     | 167/374 [00:01<00:01, 136.16batch/s]

Training batches on cuda:  49%|████▊     | 182/374 [00:01<00:01, 138.01batch/s]

Training batches on cuda:  52%|█████▏    | 196/374 [00:01<00:01, 134.22batch/s]

Training batches on cuda:  57%|█████▋    | 212/374 [00:01<00:01, 139.16batch/s]

Training batches on cuda:  61%|██████    | 228/374 [00:01<00:01, 142.33batch/s]

Training batches on cuda:  65%|██████▌   | 244/374 [00:01<00:00, 145.74batch/s]

Training batches on cuda:  69%|██████▉   | 259/374 [00:01<00:00, 145.85batch/s]

Training batches on cuda:  74%|███████▍  | 277/374 [00:01<00:00, 152.59batch/s]

Training batches on cuda:  78%|███████▊  | 293/374 [00:02<00:00, 153.48batch/s]

Training batches on cuda:  83%|████████▎ | 309/374 [00:02<00:00, 150.41batch/s]

Training batches on cuda:  87%|████████▋ | 325/374 [00:02<00:00, 149.11batch/s]

Training batches on cuda:  91%|█████████ | 341/374 [00:02<00:00, 151.85batch/s]

Training batches on cuda:  96%|█████████▌| 358/374 [00:02<00:00, 156.21batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:22,  3.18s/epoch, loss=0.00241, prev_loss=0.00241]
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:18,  3.06s/epoch, loss=0.00241, prev_loss=0.00241]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 5/374 [00:00<00:07, 46.45batch/s]

Training batches on cuda:   5%|▍         | 18/374 [00:00<00:06, 57.40batch/s]

Training batches on cuda:   9%|▊         | 32/374 [00:00<00:04, 69.08batch/s]

Training batches on cuda:  12%|█▏        | 46/374 [00:00<00:04, 80.49batch/s]

Training batches on cuda:  16%|█▌        | 60/374 [00:00<00:03, 91.18batch/s]

Training batches on cuda:  20%|█▉        | 73/374 [00:00<00:03, 99.77batch/s]

Training batches on cuda:  23%|██▎       | 85/374 [00:00<00:02, 101.41batch/s]

Training batches on cuda:  27%|██▋       | 100/374 [00:00<00:02, 111.26batch/s]

Training batches on cuda:  30%|███       | 113/374 [00:00<00:02, 115.40batch/s]

Training batches on cuda:  34%|███▍      | 128/374 [00:01<00:02, 121.16batch/s]

Training batches on cuda:  38%|███▊      | 143/374 [00:01<00:01, 127.62batch/s]

Training batches on cuda:  42%|████▏     | 157/374 [00:01<00:01, 129.54batch/s]

Training batches on cuda:  46%|████▋     | 173/374 [00:01<00:01, 135.70batch/s]

Training batches on cuda:  50%|█████     | 187/374 [00:01<00:01, 131.99batch/s]

Training batches on cuda:  54%|█████▎    | 201/374 [00:01<00:01, 130.35batch/s]

Training batches on cuda:  57%|█████▋    | 215/374 [00:01<00:01, 131.35batch/s]

Training batches on cuda:  62%|██████▏   | 231/374 [00:01<00:01, 137.86batch/s]

Training batches on cuda:  66%|██████▌   | 247/374 [00:01<00:00, 141.72batch/s]

Training batches on cuda:  70%|███████   | 262/374 [00:02<00:00, 141.54batch/s]

Training batches on cuda:  74%|███████▍  | 277/374 [00:02<00:00, 130.30batch/s]

Training batches on cuda:  78%|███████▊  | 291/374 [00:02<00:00, 132.88batch/s]

Training batches on cuda:  82%|████████▏ | 305/374 [00:02<00:00, 130.78batch/s]

Training batches on cuda:  86%|████████▌ | 320/374 [00:02<00:00, 135.26batch/s]

Training batches on cuda:  90%|████████▉ | 336/374 [00:02<00:00, 139.55batch/s]

Training batches on cuda:  94%|█████████▍| 351/374 [00:02<00:00, 142.00batch/s]

Training batches on cuda:  98%|█████████▊| 366/374 [00:02<00:00, 144.07batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:18,  3.06s/epoch, loss=0.00241, prev_loss=0.00241]
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:15,  3.05s/epoch, loss=0.00241, prev_loss=0.00241]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 10/374 [00:00<00:03, 96.37batch/s]

Training batches on cuda:   7%|▋         | 25/374 [00:00<00:03, 106.87batch/s]

Training batches on cuda:  11%|█         | 40/374 [00:00<00:02, 115.26batch/s]

Training batches on cuda:  15%|█▍        | 56/374 [00:00<00:02, 124.90batch/s]

Training batches on cuda:  19%|█▉        | 72/374 [00:00<00:02, 131.58batch/s]

Training batches on cuda:  23%|██▎       | 87/374 [00:00<00:02, 134.87batch/s]

Training batches on cuda:  27%|██▋       | 102/374 [00:00<00:01, 138.94batch/s]

Training batches on cuda:  31%|███▏      | 117/374 [00:00<00:01, 141.87batch/s]

Training batches on cuda:  36%|███▌      | 133/374 [00:00<00:01, 144.23batch/s]

Training batches on cuda:  40%|███▉      | 148/374 [00:01<00:01, 143.52batch/s]

Training batches on cuda:  44%|████▍     | 164/374 [00:01<00:01, 145.63batch/s]

Training batches on cuda:  48%|████▊     | 180/374 [00:01<00:01, 147.50batch/s]

Training batches on cuda:  52%|█████▏    | 195/374 [00:01<00:01, 146.88batch/s]

Training batches on cuda:  56%|█████▌    | 210/374 [00:01<00:01, 144.93batch/s]

Training batches on cuda:  60%|██████    | 225/374 [00:01<00:01, 141.59batch/s]

Training batches on cuda:  64%|██████▍   | 240/374 [00:01<00:01, 129.70batch/s]

Training batches on cuda:  68%|██████▊   | 255/374 [00:01<00:00, 134.20batch/s]

Training batches on cuda:  72%|███████▏  | 270/374 [00:01<00:00, 136.18batch/s]

Training batches on cuda:  76%|███████▌  | 284/374 [00:02<00:00, 133.58batch/s]

Training batches on cuda:  80%|███████▉  | 298/374 [00:02<00:00, 130.49batch/s]

Training batches on cuda:  83%|████████▎ | 312/374 [00:02<00:00, 125.60batch/s]

Training batches on cuda:  87%|████████▋ | 327/374 [00:02<00:00, 131.46batch/s]

Training batches on cuda:  91%|█████████▏| 342/374 [00:02<00:00, 135.33batch/s]

Training batches on cuda:  95%|█████████▌| 356/374 [00:02<00:00, 131.90batch/s]

Training batches on cuda:  99%|█████████▉| 370/374 [00:02<00:00, 129.18batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:15,  3.05s/epoch, loss=0.00241, prev_loss=0.00241]
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:12,  3.02s/epoch, loss=0.00241, prev_loss=0.00241]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 10/374 [00:00<00:03, 94.24batch/s]

Training batches on cuda:   6%|▋         | 24/374 [00:00<00:03, 103.12batch/s]

Training batches on cuda:   9%|▉         | 34/374 [00:00<00:03, 101.70batch/s]

Training batches on cuda:  11%|█▏        | 43/374 [00:00<00:03, 96.16batch/s] 

Training batches on cuda:  15%|█▌        | 57/374 [00:00<00:03, 104.79batch/s]

Training batches on cuda:  19%|█▉        | 71/374 [00:00<00:02, 111.41batch/s]

Training batches on cuda:  23%|██▎       | 85/374 [00:00<00:02, 117.25batch/s]

Training batches on cuda:  27%|██▋       | 100/374 [00:00<00:02, 124.73batch/s]

Training batches on cuda:  30%|███       | 113/374 [00:00<00:02, 121.19batch/s]

Training batches on cuda:  34%|███▎      | 126/374 [00:01<00:02, 123.61batch/s]

Training batches on cuda:  37%|███▋      | 139/374 [00:01<00:01, 123.91batch/s]

Training batches on cuda:  41%|████      | 152/374 [00:01<00:01, 125.56batch/s]

Training batches on cuda:  44%|████▍     | 166/374 [00:01<00:01, 129.51batch/s]

Training batches on cuda:  48%|████▊     | 179/374 [00:01<00:01, 128.65batch/s]

Training batches on cuda:  51%|█████▏    | 192/374 [00:01<00:01, 125.42batch/s]

Training batches on cuda:  55%|█████▍    | 205/374 [00:01<00:01, 123.65batch/s]

Training batches on cuda:  59%|█████▊    | 219/374 [00:01<00:01, 125.84batch/s]

Training batches on cuda:  62%|██████▏   | 233/374 [00:01<00:01, 129.52batch/s]

Training batches on cuda:  66%|██████▌   | 246/374 [00:01<00:00, 128.51batch/s]

Training batches on cuda:  69%|██████▉   | 259/374 [00:02<00:00, 123.65batch/s]

Training batches on cuda:  73%|███████▎  | 273/374 [00:02<00:00, 127.02batch/s]

Training batches on cuda:  76%|███████▋  | 286/374 [00:02<00:00, 126.47batch/s]

Training batches on cuda:  80%|████████  | 301/374 [00:02<00:00, 130.98batch/s]

Training batches on cuda:  84%|████████▍ | 315/374 [00:02<00:00, 123.90batch/s]

Training batches on cuda:  88%|████████▊ | 328/374 [00:02<00:00, 112.30batch/s]

Training batches on cuda:  91%|█████████ | 340/374 [00:02<00:00, 112.31batch/s]

Training batches on cuda:  94%|█████████▍| 352/374 [00:02<00:00, 112.19batch/s]

Training batches on cuda:  97%|█████████▋| 364/374 [00:03<00:00, 113.86batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:21<00:12,  3.02s/epoch, loss=0.00241, prev_loss=0.00241]
Training epochs on cuda:  70%|███████   | 7/10 [00:21<00:09,  3.09s/epoch, loss=0.00241, prev_loss=0.00241]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/374 [00:00<00:04, 78.71batch/s]

Training batches on cuda:   6%|▌         | 21/374 [00:00<00:04, 88.17batch/s]

Training batches on cuda:   9%|▉         | 34/374 [00:00<00:03, 96.78batch/s]

Training batches on cuda:  13%|█▎        | 47/374 [00:00<00:03, 104.79batch/s]

Training batches on cuda:  16%|█▌        | 60/374 [00:00<00:02, 109.24batch/s]

Training batches on cuda:  20%|█▉        | 74/374 [00:00<00:02, 116.91batch/s]

Training batches on cuda:  24%|██▍       | 89/374 [00:00<00:02, 124.05batch/s]

Training batches on cuda:  27%|██▋       | 102/374 [00:00<00:02, 121.31batch/s]

Training batches on cuda:  31%|███       | 115/374 [00:00<00:02, 121.54batch/s]

Training batches on cuda:  34%|███▍      | 128/374 [00:01<00:02, 122.52batch/s]

Training batches on cuda:  38%|███▊      | 141/374 [00:01<00:01, 119.99batch/s]

Training batches on cuda:  42%|████▏     | 156/374 [00:01<00:01, 125.55batch/s]

Training batches on cuda:  46%|████▌     | 171/374 [00:01<00:01, 130.52batch/s]

Training batches on cuda:  50%|████▉     | 186/374 [00:01<00:01, 133.58batch/s]

Training batches on cuda:  54%|█████▎    | 201/374 [00:01<00:01, 135.33batch/s]

Training batches on cuda:  57%|█████▋    | 215/374 [00:01<00:01, 136.54batch/s]

Training batches on cuda:  61%|██████    | 229/374 [00:01<00:01, 137.24batch/s]

Training batches on cuda:  65%|██████▌   | 244/374 [00:01<00:00, 139.76batch/s]

Training batches on cuda:  69%|██████▉   | 259/374 [00:02<00:00, 135.47batch/s]

Training batches on cuda:  73%|███████▎  | 273/374 [00:02<00:00, 131.96batch/s]

Training batches on cuda:  77%|███████▋  | 287/374 [00:02<00:00, 119.51batch/s]

Training batches on cuda:  80%|████████  | 300/374 [00:02<00:00, 122.47batch/s]

Training batches on cuda:  84%|████████▎ | 313/374 [00:02<00:00, 121.74batch/s]

Training batches on cuda:  87%|████████▋ | 326/374 [00:02<00:00, 119.86batch/s]

Training batches on cuda:  91%|█████████ | 339/374 [00:02<00:00, 119.35batch/s]

Training batches on cuda:  94%|█████████▍| 352/374 [00:02<00:00, 119.03batch/s]

Training batches on cuda:  97%|█████████▋| 364/374 [00:02<00:00, 115.03batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:24<00:09,  3.09s/epoch, loss=0.00241, prev_loss=0.00241]
Training epochs on cuda:  80%|████████  | 8/10 [00:24<00:06,  3.12s/epoch, loss=0.00241, prev_loss=0.00241]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 7/374 [00:00<00:05, 68.84batch/s]

Training batches on cuda:   5%|▍         | 18/374 [00:00<00:04, 76.38batch/s]

Training batches on cuda:   8%|▊         | 31/374 [00:00<00:03, 86.20batch/s]

Training batches on cuda:  12%|█▏        | 46/374 [00:00<00:03, 98.69batch/s]

Training batches on cuda:  16%|█▌        | 60/374 [00:00<00:02, 108.13batch/s]

Training batches on cuda:  20%|██        | 75/374 [00:00<00:02, 117.81batch/s]

Training batches on cuda:  24%|██▍       | 89/374 [00:00<00:02, 122.83batch/s]

Training batches on cuda:  28%|██▊       | 104/374 [00:00<00:02, 128.63batch/s]

Training batches on cuda:  32%|███▏      | 119/374 [00:00<00:01, 132.68batch/s]

Training batches on cuda:  36%|███▌      | 134/374 [00:01<00:01, 135.93batch/s]

Training batches on cuda:  40%|███▉      | 149/374 [00:01<00:01, 136.60batch/s]

Training batches on cuda:  44%|████▍     | 165/374 [00:01<00:01, 141.10batch/s]

Training batches on cuda:  48%|████▊     | 181/374 [00:01<00:01, 143.86batch/s]

Training batches on cuda:  53%|█████▎    | 197/374 [00:01<00:01, 147.81batch/s]

Training batches on cuda:  57%|█████▋    | 212/374 [00:01<00:01, 146.21batch/s]

Training batches on cuda:  61%|██████    | 227/374 [00:01<00:01, 143.66batch/s]

Training batches on cuda:  65%|██████▍   | 242/374 [00:01<00:00, 144.67batch/s]

Training batches on cuda:  69%|██████▊   | 257/374 [00:01<00:00, 143.51batch/s]

Training batches on cuda:  73%|███████▎  | 272/374 [00:02<00:00, 134.25batch/s]

Training batches on cuda:  76%|███████▋  | 286/374 [00:02<00:00, 130.73batch/s]

Training batches on cuda:  80%|████████  | 300/374 [00:02<00:00, 126.66batch/s]

Training batches on cuda:  84%|████████▎ | 313/374 [00:02<00:00, 126.37batch/s]

Training batches on cuda:  87%|████████▋ | 326/374 [00:02<00:00, 126.24batch/s]

Training batches on cuda:  91%|█████████ | 340/374 [00:02<00:00, 127.66batch/s]

Training batches on cuda:  95%|█████████▍| 354/374 [00:02<00:00, 128.29batch/s]

Training batches on cuda:  98%|█████████▊| 368/374 [00:02<00:00, 128.94batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:27<00:06,  3.12s/epoch, loss=0.00241, prev_loss=0.00241]
Training epochs on cuda:  90%|█████████ | 9/10 [00:27<00:03,  3.08s/epoch, loss=0.00241, prev_loss=0.00241]

Training batches on cuda:   0%|          | 0/374 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 7/374 [00:00<00:05, 65.89batch/s]

Training batches on cuda:   5%|▌         | 19/374 [00:00<00:04, 76.02batch/s]

Training batches on cuda:   9%|▉         | 33/374 [00:00<00:03, 87.05batch/s]

Training batches on cuda:  12%|█▏        | 46/374 [00:00<00:03, 96.44batch/s]

Training batches on cuda:  16%|█▌        | 60/374 [00:00<00:02, 105.27batch/s]

Training batches on cuda:  20%|█▉        | 74/374 [00:00<00:02, 113.51batch/s]

Training batches on cuda:  23%|██▎       | 87/374 [00:00<00:02, 116.96batch/s]

Training batches on cuda:  27%|██▋       | 100/374 [00:00<00:02, 119.33batch/s]

Training batches on cuda:  30%|███       | 114/374 [00:00<00:02, 122.76batch/s]

Training batches on cuda:  34%|███▍      | 127/374 [00:01<00:01, 124.28batch/s]

Training batches on cuda:  37%|███▋      | 140/374 [00:01<00:02, 111.31batch/s]

Training batches on cuda:  41%|████      | 154/374 [00:01<00:01, 117.57batch/s]

Training batches on cuda:  45%|████▌     | 170/374 [00:01<00:01, 126.01batch/s]

Training batches on cuda:  49%|████▉     | 185/374 [00:01<00:01, 129.63batch/s]

Training batches on cuda:  53%|█████▎    | 199/374 [00:01<00:01, 128.15batch/s]

Training batches on cuda:  57%|█████▋    | 213/374 [00:01<00:01, 127.75batch/s]

Training batches on cuda:  60%|██████    | 226/374 [00:01<00:01, 126.65batch/s]

Training batches on cuda:  64%|██████▍   | 239/374 [00:01<00:01, 127.54batch/s]

Training batches on cuda:  68%|██████▊   | 254/374 [00:02<00:00, 131.50batch/s]

Training batches on cuda:  72%|███████▏  | 270/374 [00:02<00:00, 137.18batch/s]

Training batches on cuda:  76%|███████▌  | 285/374 [00:02<00:00, 140.09batch/s]

Training batches on cuda:  80%|████████  | 300/374 [00:02<00:00, 141.96batch/s]

Training batches on cuda:  84%|████████▍ | 315/374 [00:02<00:00, 143.49batch/s]

Training batches on cuda:  88%|████████▊ | 330/374 [00:02<00:00, 144.69batch/s]

Training batches on cuda:  92%|█████████▏| 345/374 [00:02<00:00, 145.34batch/s]

Training batches on cuda:  97%|█████████▋| 361/374 [00:02<00:00, 146.99batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:30<00:03,  3.08s/epoch, loss=0.00241, prev_loss=0.00241]2021-09-16:18:03:14,903 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:03:15,908 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:03:30,862 INFO     [evaluator.py:657] Evaluation took 14.95s seconds
2021-09-16:18:03:31,4 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:46<00:00,  7.90s/epoch, loss=0.00241, prev_loss=0.00241]
Training epochs on cuda: 100%|██████████| 10/10 [00:46<00:00,  4.68s/epoch, loss=0.00241, prev_loss=0.00241]
2021-09-16:18:03:31,5 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp5qfwdv3_'
2021-09-16:18:03:31,76 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5qfwdv3_' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.35ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:05, 2.52ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.55ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.58ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.60ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.61ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.62ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:03:45,914 INFO     [evaluator.py:657] Evaluation took 14.82s seconds
[I 2021-09-16 18:03:45,919] Trial 26 finished with value: 0.02663455114467783 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.11635380352210326, 'optimizer.lr': 0.010133491948168614, 'negative_sampler.num_negs_per_pos': 2, 'training.batch_size': 832}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:03:45,939 WARNING  [api.py:823] No random seed is specified. Setting to 1533832958.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:02, 55.57batch/s]

Training batches on cuda:   9%|▉         | 15/170 [00:00<00:02, 61.76batch/s]

Training batches on cuda:  14%|█▍        | 24/170 [00:00<00:02, 67.73batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:00<00:01, 73.22batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:00<00:01, 77.66batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:00<00:01, 77.82batch/s]

Training batches on cuda:  35%|███▌      | 60/170 [00:00<00:01, 78.10batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:00<00:01, 77.28batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:00<00:01, 74.41batch/s]

Training batches on cuda:  50%|█████     | 85/170 [00:01<00:01, 72.02batch/s]

Training batches on cuda:  55%|█████▍    | 93/170 [00:01<00:01, 71.50batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:01<00:00, 72.90batch/s]

Training batches on cuda:  64%|██████▍   | 109/170 [00:01<00:00, 73.67batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:01<00:00, 74.48batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:01<00:00, 75.05batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:01<00:00, 77.04batch/s]

Training batches on cuda:  84%|████████▎ | 142/170 [00:01<00:00, 76.00batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:01<00:00, 74.14batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:02<00:00, 73.64batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:02<00:00, 76.20batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000744, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:21,  2.41s/epoch, loss=0.000744, prev_loss=nan]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:03, 46.42batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:02, 52.49batch/s]

Training batches on cuda:  12%|█▏        | 21/170 [00:00<00:02, 58.09batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:00<00:02, 62.57batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:00<00:01, 67.30batch/s]

Training batches on cuda:  27%|██▋       | 46/170 [00:00<00:01, 69.84batch/s]

Training batches on cuda:  32%|███▏      | 54/170 [00:00<00:01, 71.02batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:00<00:01, 72.84batch/s]

Training batches on cuda:  41%|████      | 70/170 [00:00<00:01, 68.60batch/s]

Training batches on cuda:  46%|████▌     | 78/170 [00:01<00:01, 71.13batch/s]

Training batches on cuda:  51%|█████     | 87/170 [00:01<00:01, 74.98batch/s]

Training batches on cuda:  56%|█████▋    | 96/170 [00:01<00:00, 78.34batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:01<00:00, 81.08batch/s]

Training batches on cuda:  67%|██████▋   | 114/170 [00:01<00:00, 82.61batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:01<00:00, 84.27batch/s]

Training batches on cuda:  78%|███████▊  | 132/170 [00:01<00:00, 85.10batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:01<00:00, 86.01batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:01<00:00, 85.93batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:02<00:00, 86.16batch/s]

Training batches on cuda:  99%|█████████▉| 168/170 [00:02<00:00, 77.81batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:21,  2.41s/epoch, loss=0.000729, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:19,  2.40s/epoch, loss=0.000729, prev_loss=nan]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:04, 39.41batch/s]

Training batches on cuda:   6%|▋         | 11/170 [00:00<00:03, 44.18batch/s]

Training batches on cuda:  11%|█         | 19/170 [00:00<00:02, 50.83batch/s]

Training batches on cuda:  16%|█▋        | 28/170 [00:00<00:02, 57.06batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:00<00:02, 59.64batch/s]

Training batches on cuda:  25%|██▌       | 43/170 [00:00<00:02, 62.19batch/s]

Training batches on cuda:  30%|███       | 51/170 [00:00<00:01, 65.31batch/s]

Training batches on cuda:  34%|███▍      | 58/170 [00:00<00:01, 66.29batch/s]

Training batches on cuda:  39%|███▉      | 66/170 [00:00<00:01, 67.89batch/s]

Training batches on cuda:  44%|████▍     | 75/170 [00:01<00:01, 71.51batch/s]

Training batches on cuda:  49%|████▉     | 84/170 [00:01<00:01, 75.37batch/s]

Training batches on cuda:  55%|█████▍    | 93/170 [00:01<00:00, 77.18batch/s]

Training batches on cuda:  60%|██████    | 102/170 [00:01<00:00, 79.02batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:01<00:00, 80.72batch/s]

Training batches on cuda:  71%|███████   | 120/170 [00:01<00:00, 81.26batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:01<00:00, 82.18batch/s]

Training batches on cuda:  81%|████████  | 138/170 [00:01<00:00, 82.69batch/s]

Training batches on cuda:  86%|████████▋ | 147/170 [00:01<00:00, 82.75batch/s]

Training batches on cuda:  92%|█████████▏| 156/170 [00:02<00:00, 82.69batch/s]

Training batches on cuda:  97%|█████████▋| 165/170 [00:02<00:00, 82.98batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:19,  2.40s/epoch, loss=0.00073, prev_loss=0.000729]
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:16,  2.41s/epoch, loss=0.00073, prev_loss=0.000729]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:02, 55.87batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:02, 63.19batch/s]

Training batches on cuda:  15%|█▌        | 26/170 [00:00<00:02, 69.43batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:00<00:01, 74.01batch/s]

Training batches on cuda:  26%|██▋       | 45/170 [00:00<00:01, 78.24batch/s]

Training batches on cuda:  32%|███▏      | 54/170 [00:00<00:01, 81.10batch/s]

Training batches on cuda:  37%|███▋      | 63/170 [00:00<00:01, 83.21batch/s]

Training batches on cuda:  42%|████▏     | 72/170 [00:00<00:01, 83.17batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:00<00:01, 83.54batch/s]

Training batches on cuda:  53%|█████▎    | 90/170 [00:01<00:00, 84.77batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:01<00:00, 85.61batch/s]

Training batches on cuda:  64%|██████▎   | 108/170 [00:01<00:00, 85.42batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:01<00:00, 85.51batch/s]

Training batches on cuda:  74%|███████▍  | 126/170 [00:01<00:00, 86.24batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:01<00:00, 84.73batch/s]

Training batches on cuda:  85%|████████▍ | 144/170 [00:01<00:00, 84.07batch/s]

Training batches on cuda:  90%|█████████ | 153/170 [00:01<00:00, 85.59batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:01<00:00, 87.24batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:16,  2.41s/epoch, loss=0.00073, prev_loss=0.00073] 
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:14,  2.34s/epoch, loss=0.00073, prev_loss=0.00073]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 7/170 [00:00<00:02, 64.31batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:02, 69.66batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:00<00:01, 74.59batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:00<00:01, 78.56batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:00<00:01, 81.67batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:00<00:01, 80.73batch/s]

Training batches on cuda:  36%|███▌      | 61/170 [00:00<00:01, 80.52batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:00<00:01, 79.19batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:00<00:01, 79.24batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:01<00:01, 79.50batch/s]

Training batches on cuda:  56%|█████▌    | 95/170 [00:01<00:00, 80.70batch/s]

Training batches on cuda:  61%|██████    | 103/170 [00:01<00:00, 74.80batch/s]

Training batches on cuda:  66%|██████▌   | 112/170 [00:01<00:00, 77.05batch/s]

Training batches on cuda:  71%|███████   | 121/170 [00:01<00:00, 80.32batch/s]

Training batches on cuda:  76%|███████▋  | 130/170 [00:01<00:00, 81.92batch/s]

Training batches on cuda:  82%|████████▏ | 139/170 [00:01<00:00, 81.73batch/s]

Training batches on cuda:  87%|████████▋ | 148/170 [00:01<00:00, 83.41batch/s]

Training batches on cuda:  92%|█████████▏| 157/170 [00:01<00:00, 84.98batch/s]

Training batches on cuda:  98%|█████████▊| 166/170 [00:02<00:00, 86.28batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:14,  2.34s/epoch, loss=0.00073, prev_loss=0.00073]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.32s/epoch, loss=0.00073, prev_loss=0.00073]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:02, 55.50batch/s]

Training batches on cuda:   9%|▉         | 15/170 [00:00<00:02, 61.12batch/s]

Training batches on cuda:  14%|█▎        | 23/170 [00:00<00:02, 65.51batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:00<00:01, 70.44batch/s]

Training batches on cuda:  24%|██▍       | 41/170 [00:00<00:01, 75.11batch/s]

Training batches on cuda:  30%|███       | 51/170 [00:00<00:01, 79.34batch/s]

Training batches on cuda:  36%|███▌      | 61/170 [00:00<00:01, 82.60batch/s]

Training batches on cuda:  42%|████▏     | 71/170 [00:00<00:01, 84.94batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:00<00:01, 86.69batch/s]

Training batches on cuda:  53%|█████▎    | 90/170 [00:01<00:00, 86.27batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:01<00:00, 86.97batch/s]

Training batches on cuda:  64%|██████▎   | 108/170 [00:01<00:00, 82.70batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:01<00:00, 81.52batch/s]

Training batches on cuda:  74%|███████▍  | 126/170 [00:01<00:00, 80.94batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:01<00:00, 80.98batch/s]

Training batches on cuda:  85%|████████▌ | 145/170 [00:01<00:00, 83.81batch/s]

Training batches on cuda:  91%|█████████ | 154/170 [00:01<00:00, 85.07batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:01<00:00, 86.13batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:11,  2.32s/epoch, loss=0.000731, prev_loss=0.00073]
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:09,  2.29s/epoch, loss=0.000731, prev_loss=0.00073]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:02, 59.90batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:02, 66.86batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:00<00:02, 72.43batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:00<00:01, 76.45batch/s]

Training batches on cuda:  25%|██▌       | 43/170 [00:00<00:01, 79.75batch/s]

Training batches on cuda:  31%|███       | 53/170 [00:00<00:01, 82.91batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:00<00:01, 84.34batch/s]

Training batches on cuda:  42%|████▏     | 71/170 [00:00<00:01, 74.91batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:00<00:01, 79.23batch/s]

Training batches on cuda:  54%|█████▎    | 91/170 [00:01<00:00, 82.58batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:01<00:00, 85.03batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:01<00:00, 87.03batch/s]

Training batches on cuda:  71%|███████   | 121/170 [00:01<00:00, 88.18batch/s]

Training batches on cuda:  76%|███████▋  | 130/170 [00:01<00:00, 88.71batch/s]

Training batches on cuda:  82%|████████▏ | 140/170 [00:01<00:00, 89.30batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:01<00:00, 89.98batch/s]

Training batches on cuda:  94%|█████████▍| 160/170 [00:01<00:00, 90.21batch/s]

Training batches on cuda: 100%|██████████| 170/170 [00:01<00:00, 87.16batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:16<00:09,  2.29s/epoch, loss=0.000731, prev_loss=0.000731]
Training epochs on cuda:  70%|███████   | 7/10 [00:16<00:06,  2.26s/epoch, loss=0.000731, prev_loss=0.000731]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:03, 47.25batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:02, 53.38batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:02, 59.98batch/s]

Training batches on cuda:  18%|█▊        | 30/170 [00:00<00:02, 64.16batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:00<00:01, 67.52batch/s]

Training batches on cuda:  27%|██▋       | 46/170 [00:00<00:01, 68.72batch/s]

Training batches on cuda:  32%|███▏      | 54/170 [00:00<00:01, 70.91batch/s]

Training batches on cuda:  37%|███▋      | 63/170 [00:00<00:01, 75.21batch/s]

Training batches on cuda:  42%|████▏     | 72/170 [00:00<00:01, 78.58batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:01<00:01, 81.35batch/s]

Training batches on cuda:  53%|█████▎    | 90/170 [00:01<00:00, 81.47batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:01<00:00, 83.35batch/s]

Training batches on cuda:  64%|██████▎   | 108/170 [00:01<00:00, 84.91batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:01<00:00, 82.91batch/s]

Training batches on cuda:  74%|███████▍  | 126/170 [00:01<00:00, 77.89batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:01<00:00, 74.40batch/s]

Training batches on cuda:  84%|████████▎ | 142/170 [00:01<00:00, 74.90batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:01<00:00, 76.62batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:02<00:00, 77.12batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:02<00:00, 77.58batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:18<00:06,  2.26s/epoch, loss=0.000731, prev_loss=0.000731]
Training epochs on cuda:  80%|████████  | 8/10 [00:18<00:04,  2.29s/epoch, loss=0.000731, prev_loss=0.000731]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:03, 49.82batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:02, 55.61batch/s]

Training batches on cuda:  12%|█▏        | 21/170 [00:00<00:02, 61.13batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:00<00:02, 65.03batch/s]

Training batches on cuda:  22%|██▏       | 37/170 [00:00<00:01, 68.49batch/s]

Training batches on cuda:  26%|██▋       | 45/170 [00:00<00:01, 70.83batch/s]

Training batches on cuda:  32%|███▏      | 54/170 [00:00<00:01, 73.30batch/s]

Training batches on cuda:  37%|███▋      | 63/170 [00:00<00:01, 75.28batch/s]

Training batches on cuda:  42%|████▏     | 71/170 [00:00<00:01, 76.43batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:01<00:01, 77.59batch/s]

Training batches on cuda:  52%|█████▏    | 88/170 [00:01<00:01, 77.84batch/s]

Training batches on cuda:  56%|█████▋    | 96/170 [00:01<00:00, 77.35batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:01<00:01, 65.18batch/s]

Training batches on cuda:  66%|██████▌   | 112/170 [00:01<00:00, 68.58batch/s]

Training batches on cuda:  71%|███████   | 120/170 [00:01<00:00, 70.92batch/s]

Training batches on cuda:  75%|███████▌  | 128/170 [00:01<00:00, 72.81batch/s]

Training batches on cuda:  80%|████████  | 136/170 [00:01<00:00, 74.56batch/s]

Training batches on cuda:  85%|████████▍ | 144/170 [00:01<00:00, 75.63batch/s]

Training batches on cuda:  89%|████████▉ | 152/170 [00:02<00:00, 76.56batch/s]

Training batches on cuda:  94%|█████████▍| 160/170 [00:02<00:00, 76.79batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:02<00:00, 78.18batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:20<00:04,  2.29s/epoch, loss=0.000731, prev_loss=0.000731]
Training epochs on cuda:  90%|█████████ | 9/10 [00:20<00:02,  2.34s/epoch, loss=0.000731, prev_loss=0.000731]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:04, 38.71batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:03, 46.57batch/s]

Training batches on cuda:  12%|█▏        | 21/170 [00:00<00:02, 52.92batch/s]

Training batches on cuda:  18%|█▊        | 30/170 [00:00<00:02, 59.51batch/s]

Training batches on cuda:  23%|██▎       | 39/170 [00:00<00:02, 64.92batch/s]

Training batches on cuda:  28%|██▊       | 47/170 [00:00<00:01, 68.49batch/s]

Training batches on cuda:  32%|███▏      | 55/170 [00:00<00:01, 70.53batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:00<00:01, 75.26batch/s]

Training batches on cuda:  42%|████▏     | 72/170 [00:00<00:01, 74.41batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:01<00:01, 74.98batch/s]

Training batches on cuda:  52%|█████▏    | 88/170 [00:01<00:01, 75.81batch/s]

Training batches on cuda:  56%|█████▋    | 96/170 [00:01<00:00, 76.58batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:01<00:00, 79.85batch/s]

Training batches on cuda:  68%|██████▊   | 115/170 [00:01<00:00, 82.85batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:01<00:00, 85.20batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:01<00:00, 86.81batch/s]

Training batches on cuda:  85%|████████▌ | 145/170 [00:01<00:00, 88.16batch/s]

Training batches on cuda:  91%|█████████ | 154/170 [00:01<00:00, 88.61batch/s]

Training batches on cuda:  96%|█████████▋| 164/170 [00:02<00:00, 89.41batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:23<00:02,  2.34s/epoch, loss=0.000731, prev_loss=0.000731]2021-09-16:18:04:09,515 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:04:10,502 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:04:25,469 INFO     [evaluator.py:657] Evaluation took 14.97s seconds
2021-09-16:18:04:25,611 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  7.15s/epoch, loss=0.000731, prev_loss=0.000731]
Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  3.92s/epoch, loss=0.000731, prev_loss=0.000731]
2021-09-16:18:04:25,612 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpuw15t_80'
2021-09-16:18:04:25,679 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpuw15t_80' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.51ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:18:04:40,490 INFO     [evaluator.py:657] Evaluation took 14.80s seconds
[I 2021-09-16 18:04:40,496] Trial 27 finished with value: -0.0022665269892705453 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.38412496659029327, 'optimizer.lr': 0.054442815764977615, 'negative_sampler.num_negs_per_pos': 4, 'training.batch_size': 1832}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:04:40,520 WARNING  [api.py:823] No random seed is specified. Setting to 3847536527.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/425 [00:00<00:05, 70.84batch/s]

Training batches on cuda:   4%|▍         | 17/425 [00:00<00:05, 75.41batch/s]

Training batches on cuda:   7%|▋         | 28/425 [00:00<00:04, 82.89batch/s]

Training batches on cuda:   9%|▉         | 39/425 [00:00<00:04, 88.13batch/s]

Training batches on cuda:  12%|█▏        | 50/425 [00:00<00:04, 91.97batch/s]

Training batches on cuda:  15%|█▍        | 62/425 [00:00<00:03, 97.22batch/s]

Training batches on cuda:  17%|█▋        | 74/425 [00:00<00:03, 101.29batch/s]

Training batches on cuda:  20%|█▉        | 84/425 [00:00<00:03, 100.11batch/s]

Training batches on cuda:  23%|██▎       | 97/425 [00:00<00:03, 105.81batch/s]

Training batches on cuda:  26%|██▌       | 111/425 [00:01<00:02, 112.47batch/s]

Training batches on cuda:  29%|██▉       | 124/425 [00:01<00:02, 116.52batch/s]

Training batches on cuda:  32%|███▏      | 137/425 [00:01<00:02, 119.88batch/s]

Training batches on cuda:  36%|███▌      | 151/425 [00:01<00:02, 123.35batch/s]

Training batches on cuda:  39%|███▉      | 165/425 [00:01<00:02, 125.60batch/s]

Training batches on cuda:  42%|████▏     | 179/425 [00:01<00:01, 127.04batch/s]

Training batches on cuda:  45%|████▌     | 193/425 [00:01<00:01, 128.65batch/s]

Training batches on cuda:  49%|████▊     | 207/425 [00:01<00:01, 129.46batch/s]

Training batches on cuda:  52%|█████▏    | 220/425 [00:01<00:01, 128.15batch/s]

Training batches on cuda:  55%|█████▍    | 233/425 [00:02<00:01, 121.32batch/s]

Training batches on cuda:  58%|█████▊    | 246/425 [00:02<00:01, 116.44batch/s]

Training batches on cuda:  61%|██████    | 258/425 [00:02<00:01, 112.55batch/s]

Training batches on cuda:  64%|██████▎   | 270/425 [00:02<00:01, 110.17batch/s]

Training batches on cuda:  66%|██████▋   | 282/425 [00:02<00:01, 109.48batch/s]

Training batches on cuda:  69%|██████▉   | 294/425 [00:02<00:01, 109.57batch/s]

Training batches on cuda:  72%|███████▏  | 305/425 [00:02<00:01, 108.44batch/s]

Training batches on cuda:  74%|███████▍  | 316/425 [00:02<00:01, 105.39batch/s]

Training batches on cuda:  77%|███████▋  | 327/425 [00:02<00:00, 101.91batch/s]

Training batches on cuda:  80%|███████▉  | 338/425 [00:03<00:00, 101.54batch/s]

Training batches on cuda:  82%|████████▏ | 349/425 [00:03<00:00, 103.15batch/s]

Training batches on cuda:  85%|████████▍ | 360/425 [00:03<00:00, 104.89batch/s]

Training batches on cuda:  87%|████████▋ | 371/425 [00:03<00:00, 99.90batch/s] 

Training batches on cuda:  90%|████████▉ | 382/425 [00:03<00:00, 98.94batch/s]

Training batches on cuda:  92%|█████████▏| 393/425 [00:03<00:00, 99.66batch/s]

Training batches on cuda:  95%|█████████▌| 404/425 [00:03<00:00, 98.04batch/s]

Training batches on cuda:  98%|█████████▊| 415/425 [00:03<00:00, 100.64batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.00293, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:36,  4.10s/epoch, loss=0.00293, prev_loss=nan]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 7/425 [00:00<00:06, 66.52batch/s]

Training batches on cuda:   4%|▍         | 18/425 [00:00<00:05, 73.97batch/s]

Training batches on cuda:   7%|▋         | 29/425 [00:00<00:04, 81.55batch/s]

Training batches on cuda:   9%|▉         | 40/425 [00:00<00:04, 87.60batch/s]

Training batches on cuda:  12%|█▏        | 51/425 [00:00<00:04, 92.40batch/s]

Training batches on cuda:  15%|█▍        | 62/425 [00:00<00:03, 94.61batch/s]

Training batches on cuda:  17%|█▋        | 72/425 [00:00<00:03, 95.83batch/s]

Training batches on cuda:  19%|█▉        | 82/425 [00:00<00:03, 96.59batch/s]

Training batches on cuda:  22%|██▏       | 92/425 [00:00<00:03, 95.72batch/s]

Training batches on cuda:  24%|██▍       | 103/425 [00:01<00:03, 97.50batch/s]

Training batches on cuda:  27%|██▋       | 114/425 [00:01<00:03, 99.37batch/s]

Training batches on cuda:  29%|██▉       | 125/425 [00:01<00:02, 101.63batch/s]

Training batches on cuda:  32%|███▏      | 136/425 [00:01<00:02, 103.76batch/s]

Training batches on cuda:  35%|███▍      | 147/425 [00:01<00:02, 100.73batch/s]

Training batches on cuda:  37%|███▋      | 159/425 [00:01<00:02, 102.96batch/s]

Training batches on cuda:  40%|████      | 171/425 [00:01<00:02, 105.20batch/s]

Training batches on cuda:  43%|████▎     | 182/425 [00:01<00:02, 105.43batch/s]

Training batches on cuda:  45%|████▌     | 193/425 [00:01<00:02, 106.13batch/s]

Training batches on cuda:  48%|████▊     | 204/425 [00:02<00:02, 105.02batch/s]

Training batches on cuda:  51%|█████     | 216/425 [00:02<00:01, 106.77batch/s]

Training batches on cuda:  54%|█████▎    | 228/425 [00:02<00:01, 108.35batch/s]

Training batches on cuda:  56%|█████▌    | 239/425 [00:02<00:01, 108.77batch/s]

Training batches on cuda:  59%|█████▉    | 250/425 [00:02<00:01, 107.14batch/s]

Training batches on cuda:  61%|██████▏   | 261/425 [00:02<00:01, 105.44batch/s]

Training batches on cuda:  64%|██████▍   | 272/425 [00:02<00:01, 105.16batch/s]

Training batches on cuda:  67%|██████▋   | 283/425 [00:02<00:01, 105.53batch/s]

Training batches on cuda:  69%|██████▉   | 294/425 [00:02<00:01, 103.00batch/s]

Training batches on cuda:  72%|███████▏  | 305/425 [00:02<00:01, 101.93batch/s]

Training batches on cuda:  75%|███████▍  | 317/425 [00:03<00:01, 106.49batch/s]

Training batches on cuda:  78%|███████▊  | 330/425 [00:03<00:00, 110.59batch/s]

Training batches on cuda:  81%|████████  | 343/425 [00:03<00:00, 115.68batch/s]

Training batches on cuda:  84%|████████▍ | 357/425 [00:03<00:00, 119.63batch/s]

Training batches on cuda:  87%|████████▋ | 370/425 [00:03<00:00, 120.22batch/s]

Training batches on cuda:  90%|█████████ | 383/425 [00:03<00:00, 120.48batch/s]

Training batches on cuda:  93%|█████████▎| 396/425 [00:03<00:00, 120.61batch/s]

Training batches on cuda:  96%|█████████▌| 409/425 [00:03<00:00, 120.46batch/s]

Training batches on cuda:  99%|█████████▉| 422/425 [00:03<00:00, 118.14batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:36,  4.10s/epoch, loss=0.00292, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:32,  4.12s/epoch, loss=0.00292, prev_loss=nan]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/425 [00:00<00:05, 76.00batch/s]

Training batches on cuda:   4%|▍         | 18/425 [00:00<00:05, 80.89batch/s]

Training batches on cuda:   7%|▋         | 30/425 [00:00<00:04, 88.45batch/s]

Training batches on cuda:  10%|█         | 43/425 [00:00<00:03, 97.04batch/s]

Training batches on cuda:  13%|█▎        | 56/425 [00:00<00:03, 102.82batch/s]

Training batches on cuda:  16%|█▌        | 69/425 [00:00<00:03, 109.09batch/s]

Training batches on cuda:  19%|█▉        | 82/425 [00:00<00:03, 113.02batch/s]

Training batches on cuda:  22%|██▏       | 94/425 [00:00<00:02, 114.54batch/s]

Training batches on cuda:  25%|██▌       | 107/425 [00:00<00:02, 116.79batch/s]

Training batches on cuda:  28%|██▊       | 120/425 [00:01<00:02, 118.91batch/s]

Training batches on cuda:  31%|███▏      | 133/425 [00:01<00:02, 119.63batch/s]

Training batches on cuda:  34%|███▍      | 146/425 [00:01<00:02, 120.78batch/s]

Training batches on cuda:  37%|███▋      | 159/425 [00:01<00:02, 122.55batch/s]

Training batches on cuda:  40%|████      | 172/425 [00:01<00:02, 123.15batch/s]

Training batches on cuda:  44%|████▎     | 185/425 [00:01<00:01, 123.90batch/s]

Training batches on cuda:  47%|████▋     | 198/425 [00:01<00:01, 123.04batch/s]

Training batches on cuda:  50%|████▉     | 211/425 [00:01<00:01, 121.62batch/s]

Training batches on cuda:  53%|█████▎    | 224/425 [00:01<00:01, 122.38batch/s]

Training batches on cuda:  56%|█████▌    | 237/425 [00:02<00:01, 120.97batch/s]

Training batches on cuda:  59%|█████▉    | 250/425 [00:02<00:01, 117.21batch/s]

Training batches on cuda:  62%|██████▏   | 262/425 [00:02<00:01, 114.16batch/s]

Training batches on cuda:  64%|██████▍   | 274/425 [00:02<00:01, 113.30batch/s]

Training batches on cuda:  67%|██████▋   | 286/425 [00:02<00:01, 114.47batch/s]

Training batches on cuda:  70%|███████   | 298/425 [00:02<00:01, 112.59batch/s]

Training batches on cuda:  73%|███████▎  | 310/425 [00:02<00:01, 108.84batch/s]

Training batches on cuda:  76%|███████▌  | 322/425 [00:02<00:00, 109.69batch/s]

Training batches on cuda:  79%|███████▊  | 334/425 [00:02<00:00, 112.02batch/s]

Training batches on cuda:  82%|████████▏ | 347/425 [00:02<00:00, 114.99batch/s]

Training batches on cuda:  84%|████████▍ | 359/425 [00:03<00:00, 114.63batch/s]

Training batches on cuda:  87%|████████▋ | 371/425 [00:03<00:00, 115.36batch/s]

Training batches on cuda:  90%|█████████ | 383/425 [00:03<00:00, 115.01batch/s]

Training batches on cuda:  93%|█████████▎| 395/425 [00:03<00:00, 108.46batch/s]

Training batches on cuda:  96%|█████████▌| 406/425 [00:03<00:00, 99.77batch/s] 

Training batches on cuda:  98%|█████████▊| 417/425 [00:03<00:00, 100.44batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:32,  4.12s/epoch, loss=0.00292, prev_loss=0.00292]
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:28,  4.06s/epoch, loss=0.00292, prev_loss=0.00292]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 5/425 [00:00<00:08, 48.04batch/s]

Training batches on cuda:   4%|▍         | 16/425 [00:00<00:07, 57.47batch/s]

Training batches on cuda:   7%|▋         | 28/425 [00:00<00:05, 67.76batch/s]

Training batches on cuda:   9%|▉         | 40/425 [00:00<00:04, 77.81batch/s]

Training batches on cuda:  12%|█▏        | 52/425 [00:00<00:04, 86.73batch/s]

Training batches on cuda:  15%|█▍        | 63/425 [00:00<00:03, 91.38batch/s]

Training batches on cuda:  17%|█▋        | 74/425 [00:00<00:03, 95.04batch/s]

Training batches on cuda:  20%|██        | 85/425 [00:00<00:03, 97.05batch/s]

Training batches on cuda:  23%|██▎       | 96/425 [00:00<00:03, 99.82batch/s]

Training batches on cuda:  25%|██▌       | 108/425 [00:01<00:03, 103.36batch/s]

Training batches on cuda:  28%|██▊       | 120/425 [00:01<00:02, 107.54batch/s]

Training batches on cuda:  31%|███▏      | 133/425 [00:01<00:02, 112.15batch/s]

Training batches on cuda:  34%|███▍      | 145/425 [00:01<00:02, 113.10batch/s]

Training batches on cuda:  37%|███▋      | 157/425 [00:01<00:02, 114.90batch/s]

Training batches on cuda:  40%|███▉      | 169/425 [00:01<00:02, 112.97batch/s]

Training batches on cuda:  43%|████▎     | 181/425 [00:01<00:02, 114.19batch/s]

Training batches on cuda:  45%|████▌     | 193/425 [00:01<00:02, 112.58batch/s]

Training batches on cuda:  48%|████▊     | 205/425 [00:01<00:02, 109.21batch/s]

Training batches on cuda:  51%|█████     | 216/425 [00:01<00:01, 109.32batch/s]

Training batches on cuda:  54%|█████▎    | 228/425 [00:02<00:01, 110.00batch/s]

Training batches on cuda:  56%|█████▋    | 240/425 [00:02<00:01, 109.74batch/s]

Training batches on cuda:  59%|█████▉    | 251/425 [00:02<00:01, 109.20batch/s]

Training batches on cuda:  62%|██████▏   | 262/425 [00:02<00:01, 107.05batch/s]

Training batches on cuda:  64%|██████▍   | 273/425 [00:02<00:01, 106.71batch/s]

Training batches on cuda:  67%|██████▋   | 284/425 [00:02<00:01, 106.44batch/s]

Training batches on cuda:  69%|██████▉   | 295/425 [00:02<00:01, 104.43batch/s]

Training batches on cuda:  72%|███████▏  | 306/425 [00:02<00:01, 104.21batch/s]

Training batches on cuda:  75%|███████▍  | 317/425 [00:02<00:01, 104.03batch/s]

Training batches on cuda:  77%|███████▋  | 328/425 [00:03<00:00, 103.78batch/s]

Training batches on cuda:  80%|███████▉  | 339/425 [00:03<00:00, 104.13batch/s]

Training batches on cuda:  82%|████████▏ | 350/425 [00:03<00:00, 104.38batch/s]

Training batches on cuda:  85%|████████▍ | 361/425 [00:03<00:00, 94.25batch/s] 

Training batches on cuda:  88%|████████▊ | 373/425 [00:03<00:00, 100.30batch/s]

Training batches on cuda:  91%|█████████ | 386/425 [00:03<00:00, 105.51batch/s]

Training batches on cuda:  94%|█████████▎| 398/425 [00:03<00:00, 108.73batch/s]

Training batches on cuda:  97%|█████████▋| 411/425 [00:03<00:00, 112.44batch/s]

Training batches on cuda: 100%|█████████▉| 423/425 [00:03<00:00, 106.91batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:16<00:28,  4.06s/epoch, loss=0.00292, prev_loss=0.00292]
Training epochs on cuda:  40%|████      | 4/10 [00:16<00:24,  4.11s/epoch, loss=0.00292, prev_loss=0.00292]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 5/425 [00:00<00:08, 48.64batch/s]

Training batches on cuda:   4%|▎         | 15/425 [00:00<00:07, 57.19batch/s]

Training batches on cuda:   7%|▋         | 28/425 [00:00<00:05, 67.96batch/s]

Training batches on cuda:   9%|▉         | 40/425 [00:00<00:04, 77.39batch/s]

Training batches on cuda:  12%|█▏        | 52/425 [00:00<00:04, 85.34batch/s]

Training batches on cuda:  15%|█▍        | 62/425 [00:00<00:04, 87.27batch/s]

Training batches on cuda:  17%|█▋        | 72/425 [00:00<00:03, 89.44batch/s]

Training batches on cuda:  19%|█▉        | 82/425 [00:00<00:03, 89.93batch/s]

Training batches on cuda:  22%|██▏       | 92/425 [00:00<00:03, 91.00batch/s]

Training batches on cuda:  24%|██▍       | 103/425 [00:01<00:03, 94.81batch/s]

Training batches on cuda:  27%|██▋       | 114/425 [00:01<00:03, 98.43batch/s]

Training batches on cuda:  29%|██▉       | 125/425 [00:01<00:03, 99.10batch/s]

Training batches on cuda:  32%|███▏      | 136/425 [00:01<00:02, 102.06batch/s]

Training batches on cuda:  35%|███▍      | 147/425 [00:01<00:02, 101.92batch/s]

Training batches on cuda:  37%|███▋      | 158/425 [00:01<00:02, 103.45batch/s]

Training batches on cuda:  40%|███▉      | 169/425 [00:01<00:02, 103.49batch/s]

Training batches on cuda:  42%|████▏     | 180/425 [00:01<00:02, 105.36batch/s]

Training batches on cuda:  45%|████▍     | 191/425 [00:01<00:02, 105.86batch/s]

Training batches on cuda:  48%|████▊     | 203/425 [00:01<00:02, 107.60batch/s]

Training batches on cuda:  50%|█████     | 214/425 [00:02<00:01, 105.61batch/s]

Training batches on cuda:  53%|█████▎    | 226/425 [00:02<00:01, 107.13batch/s]

Training batches on cuda:  56%|█████▌    | 238/425 [00:02<00:01, 108.51batch/s]

Training batches on cuda:  59%|█████▉    | 250/425 [00:02<00:01, 110.33batch/s]

Training batches on cuda:  62%|██████▏   | 262/425 [00:02<00:01, 111.54batch/s]

Training batches on cuda:  64%|██████▍   | 274/425 [00:02<00:01, 113.31batch/s]

Training batches on cuda:  67%|██████▋   | 286/425 [00:02<00:01, 114.52batch/s]

Training batches on cuda:  70%|███████   | 298/425 [00:02<00:01, 115.32batch/s]

Training batches on cuda:  73%|███████▎  | 310/425 [00:02<00:01, 113.03batch/s]

Training batches on cuda:  76%|███████▌  | 322/425 [00:03<00:00, 114.15batch/s]

Training batches on cuda:  79%|███████▉  | 335/425 [00:03<00:00, 116.14batch/s]

Training batches on cuda:  82%|████████▏ | 347/425 [00:03<00:00, 116.63batch/s]

Training batches on cuda:  84%|████████▍ | 359/425 [00:03<00:00, 116.48batch/s]

Training batches on cuda:  87%|████████▋ | 371/425 [00:03<00:00, 117.18batch/s]

Training batches on cuda:  90%|█████████ | 384/425 [00:03<00:00, 119.89batch/s]

Training batches on cuda:  93%|█████████▎| 397/425 [00:03<00:00, 122.22batch/s]

Training batches on cuda:  96%|█████████▋| 410/425 [00:03<00:00, 119.40batch/s]

Training batches on cuda:  99%|█████████▉| 422/425 [00:03<00:00, 114.12batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:24,  4.11s/epoch, loss=0.00292, prev_loss=0.00292]
Training epochs on cuda:  50%|█████     | 5/10 [00:20<00:20,  4.14s/epoch, loss=0.00292, prev_loss=0.00292]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 4/425 [00:00<00:11, 37.23batch/s]

Training batches on cuda:   4%|▍         | 16/425 [00:00<00:08, 46.54batch/s]

Training batches on cuda:   7%|▋         | 29/425 [00:00<00:06, 57.28batch/s]

Training batches on cuda:  10%|▉         | 42/425 [00:00<00:05, 68.54batch/s]

Training batches on cuda:  13%|█▎        | 54/425 [00:00<00:04, 78.50batch/s]

Training batches on cuda:  16%|█▌        | 66/425 [00:00<00:04, 86.31batch/s]

Training batches on cuda:  18%|█▊        | 77/425 [00:00<00:03, 91.65batch/s]

Training batches on cuda:  21%|██        | 88/425 [00:00<00:03, 93.43batch/s]

Training batches on cuda:  23%|██▎       | 99/425 [00:00<00:03, 96.66batch/s]

Training batches on cuda:  26%|██▌       | 111/425 [00:01<00:03, 101.01batch/s]

Training batches on cuda:  29%|██▊       | 122/425 [00:01<00:02, 103.48batch/s]

Training batches on cuda:  31%|███▏      | 133/425 [00:01<00:02, 103.94batch/s]

Training batches on cuda:  34%|███▍      | 144/425 [00:01<00:02, 105.32batch/s]

Training batches on cuda:  36%|███▋      | 155/425 [00:01<00:02, 104.99batch/s]

Training batches on cuda:  39%|███▉      | 166/425 [00:01<00:02, 104.58batch/s]

Training batches on cuda:  42%|████▏     | 177/425 [00:01<00:02, 102.57batch/s]

Training batches on cuda:  44%|████▍     | 188/425 [00:01<00:02, 99.51batch/s] 

Training batches on cuda:  47%|████▋     | 199/425 [00:01<00:02, 101.09batch/s]

Training batches on cuda:  49%|████▉     | 210/425 [00:02<00:02, 102.54batch/s]

Training batches on cuda:  52%|█████▏    | 221/425 [00:02<00:01, 102.24batch/s]

Training batches on cuda:  55%|█████▍    | 232/425 [00:02<00:01, 102.28batch/s]

Training batches on cuda:  57%|█████▋    | 243/425 [00:02<00:01, 104.42batch/s]

Training batches on cuda:  60%|█████▉    | 254/425 [00:02<00:01, 105.86batch/s]

Training batches on cuda:  62%|██████▏   | 265/425 [00:02<00:01, 102.65batch/s]

Training batches on cuda:  65%|██████▌   | 277/425 [00:02<00:01, 106.90batch/s]

Training batches on cuda:  68%|██████▊   | 289/425 [00:02<00:01, 108.89batch/s]

Training batches on cuda:  71%|███████   | 300/425 [00:02<00:01, 105.28batch/s]

Training batches on cuda:  73%|███████▎  | 311/425 [00:02<00:01, 98.27batch/s] 

Training batches on cuda:  76%|███████▌  | 323/425 [00:03<00:01, 101.79batch/s]

Training batches on cuda:  79%|███████▉  | 335/425 [00:03<00:00, 105.03batch/s]

Training batches on cuda:  81%|████████▏ | 346/425 [00:03<00:00, 104.60batch/s]

Training batches on cuda:  84%|████████▍ | 357/425 [00:03<00:00, 103.46batch/s]

Training batches on cuda:  87%|████████▋ | 368/425 [00:03<00:00, 103.26batch/s]

Training batches on cuda:  89%|████████▉ | 379/425 [00:03<00:00, 99.22batch/s] 

Training batches on cuda:  92%|█████████▏| 391/425 [00:03<00:00, 103.94batch/s]

Training batches on cuda:  95%|█████████▍| 403/425 [00:03<00:00, 107.26batch/s]

Training batches on cuda:  97%|█████████▋| 414/425 [00:03<00:00, 103.28batch/s]

Training batches on cuda: 100%|██████████| 425/425 [00:04<00:00, 96.77batch/s] 

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:24<00:20,  4.14s/epoch, loss=0.00292, prev_loss=0.00292]
Training epochs on cuda:  60%|██████    | 6/10 [00:24<00:16,  4.20s/epoch, loss=0.00292, prev_loss=0.00292]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/425 [00:00<00:05, 79.62batch/s]

Training batches on cuda:   4%|▍         | 19/425 [00:00<00:04, 85.58batch/s]

Training batches on cuda:   7%|▋         | 30/425 [00:00<00:04, 89.68batch/s]

Training batches on cuda:   9%|▉         | 40/425 [00:00<00:04, 91.73batch/s]

Training batches on cuda:  12%|█▏        | 52/425 [00:00<00:03, 97.24batch/s]

Training batches on cuda:  15%|█▌        | 64/425 [00:00<00:03, 100.87batch/s]

Training batches on cuda:  17%|█▋        | 74/425 [00:00<00:03, 95.97batch/s] 

Training batches on cuda:  20%|██        | 85/425 [00:00<00:03, 99.20batch/s]

Training batches on cuda:  23%|██▎       | 97/425 [00:00<00:03, 102.49batch/s]

Training batches on cuda:  25%|██▌       | 108/425 [00:01<00:03, 103.09batch/s]

Training batches on cuda:  28%|██▊       | 119/425 [00:01<00:02, 103.38batch/s]

Training batches on cuda:  31%|███       | 131/425 [00:01<00:02, 105.78batch/s]

Training batches on cuda:  34%|███▎      | 143/425 [00:01<00:02, 107.13batch/s]

Training batches on cuda:  36%|███▌      | 154/425 [00:01<00:02, 106.08batch/s]

Training batches on cuda:  39%|███▉      | 165/425 [00:01<00:02, 107.10batch/s]

Training batches on cuda:  41%|████▏     | 176/425 [00:01<00:02, 102.84batch/s]

Training batches on cuda:  44%|████▍     | 187/425 [00:01<00:02, 102.64batch/s]

Training batches on cuda:  47%|████▋     | 198/425 [00:01<00:02, 104.35batch/s]

Training batches on cuda:  49%|████▉     | 210/425 [00:02<00:02, 106.91batch/s]

Training batches on cuda:  52%|█████▏    | 221/425 [00:02<00:01, 105.28batch/s]

Training batches on cuda:  55%|█████▍    | 232/425 [00:02<00:01, 102.82batch/s]

Training batches on cuda:  57%|█████▋    | 243/425 [00:02<00:01, 101.04batch/s]

Training batches on cuda:  60%|█████▉    | 254/425 [00:02<00:01, 100.13batch/s]

Training batches on cuda:  62%|██████▏   | 265/425 [00:02<00:01, 96.74batch/s] 

Training batches on cuda:  65%|██████▍   | 275/425 [00:02<00:01, 96.01batch/s]

Training batches on cuda:  67%|██████▋   | 286/425 [00:02<00:01, 97.94batch/s]

Training batches on cuda:  70%|██████▉   | 296/425 [00:02<00:01, 96.44batch/s]

Training batches on cuda:  72%|███████▏  | 307/425 [00:03<00:01, 98.66batch/s]

Training batches on cuda:  75%|███████▍  | 318/425 [00:03<00:01, 100.83batch/s]

Training batches on cuda:  78%|███████▊  | 330/425 [00:03<00:00, 103.71batch/s]

Training batches on cuda:  80%|████████  | 341/425 [00:03<00:00, 104.54batch/s]

Training batches on cuda:  83%|████████▎ | 353/425 [00:03<00:00, 106.39batch/s]

Training batches on cuda:  86%|████████▌ | 365/425 [00:03<00:00, 107.39batch/s]

Training batches on cuda:  88%|████████▊ | 376/425 [00:03<00:00, 102.12batch/s]

Training batches on cuda:  91%|█████████ | 387/425 [00:03<00:00, 102.88batch/s]

Training batches on cuda:  94%|█████████▍| 399/425 [00:03<00:00, 106.96batch/s]

Training batches on cuda:  96%|█████████▋| 410/425 [00:03<00:00, 107.30batch/s]

Training batches on cuda:  99%|█████████▉| 421/425 [00:04<00:00, 98.01batch/s] 

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:29<00:16,  4.20s/epoch, loss=0.00292, prev_loss=0.00292]
Training epochs on cuda:  70%|███████   | 7/10 [00:29<00:12,  4.27s/epoch, loss=0.00292, prev_loss=0.00292]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 7/425 [00:00<00:06, 68.01batch/s]

Training batches on cuda:   4%|▍         | 18/425 [00:00<00:05, 75.41batch/s]

Training batches on cuda:   7%|▋         | 29/425 [00:00<00:04, 82.24batch/s]

Training batches on cuda:   9%|▉         | 39/425 [00:00<00:04, 86.55batch/s]

Training batches on cuda:  12%|█▏        | 49/425 [00:00<00:04, 88.72batch/s]

Training batches on cuda:  14%|█▍        | 60/425 [00:00<00:03, 93.24batch/s]

Training batches on cuda:  17%|█▋        | 71/425 [00:00<00:03, 97.49batch/s]

Training batches on cuda:  19%|█▉        | 82/425 [00:00<00:03, 98.65batch/s]

Training batches on cuda:  22%|██▏       | 93/425 [00:00<00:03, 99.96batch/s]

Training batches on cuda:  24%|██▍       | 103/425 [00:01<00:03, 97.34batch/s]

Training batches on cuda:  27%|██▋       | 114/425 [00:01<00:03, 99.15batch/s]

Training batches on cuda:  30%|██▉       | 126/425 [00:01<00:02, 102.83batch/s]

Training batches on cuda:  32%|███▏      | 137/425 [00:01<00:03, 94.32batch/s] 

Training batches on cuda:  35%|███▌      | 149/425 [00:01<00:02, 98.76batch/s]

Training batches on cuda:  38%|███▊      | 161/425 [00:01<00:02, 103.26batch/s]

Training batches on cuda:  41%|████      | 173/425 [00:01<00:02, 105.81batch/s]

Training batches on cuda:  44%|████▎     | 185/425 [00:01<00:02, 108.71batch/s]

Training batches on cuda:  46%|████▌     | 196/425 [00:01<00:02, 107.48batch/s]

Training batches on cuda:  49%|████▊     | 207/425 [00:02<00:02, 107.37batch/s]

Training batches on cuda:  51%|█████▏    | 218/425 [00:02<00:01, 107.70batch/s]

Training batches on cuda:  54%|█████▍    | 229/425 [00:02<00:01, 107.60batch/s]

Training batches on cuda:  56%|█████▋    | 240/425 [00:02<00:01, 107.49batch/s]

Training batches on cuda:  59%|█████▉    | 251/425 [00:02<00:01, 104.80batch/s]

Training batches on cuda:  62%|██████▏   | 262/425 [00:02<00:01, 103.64batch/s]

Training batches on cuda:  64%|██████▍   | 273/425 [00:02<00:01, 102.21batch/s]

Training batches on cuda:  67%|██████▋   | 284/425 [00:02<00:01, 101.76batch/s]

Training batches on cuda:  69%|██████▉   | 295/425 [00:02<00:01, 101.19batch/s]

Training batches on cuda:  72%|███████▏  | 306/425 [00:02<00:01, 102.01batch/s]

Training batches on cuda:  75%|███████▍  | 317/425 [00:03<00:01, 101.14batch/s]

Training batches on cuda:  77%|███████▋  | 328/425 [00:03<00:00, 101.41batch/s]

Training batches on cuda:  80%|███████▉  | 339/425 [00:03<00:00, 100.90batch/s]

Training batches on cuda:  82%|████████▏ | 350/425 [00:03<00:00, 100.66batch/s]

Training batches on cuda:  85%|████████▍ | 361/425 [00:03<00:00, 101.20batch/s]

Training batches on cuda:  88%|████████▊ | 372/425 [00:03<00:00, 103.06batch/s]

Training batches on cuda:  90%|█████████ | 383/425 [00:03<00:00, 104.18batch/s]

Training batches on cuda:  93%|█████████▎| 394/425 [00:03<00:00, 104.55batch/s]

Training batches on cuda:  95%|█████████▌| 405/425 [00:03<00:00, 105.39batch/s]

Training batches on cuda:  98%|█████████▊| 416/425 [00:04<00:00, 105.27batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:33<00:12,  4.27s/epoch, loss=0.00292, prev_loss=0.00292]
Training epochs on cuda:  80%|████████  | 8/10 [00:33<00:08,  4.30s/epoch, loss=0.00292, prev_loss=0.00292]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 6/425 [00:00<00:07, 55.56batch/s]

Training batches on cuda:   4%|▍         | 16/425 [00:00<00:06, 63.26batch/s]

Training batches on cuda:   6%|▋         | 27/425 [00:00<00:05, 72.27batch/s]

Training batches on cuda:   9%|▉         | 38/425 [00:00<00:04, 79.57batch/s]

Training batches on cuda:  12%|█▏        | 49/425 [00:00<00:04, 85.58batch/s]

Training batches on cuda:  14%|█▍        | 60/425 [00:00<00:03, 91.56batch/s]

Training batches on cuda:  17%|█▋        | 72/425 [00:00<00:03, 96.76batch/s]

Training batches on cuda:  20%|█▉        | 83/425 [00:00<00:03, 99.69batch/s]

Training batches on cuda:  22%|██▏       | 95/425 [00:00<00:03, 103.24batch/s]

Training batches on cuda:  25%|██▍       | 106/425 [00:01<00:03, 102.28batch/s]

Training batches on cuda:  28%|██▊       | 117/425 [00:01<00:02, 103.71batch/s]

Training batches on cuda:  30%|███       | 128/425 [00:01<00:02, 100.76batch/s]

Training batches on cuda:  33%|███▎      | 139/425 [00:01<00:02, 100.39batch/s]

Training batches on cuda:  36%|███▌      | 151/425 [00:01<00:02, 103.39batch/s]

Training batches on cuda:  38%|███▊      | 162/425 [00:01<00:02, 105.13batch/s]

Training batches on cuda:  41%|████      | 174/425 [00:01<00:02, 106.49batch/s]

Training batches on cuda:  44%|████▎     | 185/425 [00:01<00:02, 104.32batch/s]

Training batches on cuda:  46%|████▌     | 196/425 [00:01<00:02, 105.83batch/s]

Training batches on cuda:  49%|████▊     | 207/425 [00:02<00:02, 97.67batch/s] 

Training batches on cuda:  52%|█████▏    | 219/425 [00:02<00:01, 103.23batch/s]

Training batches on cuda:  54%|█████▍    | 231/425 [00:02<00:01, 106.75batch/s]

Training batches on cuda:  57%|█████▋    | 244/425 [00:02<00:01, 110.26batch/s]

Training batches on cuda:  60%|██████    | 256/425 [00:02<00:01, 111.97batch/s]

Training batches on cuda:  63%|██████▎   | 268/425 [00:02<00:01, 112.82batch/s]

Training batches on cuda:  66%|██████▌   | 280/425 [00:02<00:01, 113.84batch/s]

Training batches on cuda:  69%|██████▊   | 292/425 [00:02<00:01, 105.35batch/s]

Training batches on cuda:  71%|███████▏  | 303/425 [00:02<00:01, 102.81batch/s]

Training batches on cuda:  74%|███████▍  | 314/425 [00:03<00:01, 98.39batch/s] 

Training batches on cuda:  76%|███████▌  | 324/425 [00:03<00:01, 95.35batch/s]

Training batches on cuda:  79%|███████▉  | 335/425 [00:03<00:00, 97.69batch/s]

Training batches on cuda:  82%|████████▏ | 347/425 [00:03<00:00, 101.70batch/s]

Training batches on cuda:  84%|████████▍ | 358/425 [00:03<00:00, 103.30batch/s]

Training batches on cuda:  87%|████████▋ | 369/425 [00:03<00:00, 97.95batch/s] 

Training batches on cuda:  89%|████████▉ | 379/425 [00:03<00:00, 91.53batch/s]

Training batches on cuda:  92%|█████████▏| 389/425 [00:03<00:00, 86.97batch/s]

Training batches on cuda:  94%|█████████▍| 400/425 [00:03<00:00, 91.37batch/s]

Training batches on cuda:  97%|█████████▋| 411/425 [00:04<00:00, 95.67batch/s]

Training batches on cuda: 100%|█████████▉| 423/425 [00:04<00:00, 99.84batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:38<00:08,  4.30s/epoch, loss=0.00292, prev_loss=0.00292]
Training epochs on cuda:  90%|█████████ | 9/10 [00:38<00:04,  4.32s/epoch, loss=0.00292, prev_loss=0.00292]

Training batches on cuda:   0%|          | 0/425 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 8/425 [00:00<00:05, 71.18batch/s]

Training batches on cuda:   4%|▍         | 18/425 [00:00<00:05, 76.29batch/s]

Training batches on cuda:   7%|▋         | 29/425 [00:00<00:04, 82.06batch/s]

Training batches on cuda:   9%|▉         | 40/425 [00:00<00:04, 87.00batch/s]

Training batches on cuda:  12%|█▏        | 51/425 [00:00<00:04, 91.67batch/s]

Training batches on cuda:  15%|█▍        | 62/425 [00:00<00:03, 95.41batch/s]

Training batches on cuda:  17%|█▋        | 73/425 [00:00<00:03, 98.74batch/s]

Training batches on cuda:  20%|█▉        | 84/425 [00:00<00:03, 99.99batch/s]

Training batches on cuda:  22%|██▏       | 94/425 [00:00<00:03, 99.47batch/s]

Training batches on cuda:  24%|██▍       | 104/425 [00:01<00:03, 99.54batch/s]

Training batches on cuda:  27%|██▋       | 115/425 [00:01<00:03, 101.26batch/s]

Training batches on cuda:  30%|██▉       | 126/425 [00:01<00:02, 101.92batch/s]

Training batches on cuda:  32%|███▏      | 137/425 [00:01<00:02, 101.10batch/s]

Training batches on cuda:  35%|███▍      | 148/425 [00:01<00:02, 101.10batch/s]

Training batches on cuda:  37%|███▋      | 159/425 [00:01<00:02, 101.18batch/s]

Training batches on cuda:  40%|████      | 170/425 [00:01<00:02, 102.82batch/s]

Training batches on cuda:  43%|████▎     | 181/425 [00:01<00:02, 101.33batch/s]

Training batches on cuda:  45%|████▌     | 192/425 [00:01<00:02, 102.49batch/s]

Training batches on cuda:  48%|████▊     | 203/425 [00:02<00:02, 103.57batch/s]

Training batches on cuda:  51%|█████     | 215/425 [00:02<00:01, 105.63batch/s]

Training batches on cuda:  53%|█████▎    | 227/425 [00:02<00:01, 107.21batch/s]

Training batches on cuda:  56%|█████▌    | 238/425 [00:02<00:01, 107.20batch/s]

Training batches on cuda:  59%|█████▊    | 249/425 [00:02<00:01, 101.95batch/s]

Training batches on cuda:  61%|██████    | 260/425 [00:02<00:01, 103.39batch/s]

Training batches on cuda:  64%|██████▍   | 271/425 [00:02<00:01, 90.78batch/s] 

Training batches on cuda:  66%|██████▋   | 282/425 [00:02<00:01, 94.69batch/s]

Training batches on cuda:  69%|██████▉   | 293/425 [00:02<00:01, 96.82batch/s]

Training batches on cuda:  72%|███████▏  | 304/425 [00:03<00:01, 100.42batch/s]

Training batches on cuda:  74%|███████▍  | 315/425 [00:03<00:01, 102.18batch/s]

Training batches on cuda:  77%|███████▋  | 326/425 [00:03<00:00, 102.08batch/s]

Training batches on cuda:  79%|███████▉  | 337/425 [00:03<00:00, 104.26batch/s]

Training batches on cuda:  82%|████████▏ | 349/425 [00:03<00:00, 106.60batch/s]

Training batches on cuda:  85%|████████▍ | 361/425 [00:03<00:00, 109.81batch/s]

Training batches on cuda:  88%|████████▊ | 373/425 [00:03<00:00, 109.40batch/s]

Training batches on cuda:  91%|█████████ | 385/425 [00:03<00:00, 110.11batch/s]

Training batches on cuda:  93%|█████████▎| 397/425 [00:03<00:00, 107.93batch/s]

Training batches on cuda:  96%|█████████▌| 408/425 [00:03<00:00, 107.53batch/s]

Training batches on cuda:  99%|█████████▊| 419/425 [00:04<00:00, 106.91batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:42<00:04,  4.32s/epoch, loss=0.00292, prev_loss=0.00292]2021-09-16:18:05:23,555 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:05:24,583 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:05:39,552 INFO     [evaluator.py:657] Evaluation took 14.97s seconds
2021-09-16:18:05:39,697 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00,  9.20s/epoch, loss=0.00292, prev_loss=0.00292]
Training epochs on cuda: 100%|██████████| 10/10 [00:58<00:00,  5.87s/epoch, loss=0.00292, prev_loss=0.00292]
2021-09-16:18:05:39,698 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp9pmhgsvy'
2021-09-16:18:05:39,793 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9pmhgsvy' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.31ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.40ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:05:54,661 INFO     [evaluator.py:657] Evaluation took 14.85s seconds
[I 2021-09-16 18:05:54,667] Trial 28 finished with value: -0.0014360649101399225 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.22343672538652473, 'optimizer.lr': 0.031004100849755946, 'negative_sampler.num_negs_per_pos': 24, 'training.batch_size': 732}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:05:54,689 WARNING  [api.py:823] No random seed is specified. Setting to 3118491512.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 10/936 [00:00<00:09, 98.48batch/s]

Training batches on cuda:   3%|▎         | 27/936 [00:00<00:08, 112.04batch/s]

Training batches on cuda:   5%|▍         | 44/936 [00:00<00:07, 123.79batch/s]

Training batches on cuda:   6%|▋         | 60/936 [00:00<00:06, 132.14batch/s]

Training batches on cuda:   8%|▊         | 77/936 [00:00<00:06, 140.45batch/s]

Training batches on cuda:  10%|█         | 94/936 [00:00<00:05, 146.92batch/s]

Training batches on cuda:  12%|█▏        | 111/936 [00:00<00:05, 152.42batch/s]

Training batches on cuda:  14%|█▎        | 128/936 [00:00<00:05, 156.23batch/s]

Training batches on cuda:  15%|█▌        | 145/936 [00:00<00:04, 159.65batch/s]

Training batches on cuda:  17%|█▋        | 162/936 [00:01<00:04, 161.55batch/s]

Training batches on cuda:  19%|█▉        | 181/936 [00:01<00:04, 167.73batch/s]

Training batches on cuda:  21%|██▏       | 201/936 [00:01<00:04, 174.39batch/s]

Training batches on cuda:  23%|██▎       | 219/936 [00:01<00:04, 175.24batch/s]

Training batches on cuda:  25%|██▌       | 237/936 [00:01<00:04, 171.22batch/s]

Training batches on cuda:  27%|██▋       | 256/936 [00:01<00:03, 176.37batch/s]

Training batches on cuda:  29%|██▉       | 275/936 [00:01<00:03, 177.81batch/s]

Training batches on cuda:  31%|███▏      | 294/936 [00:01<00:03, 180.80batch/s]

Training batches on cuda:  33%|███▎      | 313/936 [00:01<00:03, 177.66batch/s]

Training batches on cuda:  35%|███▌      | 331/936 [00:01<00:03, 174.99batch/s]

Training batches on cuda:  37%|███▋      | 349/936 [00:02<00:03, 173.73batch/s]

Training batches on cuda:  39%|███▉      | 367/936 [00:02<00:03, 167.63batch/s]

Training batches on cuda:  41%|████      | 384/936 [00:02<00:03, 168.08batch/s]

Training batches on cuda:  43%|████▎     | 401/936 [00:02<00:03, 166.29batch/s]

Training batches on cuda:  45%|████▍     | 418/936 [00:02<00:03, 165.06batch/s]

Training batches on cuda:  46%|████▋     | 435/936 [00:02<00:03, 165.21batch/s]

Training batches on cuda:  48%|████▊     | 452/936 [00:02<00:02, 165.26batch/s]

Training batches on cuda:  50%|█████     | 469/936 [00:02<00:02, 164.07batch/s]

Training batches on cuda:  52%|█████▏    | 486/936 [00:02<00:02, 165.13batch/s]

Training batches on cuda:  54%|█████▎    | 503/936 [00:03<00:02, 163.35batch/s]

Training batches on cuda:  56%|█████▌    | 520/936 [00:03<00:02, 144.96batch/s]

Training batches on cuda:  57%|█████▋    | 537/936 [00:03<00:02, 151.03batch/s]

Training batches on cuda:  59%|█████▉    | 555/936 [00:03<00:02, 156.81batch/s]

Training batches on cuda:  61%|██████    | 573/936 [00:03<00:02, 160.68batch/s]

Training batches on cuda:  63%|██████▎   | 591/936 [00:03<00:02, 164.39batch/s]

Training batches on cuda:  65%|██████▌   | 609/936 [00:03<00:01, 167.16batch/s]

Training batches on cuda:  67%|██████▋   | 627/936 [00:03<00:01, 170.07batch/s]

Training batches on cuda:  69%|██████▉   | 645/936 [00:03<00:01, 166.59batch/s]

Training batches on cuda:  71%|███████   | 662/936 [00:04<00:01, 158.20batch/s]

Training batches on cuda:  73%|███████▎  | 680/936 [00:04<00:01, 161.95batch/s]

Training batches on cuda:  75%|███████▍  | 698/936 [00:04<00:01, 166.52batch/s]

Training batches on cuda:  76%|███████▋  | 715/936 [00:04<00:01, 163.89batch/s]

Training batches on cuda:  78%|███████▊  | 732/936 [00:04<00:01, 164.93batch/s]

Training batches on cuda:  80%|████████  | 749/936 [00:04<00:01, 161.48batch/s]

Training batches on cuda:  82%|████████▏ | 767/936 [00:04<00:01, 165.81batch/s]

Training batches on cuda:  84%|████████▍ | 788/936 [00:04<00:00, 174.68batch/s]

Training batches on cuda:  86%|████████▌ | 807/936 [00:04<00:00, 177.04batch/s]

Training batches on cuda:  88%|████████▊ | 826/936 [00:04<00:00, 180.66batch/s]

Training batches on cuda:  90%|█████████ | 845/936 [00:05<00:00, 171.91batch/s]

Training batches on cuda:  92%|█████████▏| 863/936 [00:05<00:00, 167.95batch/s]

Training batches on cuda:  94%|█████████▍| 880/936 [00:05<00:00, 166.28batch/s]

Training batches on cuda:  96%|█████████▌| 898/936 [00:05<00:00, 167.90batch/s]

Training batches on cuda:  98%|█████████▊| 916/936 [00:05<00:00, 169.20batch/s]

Training batches on cuda: 100%|█████████▉| 934/936 [00:05<00:00, 169.90batch/s]

                                                                               
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.00919, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:52,  5.88s/epoch, loss=0.00919, prev_loss=nan]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 13/936 [00:00<00:07, 127.87batch/s]

Training batches on cuda:   3%|▎         | 30/936 [00:00<00:06, 137.72batch/s]

Training batches on cuda:   5%|▌         | 49/936 [00:00<00:05, 150.09batch/s]

Training batches on cuda:   7%|▋         | 69/936 [00:00<00:05, 161.34batch/s]

Training batches on cuda:   9%|▉         | 85/936 [00:00<00:05, 160.34batch/s]

Training batches on cuda:  11%|█         | 103/936 [00:00<00:05, 165.52batch/s]

Training batches on cuda:  13%|█▎        | 119/936 [00:00<00:05, 162.77batch/s]

Training batches on cuda:  15%|█▍        | 136/936 [00:00<00:04, 162.31batch/s]

Training batches on cuda:  16%|█▌        | 152/936 [00:00<00:04, 159.87batch/s]

Training batches on cuda:  18%|█▊        | 168/936 [00:01<00:04, 158.71batch/s]

Training batches on cuda:  20%|█▉        | 187/936 [00:01<00:04, 164.39batch/s]

Training batches on cuda:  22%|██▏       | 204/936 [00:01<00:04, 165.89batch/s]

Training batches on cuda:  24%|██▍       | 223/936 [00:01<00:04, 170.49batch/s]

Training batches on cuda:  26%|██▌       | 242/936 [00:01<00:03, 173.99batch/s]

Training batches on cuda:  28%|██▊       | 261/936 [00:01<00:03, 177.60batch/s]

Training batches on cuda:  30%|███       | 282/936 [00:01<00:03, 183.79batch/s]

Training batches on cuda:  32%|███▏      | 302/936 [00:01<00:03, 187.98batch/s]

Training batches on cuda:  34%|███▍      | 321/936 [00:01<00:03, 180.23batch/s]

Training batches on cuda:  36%|███▋      | 341/936 [00:01<00:03, 185.45batch/s]

Training batches on cuda:  39%|███▊      | 362/936 [00:02<00:03, 189.85batch/s]

Training batches on cuda:  41%|████      | 382/936 [00:02<00:02, 192.24batch/s]

Training batches on cuda:  43%|████▎     | 402/936 [00:02<00:02, 187.48batch/s]

Training batches on cuda:  45%|████▍     | 421/936 [00:02<00:02, 178.20batch/s]

Training batches on cuda:  47%|████▋     | 439/936 [00:02<00:02, 167.68batch/s]

Training batches on cuda:  49%|████▉     | 457/936 [00:02<00:02, 170.94batch/s]

Training batches on cuda:  51%|█████     | 477/936 [00:02<00:02, 177.93batch/s]

Training batches on cuda:  53%|█████▎    | 499/936 [00:02<00:02, 186.93batch/s]

Training batches on cuda:  55%|█████▌    | 519/936 [00:02<00:02, 190.10batch/s]

Training batches on cuda:  58%|█████▊    | 539/936 [00:03<00:02, 190.92batch/s]

Training batches on cuda:  60%|█████▉    | 559/936 [00:03<00:02, 174.68batch/s]

Training batches on cuda:  62%|██████▏   | 577/936 [00:03<00:02, 163.48batch/s]

Training batches on cuda:  63%|██████▎   | 594/936 [00:03<00:02, 165.05batch/s]

Training batches on cuda:  65%|██████▌   | 612/936 [00:03<00:01, 169.06batch/s]

Training batches on cuda:  68%|██████▊   | 632/936 [00:03<00:01, 176.21batch/s]

Training batches on cuda:  70%|██████▉   | 651/936 [00:03<00:01, 179.66batch/s]

Training batches on cuda:  72%|███████▏  | 670/936 [00:03<00:01, 177.49batch/s]

Training batches on cuda:  74%|███████▎  | 688/936 [00:03<00:01, 177.77batch/s]

Training batches on cuda:  75%|███████▌  | 706/936 [00:04<00:01, 170.69batch/s]

Training batches on cuda:  77%|███████▋  | 724/936 [00:04<00:01, 161.78batch/s]

Training batches on cuda:  79%|███████▉  | 741/936 [00:04<00:01, 157.37batch/s]

Training batches on cuda:  81%|████████  | 757/936 [00:04<00:01, 153.82batch/s]

Training batches on cuda:  83%|████████▎ | 773/936 [00:04<00:01, 151.94batch/s]

Training batches on cuda:  84%|████████▍ | 789/936 [00:04<00:00, 151.35batch/s]

Training batches on cuda:  86%|████████▌ | 806/936 [00:04<00:00, 155.31batch/s]

Training batches on cuda:  88%|████████▊ | 823/936 [00:04<00:00, 157.89batch/s]

Training batches on cuda:  90%|████████▉ | 839/936 [00:04<00:00, 156.00batch/s]

Training batches on cuda:  91%|█████████▏| 856/936 [00:05<00:00, 158.68batch/s]

Training batches on cuda:  93%|█████████▎| 873/936 [00:05<00:00, 159.87batch/s]

Training batches on cuda:  95%|█████████▌| 890/936 [00:05<00:00, 158.85batch/s]

Training batches on cuda:  97%|█████████▋| 907/936 [00:05<00:00, 161.10batch/s]

Training batches on cuda:  99%|█████████▊| 924/936 [00:05<00:00, 160.60batch/s]

                                                                               
Training epochs on cuda:  10%|█         | 1/10 [00:11<00:52,  5.88s/epoch, loss=0.00919, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:46,  5.82s/epoch, loss=0.00919, prev_loss=nan]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 11/936 [00:00<00:08, 105.26batch/s]

Training batches on cuda:   3%|▎         | 29/936 [00:00<00:07, 119.34batch/s]

Training batches on cuda:   5%|▌         | 47/936 [00:00<00:06, 132.14batch/s]

Training batches on cuda:   7%|▋         | 65/936 [00:00<00:06, 142.48batch/s]

Training batches on cuda:   9%|▊         | 81/936 [00:00<00:05, 146.62batch/s]

Training batches on cuda:  11%|█         | 99/936 [00:00<00:05, 154.76batch/s]

Training batches on cuda:  13%|█▎        | 119/936 [00:00<00:04, 165.20batch/s]

Training batches on cuda:  15%|█▍        | 138/936 [00:00<00:04, 170.49batch/s]

Training batches on cuda:  17%|█▋        | 156/936 [00:00<00:04, 172.47batch/s]

Training batches on cuda:  19%|█▉        | 177/936 [00:01<00:04, 181.10batch/s]

Training batches on cuda:  21%|██        | 197/936 [00:01<00:03, 184.84batch/s]

Training batches on cuda:  23%|██▎       | 217/936 [00:01<00:03, 188.06batch/s]

Training batches on cuda:  25%|██▌       | 236/936 [00:01<00:03, 188.03batch/s]

Training batches on cuda:  27%|██▋       | 257/936 [00:01<00:03, 193.83batch/s]

Training batches on cuda:  30%|██▉       | 278/936 [00:01<00:03, 196.59batch/s]

Training batches on cuda:  32%|███▏      | 299/936 [00:01<00:03, 197.83batch/s]

Training batches on cuda:  34%|███▍      | 320/936 [00:01<00:03, 199.20batch/s]

Training batches on cuda:  36%|███▋      | 341/936 [00:01<00:02, 200.27batch/s]

Training batches on cuda:  39%|███▊      | 362/936 [00:01<00:02, 198.31batch/s]

Training batches on cuda:  41%|████      | 382/936 [00:02<00:02, 196.22batch/s]

Training batches on cuda:  43%|████▎     | 402/936 [00:02<00:02, 196.41batch/s]

Training batches on cuda:  45%|████▌     | 422/936 [00:02<00:02, 194.38batch/s]

Training batches on cuda:  47%|████▋     | 442/936 [00:02<00:02, 183.24batch/s]

Training batches on cuda:  49%|████▉     | 461/936 [00:02<00:02, 179.97batch/s]

Training batches on cuda:  51%|█████▏    | 480/936 [00:02<00:02, 173.93batch/s]

Training batches on cuda:  53%|█████▎    | 498/936 [00:02<00:02, 169.16batch/s]

Training batches on cuda:  55%|█████▌    | 517/936 [00:02<00:02, 174.64batch/s]

Training batches on cuda:  57%|█████▋    | 538/936 [00:02<00:02, 182.07batch/s]

Training batches on cuda:  60%|█████▉    | 557/936 [00:03<00:02, 181.82batch/s]

Training batches on cuda:  62%|██████▏   | 577/936 [00:03<00:01, 186.31batch/s]

Training batches on cuda:  64%|██████▍   | 597/936 [00:03<00:01, 189.56batch/s]

Training batches on cuda:  66%|██████▌   | 617/936 [00:03<00:01, 186.32batch/s]

Training batches on cuda:  68%|██████▊   | 636/936 [00:03<00:01, 168.73batch/s]

Training batches on cuda:  70%|██████▉   | 654/936 [00:03<00:01, 161.28batch/s]

Training batches on cuda:  72%|███████▏  | 671/936 [00:03<00:01, 160.52batch/s]

Training batches on cuda:  74%|███████▎  | 688/936 [00:03<00:01, 162.42batch/s]

Training batches on cuda:  75%|███████▌  | 705/936 [00:03<00:01, 162.86batch/s]

Training batches on cuda:  77%|███████▋  | 722/936 [00:04<00:01, 163.06batch/s]

Training batches on cuda:  79%|███████▉  | 739/936 [00:04<00:01, 164.50batch/s]

Training batches on cuda:  81%|████████  | 756/936 [00:04<00:01, 160.09batch/s]

Training batches on cuda:  83%|████████▎ | 773/936 [00:04<00:01, 159.67batch/s]

Training batches on cuda:  85%|████████▍ | 792/936 [00:04<00:00, 165.78batch/s]

Training batches on cuda:  86%|████████▋ | 809/936 [00:04<00:00, 166.64batch/s]

Training batches on cuda:  88%|████████▊ | 826/936 [00:04<00:00, 165.25batch/s]

Training batches on cuda:  90%|█████████ | 843/936 [00:04<00:00, 164.57batch/s]

Training batches on cuda:  92%|█████████▏| 860/936 [00:04<00:00, 160.06batch/s]

Training batches on cuda:  94%|█████████▎| 877/936 [00:04<00:00, 155.04batch/s]

Training batches on cuda:  95%|█████████▌| 893/936 [00:05<00:00, 153.19batch/s]

Training batches on cuda:  97%|█████████▋| 909/936 [00:05<00:00, 154.80batch/s]

Training batches on cuda:  99%|█████████▉| 926/936 [00:05<00:00, 158.10batch/s]

                                                                               
Training epochs on cuda:  20%|██        | 2/10 [00:17<00:46,  5.82s/epoch, loss=0.00919, prev_loss=0.00919]
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:40,  5.77s/epoch, loss=0.00919, prev_loss=0.00919]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 13/936 [00:00<00:07, 125.63batch/s]

Training batches on cuda:   3%|▎         | 32/936 [00:00<00:06, 138.88batch/s]

Training batches on cuda:   5%|▌         | 51/936 [00:00<00:05, 150.11batch/s]

Training batches on cuda:   7%|▋         | 70/936 [00:00<00:05, 159.62batch/s]

Training batches on cuda:  10%|▉         | 89/936 [00:00<00:05, 166.60batch/s]

Training batches on cuda:  11%|█▏        | 107/936 [00:00<00:04, 169.71batch/s]

Training batches on cuda:  13%|█▎        | 125/936 [00:00<00:04, 170.00batch/s]

Training batches on cuda:  15%|█▌        | 143/936 [00:00<00:04, 172.83batch/s]

Training batches on cuda:  17%|█▋        | 161/936 [00:00<00:04, 172.86batch/s]

Training batches on cuda:  19%|█▉        | 179/936 [00:01<00:04, 174.93batch/s]

Training batches on cuda:  21%|██        | 197/936 [00:01<00:04, 174.91batch/s]

Training batches on cuda:  23%|██▎       | 215/936 [00:01<00:04, 176.33batch/s]

Training batches on cuda:  25%|██▍       | 233/936 [00:01<00:04, 174.93batch/s]

Training batches on cuda:  27%|██▋       | 251/936 [00:01<00:03, 173.75batch/s]

Training batches on cuda:  29%|██▉       | 271/936 [00:01<00:03, 178.90batch/s]

Training batches on cuda:  31%|███       | 291/936 [00:01<00:03, 182.79batch/s]

Training batches on cuda:  33%|███▎      | 311/936 [00:01<00:03, 187.22batch/s]

Training batches on cuda:  35%|███▌      | 331/936 [00:01<00:03, 190.16batch/s]

Training batches on cuda:  38%|███▊      | 351/936 [00:01<00:03, 191.92batch/s]

Training batches on cuda:  40%|███▉      | 371/936 [00:02<00:02, 192.23batch/s]

Training batches on cuda:  42%|████▏     | 391/936 [00:02<00:02, 193.17batch/s]

Training batches on cuda:  44%|████▍     | 411/936 [00:02<00:02, 192.84batch/s]

Training batches on cuda:  46%|████▌     | 432/936 [00:02<00:02, 196.04batch/s]

Training batches on cuda:  48%|████▊     | 453/936 [00:02<00:02, 197.79batch/s]

Training batches on cuda:  51%|█████     | 473/936 [00:02<00:02, 194.85batch/s]

Training batches on cuda:  53%|█████▎    | 494/936 [00:02<00:02, 198.69batch/s]

Training batches on cuda:  55%|█████▍    | 514/936 [00:02<00:02, 195.62batch/s]

Training batches on cuda:  57%|█████▋    | 534/936 [00:02<00:02, 193.16batch/s]

Training batches on cuda:  59%|█████▉    | 554/936 [00:02<00:01, 193.31batch/s]

Training batches on cuda:  61%|██████▏   | 575/936 [00:03<00:01, 196.30batch/s]

Training batches on cuda:  64%|██████▎   | 595/936 [00:03<00:01, 185.10batch/s]

Training batches on cuda:  66%|██████▌   | 617/936 [00:03<00:01, 192.30batch/s]

Training batches on cuda:  68%|██████▊   | 638/936 [00:03<00:01, 196.69batch/s]

Training batches on cuda:  70%|███████   | 659/936 [00:03<00:01, 198.73batch/s]

Training batches on cuda:  73%|███████▎  | 681/936 [00:03<00:01, 204.28batch/s]

Training batches on cuda:  75%|███████▌  | 702/936 [00:03<00:01, 195.88batch/s]

Training batches on cuda:  77%|███████▋  | 723/936 [00:03<00:01, 199.61batch/s]

Training batches on cuda:  80%|███████▉  | 745/936 [00:03<00:00, 202.57batch/s]

Training batches on cuda:  82%|████████▏ | 766/936 [00:04<00:00, 203.88batch/s]

Training batches on cuda:  84%|████████▍ | 789/936 [00:04<00:00, 209.09batch/s]

Training batches on cuda:  87%|████████▋ | 811/936 [00:04<00:00, 209.87batch/s]

Training batches on cuda:  89%|████████▉ | 834/936 [00:04<00:00, 213.77batch/s]

Training batches on cuda:  91%|█████████▏| 856/936 [00:04<00:00, 212.39batch/s]

Training batches on cuda:  94%|█████████▍| 878/936 [00:04<00:00, 209.38batch/s]

Training batches on cuda:  96%|█████████▌| 899/936 [00:04<00:00, 206.07batch/s]

Training batches on cuda:  98%|█████████▊| 920/936 [00:04<00:00, 200.03batch/s]

                                                                               
Training epochs on cuda:  30%|███       | 3/10 [00:22<00:40,  5.77s/epoch, loss=0.00919, prev_loss=0.00919]
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:33,  5.58s/epoch, loss=0.00919, prev_loss=0.00919]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 16/936 [00:00<00:05, 158.72batch/s]

Training batches on cuda:   4%|▍         | 37/936 [00:00<00:05, 169.65batch/s]

Training batches on cuda:   6%|▌         | 57/936 [00:00<00:04, 177.66batch/s]

Training batches on cuda:   8%|▊         | 78/936 [00:00<00:04, 184.41batch/s]

Training batches on cuda:  11%|█         | 99/936 [00:00<00:04, 191.35batch/s]

Training batches on cuda:  13%|█▎        | 120/936 [00:00<00:04, 193.51batch/s]

Training batches on cuda:  15%|█▌        | 143/936 [00:00<00:03, 202.83batch/s]

Training batches on cuda:  18%|█▊        | 167/936 [00:00<00:03, 210.47batch/s]

Training batches on cuda:  20%|██        | 191/936 [00:00<00:03, 216.22batch/s]

Training batches on cuda:  23%|██▎       | 214/936 [00:01<00:03, 218.93batch/s]

Training batches on cuda:  25%|██▌       | 238/936 [00:01<00:03, 223.42batch/s]

Training batches on cuda:  28%|██▊       | 261/936 [00:01<00:03, 224.61batch/s]

Training batches on cuda:  30%|███       | 285/936 [00:01<00:02, 227.48batch/s]

Training batches on cuda:  33%|███▎      | 308/936 [00:01<00:03, 206.86batch/s]

Training batches on cuda:  35%|███▌      | 329/936 [00:01<00:03, 194.56batch/s]

Training batches on cuda:  37%|███▋      | 349/936 [00:01<00:03, 188.17batch/s]

Training batches on cuda:  39%|███▉      | 369/936 [00:01<00:03, 180.49batch/s]

Training batches on cuda:  41%|████▏     | 388/936 [00:01<00:03, 176.55batch/s]

Training batches on cuda:  43%|████▎     | 407/936 [00:02<00:02, 178.72batch/s]

Training batches on cuda:  46%|████▌     | 427/936 [00:02<00:02, 184.38batch/s]

Training batches on cuda:  48%|████▊     | 447/936 [00:02<00:02, 186.93batch/s]

Training batches on cuda:  50%|████▉     | 466/936 [00:02<00:02, 185.11batch/s]

Training batches on cuda:  52%|█████▏    | 485/936 [00:02<00:02, 183.66batch/s]

Training batches on cuda:  54%|█████▍    | 504/936 [00:02<00:02, 172.09batch/s]

Training batches on cuda:  56%|█████▌    | 522/936 [00:02<00:02, 173.84batch/s]

Training batches on cuda:  58%|█████▊    | 540/936 [00:02<00:02, 164.45batch/s]

Training batches on cuda:  60%|█████▉    | 557/936 [00:02<00:02, 163.74batch/s]

Training batches on cuda:  61%|██████▏   | 574/936 [00:03<00:02, 165.36batch/s]

Training batches on cuda:  63%|██████▎   | 591/936 [00:03<00:02, 166.59batch/s]

Training batches on cuda:  65%|██████▍   | 608/936 [00:03<00:02, 162.94batch/s]

Training batches on cuda:  67%|██████▋   | 626/936 [00:03<00:01, 165.72batch/s]

Training batches on cuda:  69%|██████▉   | 644/936 [00:03<00:01, 169.21batch/s]

Training batches on cuda:  71%|███████   | 661/936 [00:03<00:01, 169.23batch/s]

Training batches on cuda:  72%|███████▏  | 678/936 [00:03<00:01, 169.04batch/s]

Training batches on cuda:  74%|███████▍  | 695/936 [00:03<00:01, 168.40batch/s]

Training batches on cuda:  76%|███████▌  | 712/936 [00:03<00:01, 167.93batch/s]

Training batches on cuda:  78%|███████▊  | 729/936 [00:03<00:01, 168.07batch/s]

Training batches on cuda:  80%|███████▉  | 747/936 [00:04<00:01, 168.75batch/s]

Training batches on cuda:  82%|████████▏ | 764/936 [00:04<00:01, 165.06batch/s]

Training batches on cuda:  83%|████████▎ | 781/936 [00:04<00:00, 166.31batch/s]

Training batches on cuda:  85%|████████▌ | 798/936 [00:04<00:00, 163.30batch/s]

Training batches on cuda:  87%|████████▋ | 815/936 [00:04<00:00, 159.38batch/s]

Training batches on cuda:  89%|████████▉ | 831/936 [00:04<00:00, 155.86batch/s]

Training batches on cuda:  90%|█████████ | 847/936 [00:04<00:00, 155.96batch/s]

Training batches on cuda:  92%|█████████▏| 863/936 [00:04<00:00, 155.21batch/s]

Training batches on cuda:  94%|█████████▍| 881/936 [00:04<00:00, 161.44batch/s]

Training batches on cuda:  96%|█████████▌| 900/936 [00:04<00:00, 167.65batch/s]

Training batches on cuda:  98%|█████████▊| 918/936 [00:05<00:00, 170.04batch/s]

Training batches on cuda: 100%|██████████| 936/936 [00:05<00:00, 171.60batch/s]

                                                                               
Training epochs on cuda:  40%|████      | 4/10 [00:27<00:33,  5.58s/epoch, loss=0.00919, prev_loss=0.00919]
Training epochs on cuda:  50%|█████     | 5/10 [00:27<00:27,  5.52s/epoch, loss=0.00919, prev_loss=0.00919]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 15/936 [00:00<00:06, 142.79batch/s]

Training batches on cuda:   4%|▎         | 33/936 [00:00<00:05, 151.87batch/s]

Training batches on cuda:   5%|▌         | 50/936 [00:00<00:05, 156.80batch/s]

Training batches on cuda:   7%|▋         | 68/936 [00:00<00:05, 162.78batch/s]

Training batches on cuda:   9%|▉         | 87/936 [00:00<00:05, 168.69batch/s]

Training batches on cuda:  11%|█         | 104/936 [00:00<00:04, 168.70batch/s]

Training batches on cuda:  13%|█▎        | 122/936 [00:00<00:04, 170.63batch/s]

Training batches on cuda:  15%|█▍        | 140/936 [00:00<00:04, 171.02batch/s]

Training batches on cuda:  17%|█▋        | 157/936 [00:00<00:04, 159.25batch/s]

Training batches on cuda:  18%|█▊        | 173/936 [00:01<00:04, 157.15batch/s]

Training batches on cuda:  20%|██        | 189/936 [00:01<00:04, 156.21batch/s]

Training batches on cuda:  22%|██▏       | 205/936 [00:01<00:04, 152.28batch/s]

Training batches on cuda:  24%|██▎       | 221/936 [00:01<00:04, 152.65batch/s]

Training batches on cuda:  25%|██▌       | 237/936 [00:01<00:04, 150.98batch/s]

Training batches on cuda:  27%|██▋       | 253/936 [00:01<00:04, 150.17batch/s]

Training batches on cuda:  29%|██▊       | 269/936 [00:01<00:04, 152.72batch/s]

Training batches on cuda:  31%|███       | 286/936 [00:01<00:04, 155.75batch/s]

Training batches on cuda:  32%|███▏      | 303/936 [00:01<00:03, 158.64batch/s]

Training batches on cuda:  34%|███▍      | 321/936 [00:01<00:03, 163.52batch/s]

Training batches on cuda:  36%|███▋      | 340/936 [00:02<00:03, 170.37batch/s]

Training batches on cuda:  38%|███▊      | 358/936 [00:02<00:03, 167.77batch/s]

Training batches on cuda:  40%|████      | 375/936 [00:02<00:03, 162.96batch/s]

Training batches on cuda:  42%|████▏     | 393/936 [00:02<00:03, 166.73batch/s]

Training batches on cuda:  44%|████▍     | 413/936 [00:02<00:03, 173.36batch/s]

Training batches on cuda:  46%|████▋     | 433/936 [00:02<00:02, 179.02batch/s]

Training batches on cuda:  48%|████▊     | 452/936 [00:02<00:03, 154.79batch/s]

Training batches on cuda:  50%|█████     | 469/936 [00:02<00:03, 153.52batch/s]

Training batches on cuda:  52%|█████▏    | 485/936 [00:03<00:02, 154.27batch/s]

Training batches on cuda:  54%|█████▎    | 503/936 [00:03<00:02, 160.26batch/s]

Training batches on cuda:  56%|█████▌    | 520/936 [00:03<00:02, 156.38batch/s]

Training batches on cuda:  57%|█████▋    | 538/936 [00:03<00:02, 160.65batch/s]

Training batches on cuda:  59%|█████▉    | 556/936 [00:03<00:02, 164.74batch/s]

Training batches on cuda:  61%|██████▏   | 574/936 [00:03<00:02, 165.74batch/s]

Training batches on cuda:  63%|██████▎   | 592/936 [00:03<00:02, 166.74batch/s]

Training batches on cuda:  65%|██████▌   | 610/936 [00:03<00:01, 168.87batch/s]

Training batches on cuda:  67%|██████▋   | 627/936 [00:03<00:01, 163.78batch/s]

Training batches on cuda:  69%|██████▉   | 644/936 [00:03<00:01, 157.65batch/s]

Training batches on cuda:  71%|███████   | 660/936 [00:04<00:01, 154.95batch/s]

Training batches on cuda:  72%|███████▏  | 676/936 [00:04<00:01, 152.64batch/s]

Training batches on cuda:  74%|███████▍  | 693/936 [00:04<00:01, 157.01batch/s]

Training batches on cuda:  76%|███████▌  | 712/936 [00:04<00:01, 164.65batch/s]

Training batches on cuda:  78%|███████▊  | 732/936 [00:04<00:01, 172.52batch/s]

Training batches on cuda:  80%|████████  | 751/936 [00:04<00:01, 176.35batch/s]

Training batches on cuda:  82%|████████▏ | 770/936 [00:04<00:00, 178.76batch/s]

Training batches on cuda:  84%|████████▍ | 789/936 [00:04<00:00, 180.05batch/s]

Training batches on cuda:  86%|████████▋ | 808/936 [00:04<00:00, 175.04batch/s]

Training batches on cuda:  88%|████████▊ | 826/936 [00:05<00:00, 169.38batch/s]

Training batches on cuda:  90%|█████████ | 846/936 [00:05<00:00, 175.24batch/s]

Training batches on cuda:  92%|█████████▏| 865/936 [00:05<00:00, 179.10batch/s]

Training batches on cuda:  95%|█████████▍| 885/936 [00:05<00:00, 182.55batch/s]

Training batches on cuda:  97%|█████████▋| 904/936 [00:05<00:00, 182.25batch/s]

Training batches on cuda:  99%|█████████▊| 923/936 [00:05<00:00, 182.77batch/s]

                                                                               
Training epochs on cuda:  50%|█████     | 5/10 [00:33<00:27,  5.52s/epoch, loss=0.00919, prev_loss=0.00919]
Training epochs on cuda:  60%|██████    | 6/10 [00:33<00:22,  5.63s/epoch, loss=0.00919, prev_loss=0.00919]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 13/936 [00:00<00:07, 129.13batch/s]

Training batches on cuda:   3%|▎         | 30/936 [00:00<00:06, 139.06batch/s]

Training batches on cuda:   5%|▌         | 47/936 [00:00<00:06, 146.00batch/s]

Training batches on cuda:   7%|▋         | 64/936 [00:00<00:05, 151.41batch/s]

Training batches on cuda:   9%|▉         | 82/936 [00:00<00:05, 157.38batch/s]

Training batches on cuda:  11%|█         | 99/936 [00:00<00:05, 160.00batch/s]

Training batches on cuda:  12%|█▏        | 114/936 [00:00<00:05, 156.41batch/s]

Training batches on cuda:  14%|█▍        | 129/936 [00:00<00:05, 153.34batch/s]

Training batches on cuda:  16%|█▌        | 149/936 [00:00<00:04, 162.89batch/s]

Training batches on cuda:  18%|█▊        | 167/936 [00:01<00:04, 167.13batch/s]

Training batches on cuda:  20%|█▉        | 184/936 [00:01<00:04, 161.62batch/s]

Training batches on cuda:  22%|██▏       | 202/936 [00:01<00:04, 164.70batch/s]

Training batches on cuda:  24%|██▎       | 221/936 [00:01<00:04, 169.18batch/s]

Training batches on cuda:  26%|██▌       | 240/936 [00:01<00:03, 174.14batch/s]

Training batches on cuda:  28%|██▊       | 260/936 [00:01<00:03, 178.96batch/s]

Training batches on cuda:  30%|██▉       | 279/936 [00:01<00:03, 179.57batch/s]

Training batches on cuda:  32%|███▏      | 298/936 [00:01<00:03, 181.95batch/s]

Training batches on cuda:  34%|███▍      | 317/936 [00:01<00:03, 172.13batch/s]

Training batches on cuda:  36%|███▌      | 336/936 [00:01<00:03, 176.89batch/s]

Training batches on cuda:  38%|███▊      | 354/936 [00:02<00:03, 171.74batch/s]

Training batches on cuda:  40%|███▉      | 372/936 [00:02<00:03, 169.03batch/s]

Training batches on cuda:  42%|████▏     | 389/936 [00:02<00:03, 163.94batch/s]

Training batches on cuda:  43%|████▎     | 406/936 [00:02<00:03, 161.54batch/s]

Training batches on cuda:  45%|████▌     | 423/936 [00:02<00:03, 161.54batch/s]

Training batches on cuda:  47%|████▋     | 440/936 [00:02<00:03, 162.68batch/s]

Training batches on cuda:  49%|████▉     | 457/936 [00:02<00:02, 164.65batch/s]

Training batches on cuda:  51%|█████     | 474/936 [00:02<00:02, 158.81batch/s]

Training batches on cuda:  52%|█████▏    | 491/936 [00:02<00:02, 159.42batch/s]

Training batches on cuda:  54%|█████▍    | 509/936 [00:03<00:02, 164.55batch/s]

Training batches on cuda:  56%|█████▌    | 526/936 [00:03<00:02, 148.31batch/s]

Training batches on cuda:  58%|█████▊    | 544/936 [00:03<00:02, 154.16batch/s]

Training batches on cuda:  60%|██████    | 562/936 [00:03<00:02, 158.76batch/s]

Training batches on cuda:  62%|██████▏   | 580/936 [00:03<00:02, 163.04batch/s]

Training batches on cuda:  64%|██████▍   | 600/936 [00:03<00:01, 172.31batch/s]

Training batches on cuda:  66%|██████▌   | 620/936 [00:03<00:01, 177.55batch/s]

Training batches on cuda:  68%|██████▊   | 638/936 [00:03<00:01, 169.51batch/s]

Training batches on cuda:  70%|███████   | 656/936 [00:03<00:01, 168.81batch/s]

Training batches on cuda:  72%|███████▏  | 674/936 [00:04<00:01, 169.79batch/s]

Training batches on cuda:  74%|███████▍  | 692/936 [00:04<00:01, 170.29batch/s]

Training batches on cuda:  76%|███████▌  | 710/936 [00:04<00:01, 170.99batch/s]

Training batches on cuda:  78%|███████▊  | 728/936 [00:04<00:01, 169.65batch/s]

Training batches on cuda:  80%|███████▉  | 746/936 [00:04<00:01, 170.23batch/s]

Training batches on cuda:  82%|████████▏ | 764/936 [00:04<00:01, 166.31batch/s]

Training batches on cuda:  83%|████████▎ | 781/936 [00:04<00:00, 166.98batch/s]

Training batches on cuda:  85%|████████▌ | 798/936 [00:04<00:00, 165.66batch/s]

Training batches on cuda:  87%|████████▋ | 816/936 [00:04<00:00, 167.71batch/s]

Training batches on cuda:  89%|████████▉ | 834/936 [00:04<00:00, 169.55batch/s]

Training batches on cuda:  91%|█████████ | 851/936 [00:05<00:00, 169.19batch/s]

Training batches on cuda:  93%|█████████▎| 868/936 [00:05<00:00, 160.96batch/s]

Training batches on cuda:  95%|█████████▍| 885/936 [00:05<00:00, 157.11batch/s]

Training batches on cuda:  96%|█████████▋| 901/936 [00:05<00:00, 155.19batch/s]

Training batches on cuda:  98%|█████████▊| 917/936 [00:05<00:00, 155.76batch/s]

Training batches on cuda: 100%|█████████▉| 934/936 [00:05<00:00, 159.18batch/s]

                                                                               
Training epochs on cuda:  60%|██████    | 6/10 [00:39<00:22,  5.63s/epoch, loss=0.00919, prev_loss=0.00919]
Training epochs on cuda:  70%|███████   | 7/10 [00:39<00:17,  5.69s/epoch, loss=0.00919, prev_loss=0.00919]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 14/936 [00:00<00:06, 139.89batch/s]

Training batches on cuda:   3%|▎         | 32/936 [00:00<00:06, 148.84batch/s]

Training batches on cuda:   6%|▌         | 53/936 [00:00<00:05, 161.28batch/s]

Training batches on cuda:   8%|▊         | 73/936 [00:00<00:05, 170.05batch/s]

Training batches on cuda:  10%|▉         | 92/936 [00:00<00:04, 173.24batch/s]

Training batches on cuda:  12%|█▏        | 108/936 [00:00<00:04, 167.29batch/s]

Training batches on cuda:  13%|█▎        | 124/936 [00:00<00:05, 161.54batch/s]

Training batches on cuda:  15%|█▍        | 140/936 [00:00<00:05, 158.23batch/s]

Training batches on cuda:  17%|█▋        | 156/936 [00:00<00:05, 147.66batch/s]

Training batches on cuda:  18%|█▊        | 171/936 [00:01<00:05, 148.19batch/s]

Training batches on cuda:  20%|█▉        | 186/936 [00:01<00:05, 142.87batch/s]

Training batches on cuda:  21%|██▏       | 201/936 [00:01<00:05, 144.92batch/s]

Training batches on cuda:  23%|██▎       | 218/936 [00:01<00:04, 149.62batch/s]

Training batches on cuda:  25%|██▌       | 234/936 [00:01<00:04, 152.47batch/s]

Training batches on cuda:  27%|██▋       | 253/936 [00:01<00:04, 160.75batch/s]

Training batches on cuda:  29%|██▉       | 271/936 [00:01<00:04, 163.49batch/s]

Training batches on cuda:  31%|███       | 289/936 [00:01<00:03, 168.11batch/s]

Training batches on cuda:  33%|███▎      | 307/936 [00:01<00:03, 170.36batch/s]

Training batches on cuda:  35%|███▍      | 325/936 [00:02<00:03, 161.59batch/s]

Training batches on cuda:  37%|███▋      | 342/936 [00:02<00:03, 152.46batch/s]

Training batches on cuda:  38%|███▊      | 358/936 [00:02<00:04, 143.43batch/s]

Training batches on cuda:  40%|███▉      | 373/936 [00:02<00:04, 136.31batch/s]

Training batches on cuda:  42%|████▏     | 392/936 [00:02<00:03, 148.64batch/s]

Training batches on cuda:  44%|████▍     | 411/936 [00:02<00:03, 157.44batch/s]

Training batches on cuda:  46%|████▌     | 429/936 [00:02<00:03, 161.40batch/s]

Training batches on cuda:  48%|████▊     | 446/936 [00:02<00:03, 160.58batch/s]

Training batches on cuda:  50%|████▉     | 465/936 [00:02<00:02, 167.41batch/s]

Training batches on cuda:  52%|█████▏    | 483/936 [00:03<00:02, 169.82batch/s]

Training batches on cuda:  54%|█████▎    | 501/936 [00:03<00:02, 170.79batch/s]

Training batches on cuda:  56%|█████▌    | 520/936 [00:03<00:02, 175.73batch/s]

Training batches on cuda:  58%|█████▊    | 541/936 [00:03<00:02, 182.73batch/s]

Training batches on cuda:  60%|█████▉    | 561/936 [00:03<00:02, 187.40batch/s]

Training batches on cuda:  62%|██████▏   | 581/936 [00:03<00:01, 188.75batch/s]

Training batches on cuda:  64%|██████▍   | 600/936 [00:03<00:01, 188.19batch/s]

Training batches on cuda:  66%|██████▌   | 619/936 [00:03<00:01, 188.56batch/s]

Training batches on cuda:  68%|██████▊   | 639/936 [00:03<00:01, 190.17batch/s]

Training batches on cuda:  70%|███████   | 659/936 [00:03<00:01, 188.17batch/s]

Training batches on cuda:  72%|███████▏  | 678/936 [00:04<00:01, 186.15batch/s]

Training batches on cuda:  75%|███████▍  | 698/936 [00:04<00:01, 189.54batch/s]

Training batches on cuda:  77%|███████▋  | 717/936 [00:04<00:01, 188.42batch/s]

Training batches on cuda:  79%|███████▊  | 737/936 [00:04<00:01, 189.55batch/s]

Training batches on cuda:  81%|████████  | 756/936 [00:04<00:00, 189.35batch/s]

Training batches on cuda:  83%|████████▎ | 775/936 [00:04<00:00, 188.23batch/s]

Training batches on cuda:  85%|████████▍ | 795/936 [00:04<00:00, 191.21batch/s]

Training batches on cuda:  87%|████████▋ | 815/936 [00:04<00:00, 189.40batch/s]

Training batches on cuda:  89%|████████▉ | 835/936 [00:04<00:00, 190.30batch/s]

Training batches on cuda:  91%|█████████▏| 855/936 [00:04<00:00, 192.23batch/s]

Training batches on cuda:  93%|█████████▎| 875/936 [00:05<00:00, 193.66batch/s]

Training batches on cuda:  96%|█████████▌| 895/936 [00:05<00:00, 194.23batch/s]

Training batches on cuda:  98%|█████████▊| 915/936 [00:05<00:00, 194.61batch/s]

Training batches on cuda: 100%|█████████▉| 935/936 [00:05<00:00, 195.45batch/s]

                                                                               
Training epochs on cuda:  70%|███████   | 7/10 [00:45<00:17,  5.69s/epoch, loss=0.00919, prev_loss=0.00919]
Training epochs on cuda:  80%|████████  | 8/10 [00:45<00:11,  5.67s/epoch, loss=0.00919, prev_loss=0.00919]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 14/936 [00:00<00:06, 135.50batch/s]

Training batches on cuda:   3%|▎         | 30/936 [00:00<00:06, 141.19batch/s]

Training batches on cuda:   5%|▍         | 46/936 [00:00<00:06, 146.11batch/s]

Training batches on cuda:   7%|▋         | 64/936 [00:00<00:05, 153.96batch/s]

Training batches on cuda:   9%|▉         | 84/936 [00:00<00:05, 163.98batch/s]

Training batches on cuda:  11%|█         | 103/936 [00:00<00:04, 169.14batch/s]

Training batches on cuda:  13%|█▎        | 122/936 [00:00<00:04, 172.72batch/s]

Training batches on cuda:  15%|█▍        | 140/936 [00:00<00:04, 174.43batch/s]

Training batches on cuda:  17%|█▋        | 159/936 [00:00<00:04, 176.58batch/s]

Training batches on cuda:  19%|█▉        | 178/936 [00:01<00:04, 178.49batch/s]

Training batches on cuda:  21%|██        | 197/936 [00:01<00:04, 179.05batch/s]

Training batches on cuda:  23%|██▎       | 216/936 [00:01<00:04, 179.45batch/s]

Training batches on cuda:  25%|██▌       | 235/936 [00:01<00:03, 181.38batch/s]

Training batches on cuda:  27%|██▋       | 255/936 [00:01<00:03, 185.45batch/s]

Training batches on cuda:  29%|██▉       | 275/936 [00:01<00:03, 188.19batch/s]

Training batches on cuda:  32%|███▏      | 295/936 [00:01<00:03, 189.38batch/s]

Training batches on cuda:  34%|███▎      | 314/936 [00:01<00:03, 174.68batch/s]

Training batches on cuda:  36%|███▌      | 334/936 [00:01<00:03, 181.12batch/s]

Training batches on cuda:  38%|███▊      | 354/936 [00:01<00:03, 185.95batch/s]

Training batches on cuda:  40%|███▉      | 374/936 [00:02<00:02, 189.25batch/s]

Training batches on cuda:  42%|████▏     | 394/936 [00:02<00:02, 185.21batch/s]

Training batches on cuda:  44%|████▍     | 413/936 [00:02<00:02, 184.72batch/s]

Training batches on cuda:  46%|████▌     | 432/936 [00:02<00:02, 184.47batch/s]

Training batches on cuda:  48%|████▊     | 451/936 [00:02<00:02, 185.13batch/s]

Training batches on cuda:  50%|█████     | 470/936 [00:02<00:02, 185.18batch/s]

Training batches on cuda:  52%|█████▏    | 489/936 [00:02<00:02, 185.12batch/s]

Training batches on cuda:  54%|█████▍    | 508/936 [00:02<00:02, 185.22batch/s]

Training batches on cuda:  56%|█████▋    | 527/936 [00:02<00:02, 184.57batch/s]

Training batches on cuda:  58%|█████▊    | 546/936 [00:03<00:02, 184.66batch/s]

Training batches on cuda:  60%|██████    | 565/936 [00:03<00:02, 183.88batch/s]

Training batches on cuda:  62%|██████▏   | 584/936 [00:03<00:01, 184.16batch/s]

Training batches on cuda:  64%|██████▍   | 603/936 [00:03<00:01, 182.55batch/s]

Training batches on cuda:  66%|██████▋   | 622/936 [00:03<00:01, 176.75batch/s]

Training batches on cuda:  68%|██████▊   | 640/936 [00:03<00:01, 176.16batch/s]

Training batches on cuda:  70%|███████   | 658/936 [00:03<00:01, 175.32batch/s]

Training batches on cuda:  72%|███████▏  | 676/936 [00:03<00:01, 176.08batch/s]

Training batches on cuda:  74%|███████▍  | 696/936 [00:03<00:01, 182.25batch/s]

Training batches on cuda:  77%|███████▋  | 717/936 [00:03<00:01, 188.20batch/s]

Training batches on cuda:  79%|███████▊  | 736/936 [00:04<00:01, 187.32batch/s]

Training batches on cuda:  81%|████████  | 755/936 [00:04<00:01, 165.24batch/s]

Training batches on cuda:  83%|████████▎ | 774/936 [00:04<00:00, 171.06batch/s]

Training batches on cuda:  85%|████████▍ | 793/936 [00:04<00:00, 175.52batch/s]

Training batches on cuda:  87%|████████▋ | 812/936 [00:04<00:00, 177.74batch/s]

Training batches on cuda:  89%|████████▉ | 831/936 [00:04<00:00, 179.32batch/s]

Training batches on cuda:  91%|█████████ | 850/936 [00:04<00:00, 179.99batch/s]

Training batches on cuda:  93%|█████████▎| 869/936 [00:04<00:00, 181.01batch/s]

Training batches on cuda:  95%|█████████▍| 888/936 [00:04<00:00, 182.01batch/s]

Training batches on cuda:  97%|█████████▋| 907/936 [00:05<00:00, 183.57batch/s]

Training batches on cuda:  99%|█████████▉| 927/936 [00:05<00:00, 185.19batch/s]

                                                                               
Training epochs on cuda:  80%|████████  | 8/10 [00:50<00:11,  5.67s/epoch, loss=0.00919, prev_loss=0.00919]
Training epochs on cuda:  90%|█████████ | 9/10 [00:50<00:05,  5.59s/epoch, loss=0.00919, prev_loss=0.00919]

Training batches on cuda:   0%|          | 0/936 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 16/936 [00:00<00:05, 156.71batch/s]

Training batches on cuda:   4%|▍         | 36/936 [00:00<00:05, 166.91batch/s]

Training batches on cuda:   6%|▌         | 56/936 [00:00<00:05, 174.36batch/s]

Training batches on cuda:   8%|▊         | 77/936 [00:00<00:04, 182.28batch/s]

Training batches on cuda:  10%|█         | 98/936 [00:00<00:04, 187.65batch/s]

Training batches on cuda:  13%|█▎        | 119/936 [00:00<00:04, 191.95batch/s]

Training batches on cuda:  15%|█▍        | 140/936 [00:00<00:04, 194.63batch/s]

Training batches on cuda:  17%|█▋        | 160/936 [00:00<00:03, 195.63batch/s]

Training batches on cuda:  19%|█▉        | 179/936 [00:00<00:03, 193.69batch/s]

Training batches on cuda:  21%|██▏       | 199/936 [00:01<00:03, 194.94batch/s]

Training batches on cuda:  23%|██▎       | 219/936 [00:01<00:03, 196.03batch/s]

Training batches on cuda:  26%|██▌       | 239/936 [00:01<00:03, 196.78batch/s]

Training batches on cuda:  28%|██▊       | 260/936 [00:01<00:03, 198.58batch/s]

Training batches on cuda:  30%|███       | 281/936 [00:01<00:03, 200.74batch/s]

Training batches on cuda:  32%|███▏      | 301/936 [00:01<00:03, 199.91batch/s]

Training batches on cuda:  34%|███▍      | 322/936 [00:01<00:03, 200.16batch/s]

Training batches on cuda:  37%|███▋      | 343/936 [00:01<00:02, 200.55batch/s]

Training batches on cuda:  39%|███▉      | 364/936 [00:01<00:02, 200.23batch/s]

Training batches on cuda:  41%|████      | 385/936 [00:01<00:02, 200.62batch/s]

Training batches on cuda:  43%|████▎     | 406/936 [00:02<00:02, 201.10batch/s]

Training batches on cuda:  46%|████▌     | 427/936 [00:02<00:02, 198.93batch/s]

Training batches on cuda:  48%|████▊     | 448/936 [00:02<00:02, 199.58batch/s]

Training batches on cuda:  50%|█████     | 469/936 [00:02<00:02, 200.28batch/s]

Training batches on cuda:  52%|█████▏    | 490/936 [00:02<00:02, 198.99batch/s]

Training batches on cuda:  54%|█████▍    | 510/936 [00:02<00:02, 195.40batch/s]

Training batches on cuda:  57%|█████▋    | 530/936 [00:02<00:02, 186.94batch/s]

Training batches on cuda:  59%|█████▉    | 550/936 [00:02<00:02, 188.54batch/s]

Training batches on cuda:  61%|██████    | 569/936 [00:02<00:01, 186.80batch/s]

Training batches on cuda:  63%|██████▎   | 588/936 [00:03<00:01, 184.12batch/s]

Training batches on cuda:  65%|██████▌   | 609/936 [00:03<00:01, 191.13batch/s]

Training batches on cuda:  67%|██████▋   | 631/936 [00:03<00:01, 198.91batch/s]

Training batches on cuda:  70%|██████▉   | 652/936 [00:03<00:01, 196.89batch/s]

Training batches on cuda:  72%|███████▏  | 673/936 [00:03<00:01, 200.12batch/s]

Training batches on cuda:  74%|███████▍  | 694/936 [00:03<00:01, 202.42batch/s]

Training batches on cuda:  76%|███████▋  | 715/936 [00:03<00:01, 203.82batch/s]

Training batches on cuda:  79%|███████▊  | 736/936 [00:03<00:00, 200.90batch/s]

Training batches on cuda:  81%|████████  | 757/936 [00:03<00:00, 199.37batch/s]

Training batches on cuda:  83%|████████▎ | 777/936 [00:03<00:00, 197.87batch/s]

Training batches on cuda:  85%|████████▌ | 797/936 [00:04<00:00, 198.40batch/s]

Training batches on cuda:  88%|████████▊ | 819/936 [00:04<00:00, 201.58batch/s]

Training batches on cuda:  90%|████████▉ | 840/936 [00:04<00:00, 202.44batch/s]

Training batches on cuda:  92%|█████████▏| 861/936 [00:04<00:00, 200.95batch/s]

Training batches on cuda:  94%|█████████▍| 882/936 [00:04<00:00, 199.53batch/s]

Training batches on cuda:  96%|█████████▋| 903/936 [00:04<00:00, 200.34batch/s]

Training batches on cuda:  99%|█████████▊| 924/936 [00:04<00:00, 201.16batch/s]

                                                                               
Training epochs on cuda:  90%|█████████ | 9/10 [00:55<00:05,  5.59s/epoch, loss=0.00919, prev_loss=0.00919]2021-09-16:18:06:50,523 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:06:51,557 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:07:06,497 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:18:07:06,635 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 10.22s/epoch, loss=0.00919, prev_loss=0.00919]
Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00,  7.15s/epoch, loss=0.00919, prev_loss=0.00919]
2021-09-16:18:07:06,635 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp2ozx8a8f'
2021-09-16:18:07:06,712 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp2ozx8a8f' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.53ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.65ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.65ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.66ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:18:07:21,471 INFO     [evaluator.py:657] Evaluation took 14.74s seconds
[I 2021-09-16 18:07:21,476] Trial 29 finished with value: 0.018908618826227075 and parameters: {'loss.margin': 3, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.06810237981930033, 'optimizer.lr': 0.008136371172701898, 'negative_sampler.num_negs_per_pos': 5, 'training.batch_size': 332}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:07:21,497 WARNING  [api.py:823] No random seed is specified. Setting to 2852144923.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 37.75batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:03, 42.19batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:02, 45.90batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:02, 48.97batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:02, 51.27batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:01, 53.18batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:00<00:01, 54.40batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:00<00:01, 55.44batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:00<00:01, 56.30batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:01, 56.82batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:01, 57.20batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 57.56batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:01, 57.61batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:01<00:01, 57.66batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:01<00:00, 57.90batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:01<00:00, 58.00batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:01<00:00, 57.74batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:01<00:00, 58.03batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:01<00:00, 58.18batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:02<00:00, 58.35batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:02<00:00, 58.20batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:02<00:00, 58.38batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:02<00:00, 58.13batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000592, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:23,  2.64s/epoch, loss=0.000592, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 36.41batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:03, 40.21batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:02, 43.76batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:02, 46.40batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:02, 47.95batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:02, 50.00batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:00<00:01, 51.02batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:00<00:01, 51.73batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:01<00:01, 52.28batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:01, 52.46batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:01, 53.48batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 54.60batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:01, 55.53batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:01<00:01, 56.35batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:01<00:00, 56.97batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:01<00:00, 57.12batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:01<00:00, 57.41batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:01<00:00, 57.49batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:02<00:00, 51.84batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:02<00:00, 53.57batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:02<00:00, 54.75batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:02<00:00, 55.65batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:02<00:00, 56.43batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:23,  2.64s/epoch, loss=0.000587, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:21,  2.68s/epoch, loss=0.000587, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 25.81batch/s]

Training batches on cuda:   6%|▋         | 9/140 [00:00<00:04, 30.63batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:03, 35.41batch/s]

Training batches on cuda:  15%|█▌        | 21/140 [00:00<00:02, 39.85batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:02, 43.47batch/s]

Training batches on cuda:  24%|██▎       | 33/140 [00:00<00:02, 47.04batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:00<00:01, 50.46batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:00<00:01, 53.15batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:00<00:01, 55.29batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:01<00:01, 56.82batch/s]

Training batches on cuda:  49%|████▊     | 68/140 [00:01<00:01, 58.15batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:01<00:01, 60.02batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:01<00:00, 62.15batch/s]

Training batches on cuda:  64%|██████▎   | 89/140 [00:01<00:00, 64.29batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:01<00:00, 65.66batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:01<00:00, 66.70batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:01<00:00, 67.46batch/s]

Training batches on cuda:  84%|████████▎ | 117/140 [00:01<00:00, 68.05batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:02<00:00, 68.38batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:02<00:00, 68.84batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:02<00:00, 69.01batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:21,  2.68s/epoch, loss=0.000575, prev_loss=0.000587]
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:18,  2.62s/epoch, loss=0.000575, prev_loss=0.000587]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:04, 32.16batch/s]

Training batches on cuda:   6%|▋         | 9/140 [00:00<00:03, 34.86batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:03, 39.09batch/s]

Training batches on cuda:  15%|█▌        | 21/140 [00:00<00:02, 42.88batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:02, 46.18batch/s]

Training batches on cuda:  24%|██▎       | 33/140 [00:00<00:02, 49.31batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:00<00:01, 51.77batch/s]

Training batches on cuda:  32%|███▏      | 45/140 [00:00<00:01, 53.50batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:00<00:01, 54.55batch/s]

Training batches on cuda:  41%|████      | 57/140 [00:01<00:01, 54.98batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:01, 54.88batch/s]

Training batches on cuda:  49%|████▉     | 69/140 [00:01<00:01, 53.14batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:01, 56.00batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:01<00:01, 53.06batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:01<00:01, 51.57batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:01<00:00, 52.39batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:01<00:00, 53.63batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:01<00:00, 54.92batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:02<00:00, 55.33batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:02<00:00, 55.99batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:02<00:00, 55.26batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:02<00:00, 56.02batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:02<00:00, 56.51batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:18,  2.62s/epoch, loss=0.000557, prev_loss=0.000575]
Training epochs on cuda:  40%|████      | 4/10 [00:10<00:16,  2.67s/epoch, loss=0.000557, prev_loss=0.000575]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 39.32batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:02, 44.68batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:02, 48.96batch/s]

Training batches on cuda:  18%|█▊        | 25/140 [00:00<00:02, 52.94batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:01, 56.61batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:00<00:01, 59.72batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:00<00:01, 62.39batch/s]

Training batches on cuda:  38%|███▊      | 53/140 [00:00<00:01, 60.90batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:00<00:01, 62.09batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:01, 63.49batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:01<00:01, 64.29batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:01<00:00, 64.47batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:01<00:00, 63.08batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:01<00:00, 59.04batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:01<00:00, 59.18batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:01<00:00, 58.63batch/s]

Training batches on cuda:  81%|████████  | 113/140 [00:01<00:00, 58.44batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:01<00:00, 58.56batch/s]

Training batches on cuda:  89%|████████▉ | 125/140 [00:02<00:00, 58.65batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:02<00:00, 58.74batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:02<00:00, 58.74batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:13<00:16,  2.67s/epoch, loss=0.000541, prev_loss=0.000557]
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:13,  2.62s/epoch, loss=0.000541, prev_loss=0.000557]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 34.08batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:03, 39.00batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:02, 43.19batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:02, 46.85batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:02, 49.71batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:02, 52.11batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:00<00:01, 53.97batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:00<00:01, 55.37batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:00<00:01, 56.35batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:01, 57.00batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:01, 57.35batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 57.47batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:01, 57.62batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:01<00:01, 56.99batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:01<00:01, 51.12batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:01<00:00, 52.81batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:01<00:00, 53.89batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:01<00:00, 55.32batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:02<00:00, 55.44batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:02<00:00, 56.06batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:02<00:00, 56.27batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:02<00:00, 55.72batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:02<00:00, 54.44batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:13,  2.62s/epoch, loss=0.000528, prev_loss=0.000541]
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:10,  2.66s/epoch, loss=0.000528, prev_loss=0.000541]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 34.31batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:03, 38.90batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:02, 41.76batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:02, 44.55batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:02, 46.76batch/s]

Training batches on cuda:  24%|██▎       | 33/140 [00:00<00:02, 47.02batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:00<00:02, 47.88batch/s]

Training batches on cuda:  32%|███▏      | 45/140 [00:00<00:01, 48.89batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:01, 49.89batch/s]

Training batches on cuda:  41%|████      | 57/140 [00:01<00:01, 51.10batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:01, 52.74batch/s]

Training batches on cuda:  49%|████▉     | 69/140 [00:01<00:01, 53.67batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:01<00:01, 54.33batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:01<00:01, 55.37batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:01<00:00, 55.51batch/s]

Training batches on cuda:  66%|██████▋   | 93/140 [00:01<00:00, 55.59batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:01<00:00, 55.94batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:01<00:00, 56.59batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:02<00:00, 56.48batch/s]

Training batches on cuda:  84%|████████▎ | 117/140 [00:02<00:00, 56.34batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:02<00:00, 56.08batch/s]

Training batches on cuda:  92%|█████████▏| 129/140 [00:02<00:00, 55.55batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:02<00:00, 55.66batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:10,  2.66s/epoch, loss=0.000438, prev_loss=0.000528]
Training epochs on cuda:  70%|███████   | 7/10 [00:18<00:08,  2.70s/epoch, loss=0.000438, prev_loss=0.000528]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 37.65batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:03, 41.97batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:02, 45.85batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:02, 50.23batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:02, 53.70batch/s]

Training batches on cuda:  26%|██▋       | 37/140 [00:00<00:01, 55.25batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:00<00:01, 54.96batch/s]

Training batches on cuda:  35%|███▌      | 49/140 [00:00<00:01, 52.32batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:00<00:01, 53.99batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:01<00:01, 53.68batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:01, 54.88batch/s]

Training batches on cuda:  52%|█████▏    | 73/140 [00:01<00:01, 52.53batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:01<00:01, 54.04batch/s]

Training batches on cuda:  61%|██████    | 85/140 [00:01<00:00, 55.18batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:01<00:00, 54.71batch/s]

Training batches on cuda:  69%|██████▉   | 97/140 [00:01<00:00, 51.51batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:01<00:00, 50.73batch/s]

Training batches on cuda:  78%|███████▊  | 109/140 [00:02<00:00, 52.91batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:02<00:00, 54.38batch/s]

Training batches on cuda:  86%|████████▋ | 121/140 [00:02<00:00, 55.18batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:02<00:00, 55.58batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:02<00:00, 56.33batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:02<00:00, 56.93batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:21<00:08,  2.70s/epoch, loss=0.000345, prev_loss=0.000438]
Training epochs on cuda:  80%|████████  | 8/10 [00:21<00:05,  2.74s/epoch, loss=0.000345, prev_loss=0.000438]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:04, 29.07batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:03, 34.55batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:03, 37.57batch/s]

Training batches on cuda:  15%|█▌        | 21/140 [00:00<00:02, 40.85batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:02, 43.12batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:02, 45.17batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:00<00:02, 47.46batch/s]

Training batches on cuda:  32%|███▏      | 45/140 [00:00<00:01, 50.74batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:00<00:01, 53.63batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:01, 55.94batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:01, 58.21batch/s]

Training batches on cuda:  52%|█████▏    | 73/140 [00:01<00:01, 59.93batch/s]

Training batches on cuda:  57%|█████▋    | 80/140 [00:01<00:00, 60.49batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:01<00:00, 56.73batch/s]

Training batches on cuda:  66%|██████▋   | 93/140 [00:01<00:00, 54.47batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:01<00:00, 53.39batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:01<00:00, 51.35batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:02<00:00, 51.23batch/s]

Training batches on cuda:  84%|████████▎ | 117/140 [00:02<00:00, 49.04batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:02<00:00, 48.16batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:02<00:00, 47.77batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:02<00:00, 48.04batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:02<00:00, 46.31batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:24<00:05,  2.74s/epoch, loss=0.0003, prev_loss=0.000345]  
Training epochs on cuda:  90%|█████████ | 9/10 [00:24<00:02,  2.79s/epoch, loss=0.0003, prev_loss=0.000345]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 39.65batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:02, 45.06batch/s]

Training batches on cuda:  12%|█▏        | 17/140 [00:00<00:02, 47.40batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:02, 49.82batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:02, 53.16batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:00<00:01, 54.81batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:00<00:01, 57.25batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:00<00:01, 60.24batch/s]

Training batches on cuda:  41%|████      | 57/140 [00:00<00:01, 62.58batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:01, 63.76batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:01<00:01, 61.44batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:01<00:01, 59.36batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:01<00:00, 58.10batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:01<00:00, 57.01batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:01<00:00, 57.16batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:01<00:00, 57.48batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:01<00:00, 57.42batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:01<00:00, 57.94batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:02<00:00, 57.79batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:02<00:00, 57.81batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:02<00:00, 57.86batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:02<00:00, 57.52batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:27<00:02,  2.79s/epoch, loss=0.00027, prev_loss=0.0003] 2021-09-16:18:07:49,71 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:07:50,63 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:08:04,996 INFO     [evaluator.py:657] Evaluation took 14.93s seconds
2021-09-16:18:08:05,122 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:43<00:00,  7.54s/epoch, loss=0.00027, prev_loss=0.0003]
Training epochs on cuda: 100%|██████████| 10/10 [00:43<00:00,  4.31s/epoch, loss=0.00027, prev_loss=0.0003]
2021-09-16:18:08:05,122 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpfb01bqtj'
2021-09-16:18:08:05,187 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpfb01bqtj' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.51ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:18:08:19,999 INFO     [evaluator.py:657] Evaluation took 14.79s seconds
[I 2021-09-16 18:08:20,005] Trial 30 finished with value: 0.9443477494700148 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.012682029549232143, 'optimizer.lr': 0.04073053253633309, 'negative_sampler.num_negs_per_pos': 8, 'training.batch_size': 2232}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:08:20,25 WARNING  [api.py:823] No random seed is specified. Setting to 2891448783.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:04, 28.26batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:03, 32.25batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:03, 35.68batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:00<00:02, 39.22batch/s]

Training batches on cuda:  20%|█▉        | 24/123 [00:00<00:02, 40.13batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:00<00:02, 42.33batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:00<00:02, 43.94batch/s]

Training batches on cuda:  33%|███▎      | 40/123 [00:00<00:01, 45.34batch/s]

Training batches on cuda:  37%|███▋      | 46/123 [00:00<00:01, 46.85batch/s]

Training batches on cuda:  42%|████▏     | 52/123 [00:01<00:01, 47.94batch/s]

Training batches on cuda:  47%|████▋     | 58/123 [00:01<00:01, 48.64batch/s]

Training batches on cuda:  52%|█████▏    | 64/123 [00:01<00:01, 49.24batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:01<00:01, 49.36batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:01<00:00, 49.88batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:01<00:00, 43.61batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:01<00:00, 45.69batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:01<00:00, 47.15batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:02<00:00, 48.11batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:02<00:00, 47.41batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:02<00:00, 48.54batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:02<00:00, 49.19batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:02<00:00, 49.77batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000521, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:25,  2.78s/epoch, loss=0.000521, prev_loss=nan]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 35.11batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:02, 38.31batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:02, 40.98batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:00<00:02, 42.00batch/s]

Training batches on cuda:  20%|█▉        | 24/123 [00:00<00:02, 43.59batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:00<00:02, 44.78batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:00<00:01, 46.05batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:00<00:01, 46.97batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:00<00:01, 47.90batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:01<00:01, 48.03batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:01<00:01, 48.75batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:01<00:01, 49.34batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:01<00:01, 48.77batch/s]

Training batches on cuda:  59%|█████▊    | 72/123 [00:01<00:01, 48.54batch/s]

Training batches on cuda:  63%|██████▎   | 78/123 [00:01<00:00, 48.84batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:01<00:00, 45.53batch/s]

Training batches on cuda:  72%|███████▏  | 88/123 [00:01<00:00, 44.40batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:01<00:00, 44.77batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:02<00:00, 45.38batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:02<00:00, 46.42batch/s]

Training batches on cuda:  88%|████████▊ | 108/123 [00:02<00:00, 47.36batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:02<00:00, 46.31batch/s]

Training batches on cuda:  96%|█████████▌| 118/123 [00:02<00:00, 47.05batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:02<00:00, 47.31batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:25,  2.78s/epoch, loss=0.00052, prev_loss=nan] 
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:22,  2.80s/epoch, loss=0.00052, prev_loss=nan]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 36.38batch/s]

Training batches on cuda:   8%|▊         | 10/123 [00:00<00:02, 40.44batch/s]

Training batches on cuda:  13%|█▎        | 16/123 [00:00<00:02, 43.56batch/s]

Training batches on cuda:  18%|█▊        | 22/123 [00:00<00:02, 46.03batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:00<00:02, 45.86batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:00<00:02, 43.10batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:00<00:02, 42.33batch/s]

Training batches on cuda:  34%|███▍      | 42/123 [00:00<00:01, 42.59batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:01, 43.18batch/s]

Training batches on cuda:  42%|████▏     | 52/123 [00:01<00:01, 39.26batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:01<00:01, 40.64batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:01<00:01, 41.84batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:01<00:01, 42.24batch/s]

Training batches on cuda:  59%|█████▊    | 72/123 [00:01<00:01, 42.59batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:01<00:01, 42.95batch/s]

Training batches on cuda:  67%|██████▋   | 82/123 [00:01<00:00, 43.28batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:01<00:00, 42.80batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:02<00:00, 43.15batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:02<00:00, 43.03batch/s]

Training batches on cuda:  83%|████████▎ | 102/123 [00:02<00:00, 43.25batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:02<00:00, 43.29batch/s]

Training batches on cuda:  91%|█████████ | 112/123 [00:02<00:00, 43.55batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:02<00:00, 40.08batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:02<00:00, 42.99batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:22,  2.80s/epoch, loss=0.000519, prev_loss=0.00052]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:20,  2.89s/epoch, loss=0.000519, prev_loss=0.00052]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:04, 28.07batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:03, 32.84batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:00<00:02, 37.03batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:00<00:02, 41.03batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:00<00:02, 43.94batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:00<00:02, 45.16batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:00<00:01, 46.46batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:00<00:01, 47.73batch/s]

Training batches on cuda:  39%|███▉      | 48/123 [00:00<00:01, 47.79batch/s]

Training batches on cuda:  44%|████▍     | 54/123 [00:01<00:01, 49.13batch/s]

Training batches on cuda:  49%|████▉     | 60/123 [00:01<00:01, 49.81batch/s]

Training batches on cuda:  54%|█████▎    | 66/123 [00:01<00:01, 50.06batch/s]

Training batches on cuda:  59%|█████▊    | 72/123 [00:01<00:01, 49.33batch/s]

Training batches on cuda:  63%|██████▎   | 78/123 [00:01<00:00, 50.51batch/s]

Training batches on cuda:  68%|██████▊   | 84/123 [00:01<00:00, 49.28batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:01<00:00, 47.39batch/s]

Training batches on cuda:  76%|███████▋  | 94/123 [00:01<00:00, 46.63batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:02<00:00, 45.31batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:02<00:00, 44.77batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:02<00:00, 44.80batch/s]

Training batches on cuda:  93%|█████████▎| 114/123 [00:02<00:00, 44.95batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:02<00:00, 44.40batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:20,  2.89s/epoch, loss=0.000517, prev_loss=0.000519]
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:17,  2.88s/epoch, loss=0.000517, prev_loss=0.000519]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 33.29batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:07, 16.63batch/s]

Training batches on cuda:   8%|▊         | 10/123 [00:00<00:05, 20.54batch/s]

Training batches on cuda:  13%|█▎        | 16/123 [00:00<00:04, 25.16batch/s]

Training batches on cuda:  18%|█▊        | 22/123 [00:00<00:03, 29.80batch/s]

Training batches on cuda:  23%|██▎       | 28/123 [00:00<00:02, 34.18batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:00<00:02, 38.14batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:00<00:02, 37.19batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:02, 39.29batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:01<00:01, 41.81batch/s]

Training batches on cuda:  44%|████▍     | 54/123 [00:01<00:01, 43.78batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:01<00:01, 45.00batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:01<00:01, 46.50batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:01<00:01, 47.85batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:01<00:00, 48.67batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:01<00:00, 49.48batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:01<00:00, 49.80batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:02<00:00, 49.32batch/s]

Training batches on cuda:  81%|████████▏ | 100/123 [00:02<00:00, 48.99batch/s]

Training batches on cuda:  86%|████████▌ | 106/123 [00:02<00:00, 49.49batch/s]

Training batches on cuda:  91%|█████████ | 112/123 [00:02<00:00, 49.89batch/s]

Training batches on cuda:  96%|█████████▌| 118/123 [00:02<00:00, 49.25batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:17,  2.88s/epoch, loss=0.000501, prev_loss=0.000517]
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:14,  2.87s/epoch, loss=0.000501, prev_loss=0.000517]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 34.65batch/s]

Training batches on cuda:   8%|▊         | 10/123 [00:00<00:02, 38.42batch/s]

Training batches on cuda:  13%|█▎        | 16/123 [00:00<00:02, 41.51batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:00<00:02, 43.25batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:00<00:02, 45.35batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:00<00:01, 47.07batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:00<00:01, 48.43batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:00<00:01, 49.41batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:01<00:01, 49.71batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:01<00:01, 49.44batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:01<00:01, 49.85batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:01<00:01, 49.76batch/s]

Training batches on cuda:  59%|█████▊    | 72/123 [00:01<00:01, 49.50batch/s]

Training batches on cuda:  63%|██████▎   | 78/123 [00:01<00:00, 49.72batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:01<00:00, 49.52batch/s]

Training batches on cuda:  72%|███████▏  | 88/123 [00:01<00:00, 49.20batch/s]

Training batches on cuda:  76%|███████▋  | 94/123 [00:01<00:00, 49.60batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:02<00:00, 49.65batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:02<00:00, 49.72batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:02<00:00, 49.47batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:02<00:00, 49.73batch/s]

Training batches on cuda:  98%|█████████▊| 120/123 [00:02<00:00, 36.48batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:14,  2.87s/epoch, loss=0.000455, prev_loss=0.000501]
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:11,  2.85s/epoch, loss=0.000455, prev_loss=0.000501]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 32.54batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:03, 36.09batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:02, 39.22batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:00<00:02, 41.62batch/s]

Training batches on cuda:  20%|█▉        | 24/123 [00:00<00:02, 43.77batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:00<00:02, 45.42batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:00<00:01, 46.70batch/s]

Training batches on cuda:  33%|███▎      | 40/123 [00:00<00:01, 47.71batch/s]

Training batches on cuda:  37%|███▋      | 46/123 [00:00<00:01, 48.58batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:01<00:01, 48.85batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:01<00:01, 49.36batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:01<00:01, 49.71batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:01<00:01, 49.70batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:01<00:00, 50.18batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:01<00:00, 50.11batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:01<00:00, 50.13batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:01<00:00, 50.35batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:01<00:00, 50.78batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:02<00:00, 50.52batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:02<00:00, 50.87batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:02<00:00, 51.30batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:02<00:00, 51.39batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:19<00:11,  2.85s/epoch, loss=0.000427, prev_loss=0.000455]
Training epochs on cuda:  70%|███████   | 7/10 [00:19<00:08,  2.80s/epoch, loss=0.000427, prev_loss=0.000455]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 35.46batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:03, 37.23batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:02, 39.85batch/s]

Training batches on cuda:  15%|█▍        | 18/123 [00:00<00:02, 39.60batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:02, 41.73batch/s]

Training batches on cuda:  23%|██▎       | 28/123 [00:00<00:02, 43.85batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:00<00:01, 45.42batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:00<00:01, 46.57batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:00<00:01, 46.35batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:01<00:01, 45.68batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:01<00:01, 46.78batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:01<00:01, 48.00batch/s]

Training batches on cuda:  54%|█████▎    | 66/123 [00:01<00:01, 46.98batch/s]

Training batches on cuda:  59%|█████▊    | 72/123 [00:01<00:01, 47.91batch/s]

Training batches on cuda:  63%|██████▎   | 78/123 [00:01<00:00, 48.56batch/s]

Training batches on cuda:  68%|██████▊   | 84/123 [00:01<00:00, 49.52batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:02<00:00, 49.63batch/s]

Training batches on cuda:  76%|███████▋  | 94/123 [00:02<00:01, 23.17batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:02<00:00, 27.28batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:02<00:00, 31.36batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:02<00:00, 35.47batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:02<00:00, 38.01batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:02<00:00, 41.87batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:23<00:08,  2.80s/epoch, loss=0.000338, prev_loss=0.000427]
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:05,  2.91s/epoch, loss=0.000338, prev_loss=0.000427]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 33.62batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:03, 36.29batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:02, 39.02batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:00<00:02, 40.73batch/s]

Training batches on cuda:  20%|█▉        | 24/123 [00:00<00:02, 43.08batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:00<00:02, 44.69batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:00<00:02, 43.30batch/s]

Training batches on cuda:  33%|███▎      | 40/123 [00:00<00:01, 43.54batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:01<00:01, 44.88batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:01<00:01, 44.74batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:01<00:01, 45.69batch/s]

Training batches on cuda:  49%|████▉     | 60/123 [00:01<00:01, 46.32batch/s]

Training batches on cuda:  54%|█████▎    | 66/123 [00:01<00:01, 47.54batch/s]

Training batches on cuda:  59%|█████▊    | 72/123 [00:01<00:01, 48.45batch/s]

Training batches on cuda:  63%|██████▎   | 78/123 [00:01<00:00, 49.19batch/s]

Training batches on cuda:  68%|██████▊   | 84/123 [00:01<00:00, 49.58batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:01<00:00, 47.33batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:02<00:00, 48.08batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:02<00:00, 48.70batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:02<00:00, 49.45batch/s]

Training batches on cuda:  91%|█████████ | 112/123 [00:02<00:00, 48.61batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:02<00:00, 48.11batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:02<00:00, 46.74batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:05,  2.91s/epoch, loss=0.000314, prev_loss=0.000338]
Training epochs on cuda:  90%|█████████ | 9/10 [00:26<00:02,  2.91s/epoch, loss=0.000314, prev_loss=0.000338]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:04, 26.22batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:03, 29.91batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:03, 33.83batch/s]

Training batches on cuda:  15%|█▍        | 18/123 [00:00<00:02, 37.39batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:02, 40.19batch/s]

Training batches on cuda:  23%|██▎       | 28/123 [00:00<00:02, 41.46batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:00<00:02, 41.41batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:00<00:01, 43.31batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:00<00:01, 44.28batch/s]

Training batches on cuda:  39%|███▉      | 48/123 [00:01<00:01, 45.27batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:01<00:01, 45.23batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:01<00:01, 46.38batch/s]

Training batches on cuda:  52%|█████▏    | 64/123 [00:01<00:01, 46.11batch/s]

Training batches on cuda:  57%|█████▋    | 70/123 [00:01<00:01, 47.38batch/s]

Training batches on cuda:  62%|██████▏   | 76/123 [00:01<00:00, 48.43batch/s]

Training batches on cuda:  67%|██████▋   | 82/123 [00:01<00:00, 48.99batch/s]

Training batches on cuda:  72%|███████▏  | 88/123 [00:01<00:00, 49.56batch/s]

Training batches on cuda:  76%|███████▋  | 94/123 [00:02<00:00, 50.16batch/s]

Training batches on cuda:  81%|████████▏ | 100/123 [00:02<00:00, 50.12batch/s]

Training batches on cuda:  86%|████████▌ | 106/123 [00:02<00:00, 50.20batch/s]

Training batches on cuda:  91%|█████████ | 112/123 [00:02<00:00, 50.11batch/s]

Training batches on cuda:  96%|█████████▌| 118/123 [00:02<00:00, 50.61batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:28<00:02,  2.91s/epoch, loss=0.000312, prev_loss=0.000314]2021-09-16:18:08:49,256 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:08:50,275 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:09:05,884 INFO     [evaluator.py:657] Evaluation took 15.61s seconds
2021-09-16:18:09:06,231 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  7.97s/epoch, loss=0.000312, prev_loss=0.000314]
Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  4.58s/epoch, loss=0.000312, prev_loss=0.000314]
2021-09-16:18:09:06,232 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpjvpd8fg7'
2021-09-16:18:09:06,310 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpjvpd8fg7' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▉         | 3.58k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:10, 2.08ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:09, 2.23ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:08, 2.34ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:08, 2.43ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.49ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.53ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.57ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:07, 2.59ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:09:21,298 INFO     [evaluator.py:657] Evaluation took 14.97s seconds
[I 2021-09-16 18:09:21,303] Trial 31 finished with value: 0.9301303245117661 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.025826301758320118, 'optimizer.lr': 0.042593658943008254, 'negative_sampler.num_negs_per_pos': 8, 'training.batch_size': 2532}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:09:21,371 WARNING  [api.py:823] No random seed is specified. Setting to 656976402.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:02, 47.40batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:00<00:02, 53.33batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:00<00:01, 57.59batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:00<00:01, 61.40batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:00<00:01, 63.99batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:00<00:01, 66.44batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:00<00:01, 67.90batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:00<00:01, 69.17batch/s]

Training batches on cuda:  51%|█████     | 68/134 [00:01<00:01, 51.65batch/s]

Training batches on cuda:  57%|█████▋    | 76/134 [00:01<00:01, 56.94batch/s]

Training batches on cuda:  63%|██████▎   | 84/134 [00:01<00:00, 60.79batch/s]

Training batches on cuda:  69%|██████▊   | 92/134 [00:01<00:00, 64.22batch/s]

Training batches on cuda:  75%|███████▍  | 100/134 [00:01<00:00, 66.11batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:01<00:00, 64.88batch/s]

Training batches on cuda:  85%|████████▌ | 114/134 [00:01<00:00, 64.41batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:01<00:00, 64.90batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:01<00:00, 67.19batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000567, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:20,  2.23s/epoch, loss=0.000567, prev_loss=nan]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/134 [00:00<00:03, 38.43batch/s]

Training batches on cuda:   9%|▉         | 12/134 [00:00<00:02, 44.57batch/s]

Training batches on cuda:  15%|█▍        | 20/134 [00:00<00:02, 50.30batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:00<00:01, 54.87batch/s]

Training batches on cuda:  25%|██▌       | 34/134 [00:00<00:01, 58.18batch/s]

Training batches on cuda:  31%|███▏      | 42/134 [00:00<00:01, 61.35batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:00<00:01, 63.44batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:00<00:01, 65.57batch/s]

Training batches on cuda:  48%|████▊     | 64/134 [00:00<00:01, 66.04batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:01<00:00, 67.06batch/s]

Training batches on cuda:  58%|█████▊    | 78/134 [00:01<00:00, 67.90batch/s]

Training batches on cuda:  64%|██████▍   | 86/134 [00:01<00:00, 69.17batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:01<00:00, 70.46batch/s]

Training batches on cuda:  76%|███████▌  | 102/134 [00:01<00:00, 64.16batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:01<00:00, 59.09batch/s]

Training batches on cuda:  87%|████████▋ | 116/134 [00:01<00:00, 57.49batch/s]

Training batches on cuda:  91%|█████████ | 122/134 [00:01<00:00, 57.32batch/s]

Training batches on cuda:  96%|█████████▌| 128/134 [00:02<00:00, 57.78batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:20,  2.23s/epoch, loss=0.000562, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:17,  2.24s/epoch, loss=0.000562, prev_loss=nan]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:02, 53.53batch/s]

Training batches on cuda:  10%|█         | 14/134 [00:00<00:02, 57.50batch/s]

Training batches on cuda:  15%|█▍        | 20/134 [00:00<00:01, 58.16batch/s]

Training batches on cuda:  19%|█▉        | 26/134 [00:00<00:01, 58.18batch/s]

Training batches on cuda:  24%|██▍       | 32/134 [00:00<00:01, 58.15batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:00<00:01, 60.95batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:00<00:01, 64.78batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:00<00:01, 67.47batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:00<00:01, 69.63batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:01<00:00, 71.05batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:01<00:00, 71.76batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:01<00:00, 70.06batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:01<00:00, 69.06batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:01<00:00, 51.75batch/s]

Training batches on cuda:  81%|████████  | 108/134 [00:01<00:00, 54.73batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:01<00:00, 57.69batch/s]

Training batches on cuda:  91%|█████████ | 122/134 [00:01<00:00, 60.74batch/s]

Training batches on cuda:  97%|█████████▋| 130/134 [00:02<00:00, 63.81batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:17,  2.24s/epoch, loss=0.000525, prev_loss=0.000562]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:15,  2.27s/epoch, loss=0.000525, prev_loss=0.000562]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:02, 49.53batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:00<00:02, 55.00batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:00<00:01, 59.44batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:00<00:01, 63.33batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:00<00:01, 66.66batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:00<00:01, 69.46batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:00<00:01, 71.91batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:00<00:00, 73.47batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:00<00:00, 74.76batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:01<00:00, 74.98batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:01<00:00, 75.60batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:01<00:00, 75.76batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:01<00:00, 75.59batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:01<00:00, 75.12batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:01<00:00, 75.72batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:01<00:00, 75.67batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:01<00:00, 75.48batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:15,  2.27s/epoch, loss=0.000457, prev_loss=0.000525]
Training epochs on cuda:  40%|████      | 4/10 [00:08<00:13,  2.18s/epoch, loss=0.000457, prev_loss=0.000525]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:02, 55.16batch/s]

Training batches on cuda:  10%|█         | 14/134 [00:00<00:01, 60.45batch/s]

Training batches on cuda:  16%|█▋        | 22/134 [00:00<00:01, 64.59batch/s]

Training batches on cuda:  22%|██▏       | 30/134 [00:00<00:01, 67.51batch/s]

Training batches on cuda:  28%|██▊       | 38/134 [00:00<00:01, 70.31batch/s]

Training batches on cuda:  34%|███▍      | 46/134 [00:00<00:01, 72.02batch/s]

Training batches on cuda:  40%|████      | 54/134 [00:00<00:01, 73.13batch/s]

Training batches on cuda:  46%|████▋     | 62/134 [00:00<00:00, 73.35batch/s]

Training batches on cuda:  52%|█████▏    | 70/134 [00:00<00:00, 74.34batch/s]

Training batches on cuda:  58%|█████▊    | 78/134 [00:01<00:00, 74.52batch/s]

Training batches on cuda:  64%|██████▍   | 86/134 [00:01<00:00, 74.49batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:01<00:00, 75.25batch/s]

Training batches on cuda:  76%|███████▌  | 102/134 [00:01<00:00, 75.18batch/s]

Training batches on cuda:  82%|████████▏ | 110/134 [00:01<00:00, 75.66batch/s]

Training batches on cuda:  88%|████████▊ | 118/134 [00:01<00:00, 76.78batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:01<00:00, 77.72batch/s]

Training batches on cuda: 100%|██████████| 134/134 [00:01<00:00, 77.84batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:10<00:13,  2.18s/epoch, loss=0.000392, prev_loss=0.000457]
Training epochs on cuda:  50%|█████     | 5/10 [00:10<00:10,  2.12s/epoch, loss=0.000392, prev_loss=0.000457]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:02, 46.12batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:05, 21.43batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:00<00:04, 27.31batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:00<00:03, 33.83batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:00<00:02, 40.68batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:00<00:01, 47.50batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:00<00:01, 53.26batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:00<00:01, 58.28batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:01<00:01, 63.16batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:01<00:00, 66.82batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:01<00:00, 69.20batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:01<00:00, 71.39batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:01<00:00, 73.07batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:01<00:00, 74.24batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:01<00:00, 74.87batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:01<00:00, 75.42batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:01<00:00, 75.41batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:12<00:10,  2.12s/epoch, loss=0.000328, prev_loss=0.000392]
Training epochs on cuda:  60%|██████    | 6/10 [00:12<00:08,  2.13s/epoch, loss=0.000328, prev_loss=0.000392]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:02, 58.28batch/s]

Training batches on cuda:  10%|█         | 14/134 [00:00<00:01, 62.94batch/s]

Training batches on cuda:  16%|█▋        | 22/134 [00:00<00:01, 66.51batch/s]

Training batches on cuda:  22%|██▏       | 30/134 [00:00<00:01, 69.57batch/s]

Training batches on cuda:  28%|██▊       | 38/134 [00:00<00:01, 71.10batch/s]

Training batches on cuda:  34%|███▍      | 46/134 [00:00<00:01, 72.40batch/s]

Training batches on cuda:  40%|████      | 54/134 [00:00<00:01, 73.33batch/s]

Training batches on cuda:  46%|████▋     | 62/134 [00:00<00:00, 74.12batch/s]

Training batches on cuda:  52%|█████▏    | 70/134 [00:00<00:00, 75.74batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:01<00:00, 77.01batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:01<00:00, 76.57batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:01<00:00, 75.60batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:01<00:00, 76.05batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:01<00:00, 76.55batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:01<00:00, 76.90batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:01<00:00, 77.25batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:14<00:08,  2.13s/epoch, loss=0.000289, prev_loss=0.000328]
Training epochs on cuda:  70%|███████   | 7/10 [00:14<00:06,  2.07s/epoch, loss=0.000289, prev_loss=0.000328]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:02, 44.23batch/s]

Training batches on cuda:   9%|▉         | 12/134 [00:00<00:02, 49.53batch/s]

Training batches on cuda:  15%|█▍        | 20/134 [00:00<00:02, 54.47batch/s]

Training batches on cuda:  19%|█▉        | 26/134 [00:00<00:01, 54.82batch/s]

Training batches on cuda:  24%|██▍       | 32/134 [00:00<00:01, 54.55batch/s]

Training batches on cuda:  28%|██▊       | 38/134 [00:00<00:01, 55.67batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:00<00:01, 57.65batch/s]

Training batches on cuda:  39%|███▉      | 52/134 [00:00<00:01, 58.85batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:00<00:01, 60.24batch/s]

Training batches on cuda:  49%|████▉     | 66/134 [00:01<00:01, 61.68batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:01<00:01, 39.21batch/s]

Training batches on cuda:  60%|█████▉    | 80/134 [00:01<00:01, 44.45batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:01<00:00, 48.96batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:01<00:00, 53.07batch/s]

Training batches on cuda:  76%|███████▌  | 102/134 [00:01<00:00, 58.08batch/s]

Training batches on cuda:  82%|████████▏ | 110/134 [00:01<00:00, 62.58batch/s]

Training batches on cuda:  88%|████████▊ | 118/134 [00:02<00:00, 66.39batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:02<00:00, 68.50batch/s]

Training batches on cuda: 100%|██████████| 134/134 [00:02<00:00, 70.68batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:06,  2.07s/epoch, loss=0.000263, prev_loss=0.000289]
Training epochs on cuda:  80%|████████  | 8/10 [00:17<00:04,  2.19s/epoch, loss=0.000263, prev_loss=0.000289]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:02, 55.25batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:00<00:02, 57.42batch/s]

Training batches on cuda:  15%|█▍        | 20/134 [00:00<00:01, 59.66batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:00<00:01, 61.37batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:00<00:01, 64.19batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:00<00:01, 67.45batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:00<00:01, 69.13batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:00<00:01, 70.05batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:00<00:00, 71.69batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:01<00:00, 73.05batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:01<00:00, 73.99batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:01<00:00, 74.00batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:01<00:00, 73.34batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:01<00:00, 72.68batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:01<00:00, 72.22batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:01<00:00, 73.00batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:01<00:00, 73.11batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:19<00:04,  2.19s/epoch, loss=0.000244, prev_loss=0.000263]
Training epochs on cuda:  90%|█████████ | 9/10 [00:19<00:02,  2.16s/epoch, loss=0.000244, prev_loss=0.000263]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/134 [00:00<00:03, 36.32batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:00<00:02, 41.25batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:00<00:02, 43.97batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:00<00:02, 47.37batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:00<00:02, 49.70batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:00<00:01, 51.39batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:00<00:01, 53.10batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:00<00:01, 54.30batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:00<00:01, 55.69batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:01<00:01, 55.76batch/s]

Training batches on cuda:  49%|████▉     | 66/134 [00:01<00:01, 59.22batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:01<00:01, 59.87batch/s]

Training batches on cuda:  60%|█████▉    | 80/134 [00:01<00:00, 60.71batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:01<00:00, 62.06batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:01<00:00, 63.24batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:01<00:00, 43.49batch/s]

Training batches on cuda:  81%|████████  | 108/134 [00:01<00:00, 48.79batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:02<00:00, 53.15batch/s]

Training batches on cuda:  91%|█████████ | 122/134 [00:02<00:00, 56.71batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:02<00:00, 60.08batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:22<00:02,  2.16s/epoch, loss=0.000244, prev_loss=0.000244]2021-09-16:18:09:43,811 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:09:46,507 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:10:01,489 INFO     [evaluator.py:657] Evaluation took 14.98s seconds
2021-09-16:18:10:01,611 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  7.62s/epoch, loss=0.000244, prev_loss=0.000244]
Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  3.98s/epoch, loss=0.000244, prev_loss=0.000244]
2021-09-16:18:10:01,612 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp3cfi4zk1'
2021-09-16:18:10:01,686 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3cfi4zk1' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.52ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:16, 2.07ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:15, 2.21ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:02<00:14, 2.33ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:13, 2.41ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:13, 2.48ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.53ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.56ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.60ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:10:16,587 INFO     [evaluator.py:657] Evaluation took 14.89s seconds
[I 2021-09-16 18:10:16,592] Trial 32 finished with value: 0.9452789663374941 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.011257074942701784, 'optimizer.lr': 0.040989961651008024, 'negative_sampler.num_negs_per_pos': 4, 'training.batch_size': 2332}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:10:16,613 WARNING  [api.py:823] No random seed is specified. Setting to 1204193666.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:02, 56.88batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:02, 61.86batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:01, 66.09batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:01, 70.10batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:00<00:01, 73.39batch/s]

Training batches on cuda:  35%|███▌      | 49/140 [00:00<00:01, 75.76batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:00<00:01, 77.70batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:00<00:00, 78.64batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:00<00:00, 78.50batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:01<00:00, 78.36batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:01<00:00, 78.96batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:01<00:00, 78.58batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:01<00:00, 78.36batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:01<00:00, 62.85batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:01<00:00, 67.02batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:01<00:00, 69.44batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000601, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:18,  2.06s/epoch, loss=0.000601, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:02, 56.01batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:02, 60.94batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:01, 63.65batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:01, 67.41batch/s]

Training batches on cuda:  26%|██▋       | 37/140 [00:00<00:01, 66.51batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:00<00:01, 70.09batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:00<00:01, 71.87batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:00<00:01, 73.15batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:00<00:00, 74.38batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:01<00:00, 75.90batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:01<00:00, 77.07batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:01<00:00, 76.91batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:01<00:00, 77.57batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:01<00:00, 77.61batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:01<00:00, 77.48batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:01<00:00, 76.32batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:01<00:00, 74.56batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:18,  2.06s/epoch, loss=0.000593, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:16,  2.05s/epoch, loss=0.000593, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:02, 53.78batch/s]

Training batches on cuda:   9%|▉         | 13/140 [00:00<00:02, 56.72batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:02, 58.35batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:01, 61.88batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:00<00:01, 64.69batch/s]

Training batches on cuda:  31%|███▏      | 44/140 [00:00<00:01, 67.56batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:00<00:01, 69.77batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:00<00:01, 71.27batch/s]

Training batches on cuda:  49%|████▊     | 68/140 [00:00<00:00, 72.20batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:00, 71.05batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:01<00:00, 73.20batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:01<00:00, 73.19batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:01<00:00, 74.97batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:01<00:00, 75.89batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:01<00:00, 76.52batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:01<00:00, 76.91batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:01<00:00, 77.04batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:16,  2.05s/epoch, loss=0.000593, prev_loss=0.000593]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:14,  2.07s/epoch, loss=0.000593, prev_loss=0.000593]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:02, 53.32batch/s]

Training batches on cuda:   6%|▌         | 8/140 [00:00<00:06, 21.90batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:04, 27.69batch/s]

Training batches on cuda:  17%|█▋        | 24/140 [00:00<00:03, 34.10batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:02, 40.79batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:00<00:02, 47.12batch/s]

Training batches on cuda:  34%|███▍      | 48/140 [00:00<00:01, 52.71batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:00<00:01, 57.60batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:01, 61.88batch/s]

Training batches on cuda:  51%|█████▏    | 72/140 [00:01<00:01, 65.00batch/s]

Training batches on cuda:  57%|█████▋    | 80/140 [00:01<00:00, 67.13batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:01<00:00, 68.77batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:01<00:00, 70.80batch/s]

Training batches on cuda:  74%|███████▍  | 104/140 [00:01<00:00, 72.33batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:01<00:00, 72.69batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:01<00:00, 73.43batch/s]

Training batches on cuda:  91%|█████████▏| 128/140 [00:01<00:00, 73.72batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:02<00:00, 74.55batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:14,  2.07s/epoch, loss=0.000593, prev_loss=0.000593]
Training epochs on cuda:  40%|████      | 4/10 [00:08<00:12,  2.14s/epoch, loss=0.000593, prev_loss=0.000593]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:04, 29.67batch/s]

Training batches on cuda:   6%|▋         | 9/140 [00:00<00:03, 34.48batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:03, 39.40batch/s]

Training batches on cuda:  15%|█▌        | 21/140 [00:00<00:02, 43.81batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:02, 48.81batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:00<00:01, 53.59batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:00<00:01, 56.85batch/s]

Training batches on cuda:  35%|███▌      | 49/140 [00:00<00:01, 59.69batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:00<00:01, 60.55batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:01, 59.59batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 58.60batch/s]

Training batches on cuda:  55%|█████▌    | 77/140 [00:01<00:01, 60.16batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:01<00:00, 61.43batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:01<00:00, 62.36batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:01<00:00, 63.09batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:01<00:00, 64.27batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:01<00:00, 64.55batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:01<00:00, 65.19batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:02<00:00, 62.69batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:02<00:00, 62.69batch/s]

Training batches on cuda: 100%|██████████| 140/140 [00:02<00:00, 64.14batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:12,  2.14s/epoch, loss=0.000593, prev_loss=0.000593]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.25s/epoch, loss=0.000593, prev_loss=0.000593]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 5/140 [00:00<00:03, 44.96batch/s]

Training batches on cuda:   9%|▉         | 13/140 [00:00<00:02, 51.52batch/s]

Training batches on cuda:  15%|█▌        | 21/140 [00:00<00:02, 57.36batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:02, 38.76batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:02, 45.69batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:00<00:01, 52.26batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:00<00:01, 58.13batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:00<00:01, 62.93batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:01, 66.84batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:01<00:00, 69.72batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:01<00:00, 71.95batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:01<00:00, 74.13batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:01<00:00, 75.49batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:01<00:00, 76.35batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:01<00:00, 77.08batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:01<00:00, 77.65batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:01<00:00, 77.19batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:01<00:00, 77.93batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:11,  2.25s/epoch, loss=0.000593, prev_loss=0.000593]
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:08,  2.23s/epoch, loss=0.000593, prev_loss=0.000593]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:02, 58.30batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:02, 62.77batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:01, 66.74batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:01, 69.63batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:00<00:01, 70.37batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:00<00:01, 72.54batch/s]

Training batches on cuda:  38%|███▊      | 53/140 [00:00<00:01, 70.04batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:00<00:01, 71.58batch/s]

Training batches on cuda:  49%|████▉     | 69/140 [00:00<00:00, 73.31batch/s]

Training batches on cuda:  55%|█████▌    | 77/140 [00:01<00:00, 74.39batch/s]

Training batches on cuda:  61%|██████    | 85/140 [00:01<00:00, 75.69batch/s]

Training batches on cuda:  66%|██████▋   | 93/140 [00:01<00:00, 76.54batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:01<00:00, 74.44batch/s]

Training batches on cuda:  78%|███████▊  | 109/140 [00:01<00:00, 75.16batch/s]

Training batches on cuda:  84%|████████▎ | 117/140 [00:01<00:00, 76.43batch/s]

Training batches on cuda:  89%|████████▉ | 125/140 [00:01<00:00, 77.06batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:01<00:00, 76.78batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:08,  2.23s/epoch, loss=0.000593, prev_loss=0.000593]
Training epochs on cuda:  70%|███████   | 7/10 [00:15<00:06,  2.18s/epoch, loss=0.000593, prev_loss=0.000593]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:02, 52.09batch/s]

Training batches on cuda:   9%|▉         | 13/140 [00:00<00:02, 55.01batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:02, 58.14batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:01, 60.47batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:01, 61.09batch/s]

Training batches on cuda:  29%|██▉       | 41/140 [00:00<00:01, 61.45batch/s]

Training batches on cuda:  34%|███▍      | 48/140 [00:00<00:01, 63.24batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:00<00:01, 64.21batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:00<00:01, 66.84batch/s]

Training batches on cuda:  51%|█████▏    | 72/140 [00:01<00:00, 70.42batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:01<00:00, 63.50batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:01<00:00, 67.06batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:01<00:00, 70.44batch/s]

Training batches on cuda:  74%|███████▍  | 104/140 [00:01<00:00, 72.66batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:01<00:00, 74.35batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:01<00:00, 74.44batch/s]

Training batches on cuda:  91%|█████████▏| 128/140 [00:01<00:00, 75.20batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:01<00:00, 76.74batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:06,  2.18s/epoch, loss=0.000593, prev_loss=0.000593]
Training epochs on cuda:  80%|████████  | 8/10 [00:17<00:04,  2.18s/epoch, loss=0.000593, prev_loss=0.000593]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:02, 62.58batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:01, 67.54batch/s]

Training batches on cuda:  18%|█▊        | 25/140 [00:00<00:01, 71.24batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:01, 74.31batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:00<00:01, 76.33batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:00<00:01, 77.56batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:00<00:01, 77.80batch/s]

Training batches on cuda:  49%|████▊     | 68/140 [00:00<00:00, 77.11batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:00<00:00, 74.98batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:01<00:00, 71.79batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:01<00:00, 72.11batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:01<00:00, 73.47batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:01<00:00, 73.70batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:01<00:00, 73.55batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:01<00:00, 74.09batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:01<00:00, 73.89batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:19<00:04,  2.18s/epoch, loss=0.000593, prev_loss=0.000593]
Training epochs on cuda:  90%|█████████ | 9/10 [00:19<00:02,  2.14s/epoch, loss=0.000593, prev_loss=0.000593]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 37.78batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:02, 43.49batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:02, 48.26batch/s]

Training batches on cuda:  17%|█▋        | 24/140 [00:00<00:02, 50.44batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:01, 55.36batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:00<00:01, 60.76batch/s]

Training batches on cuda:  34%|███▍      | 48/140 [00:00<00:01, 64.71batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:00<00:01, 68.30batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:00<00:01, 70.88batch/s]

Training batches on cuda:  51%|█████▏    | 72/140 [00:01<00:00, 72.57batch/s]

Training batches on cuda:  57%|█████▋    | 80/140 [00:01<00:00, 74.17batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:01<00:00, 74.56batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:01<00:00, 75.03batch/s]

Training batches on cuda:  74%|███████▍  | 104/140 [00:01<00:00, 75.70batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:01<00:00, 75.71batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:01<00:00, 75.92batch/s]

Training batches on cuda:  91%|█████████▏| 128/140 [00:01<00:00, 76.08batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:01<00:00, 77.15batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:21<00:02,  2.14s/epoch, loss=0.000593, prev_loss=0.000593]2021-09-16:18:10:38,756 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:10:39,807 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:10:54,799 INFO     [evaluator.py:657] Evaluation took 14.99s seconds
2021-09-16:18:10:54,949 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:37<00:00,  7.03s/epoch, loss=0.000593, prev_loss=0.000593]
Training epochs on cuda: 100%|██████████| 10/10 [00:37<00:00,  3.79s/epoch, loss=0.000593, prev_loss=0.000593]
2021-09-16:18:10:54,950 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp6z6v7rzw'
2021-09-16:18:10:55,5 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6z6v7rzw' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.52ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.61ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:11:09,843 INFO     [evaluator.py:657] Evaluation took 14.82s seconds
[I 2021-09-16 18:11:09,848] Trial 33 finished with value: 0.006344909927362807 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.1521522566515936, 'optimizer.lr': 0.018936111431911762, 'negative_sampler.num_negs_per_pos': 3, 'training.batch_size': 2232}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:11:09,871 WARNING  [api.py:823] No random seed is specified. Setting to 1513761445.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:02, 43.49batch/s]

Training batches on cuda:  10%|▉         | 12/123 [00:00<00:02, 48.73batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:01, 53.71batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:00<00:01, 57.59batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:00<00:01, 60.73batch/s]

Training batches on cuda:  34%|███▍      | 42/123 [00:00<00:01, 63.42batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:00<00:01, 65.95batch/s]

Training batches on cuda:  47%|████▋     | 58/123 [00:00<00:00, 67.94batch/s]

Training batches on cuda:  54%|█████▎    | 66/123 [00:00<00:00, 68.96batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:01<00:00, 69.19batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:01<00:00, 69.07batch/s]

Training batches on cuda:  72%|███████▏  | 88/123 [00:01<00:00, 69.18batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:01<00:00, 69.19batch/s]

Training batches on cuda:  83%|████████▎ | 102/123 [00:01<00:00, 68.56batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:01<00:00, 69.34batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:01<00:00, 65.94batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000521, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:18,  2.04s/epoch, loss=0.000521, prev_loss=nan]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:02, 45.01batch/s]

Training batches on cuda:  10%|▉         | 12/123 [00:00<00:02, 48.67batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:00<00:01, 52.47batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:00<00:01, 56.71batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:00<00:01, 58.91batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:00<00:01, 60.82batch/s]

Training batches on cuda:  39%|███▉      | 48/123 [00:00<00:01, 62.43batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:00<00:01, 64.07batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:00<00:00, 64.41batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:01<00:00, 65.34batch/s]

Training batches on cuda:  62%|██████▏   | 76/123 [00:01<00:00, 64.74batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:01<00:00, 64.74batch/s]

Training batches on cuda:  73%|███████▎  | 90/123 [00:01<00:00, 64.92batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:01<00:00, 57.00batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:01<00:00, 59.75batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:01<00:00, 61.57batch/s]

Training batches on cuda:  96%|█████████▌| 118/123 [00:01<00:00, 62.75batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:18,  2.04s/epoch, loss=0.000519, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:16,  2.08s/epoch, loss=0.000519, prev_loss=nan]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:06, 19.03batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:04, 23.50batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:03, 27.66batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:03, 33.06batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:00<00:02, 38.31batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:00<00:02, 42.83batch/s]

Training batches on cuda:  33%|███▎      | 40/123 [00:00<00:01, 48.15batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:00<00:01, 52.59batch/s]

Training batches on cuda:  44%|████▍     | 54/123 [00:00<00:01, 55.94batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:01<00:01, 58.64batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:01<00:00, 58.35batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:01<00:00, 58.98batch/s]

Training batches on cuda:  67%|██████▋   | 82/123 [00:01<00:00, 55.11batch/s]

Training batches on cuda:  72%|███████▏  | 88/123 [00:01<00:00, 53.59batch/s]

Training batches on cuda:  76%|███████▋  | 94/123 [00:01<00:00, 55.30batch/s]

Training batches on cuda:  81%|████████▏ | 100/123 [00:01<00:00, 56.26batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:01<00:00, 57.55batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:02<00:00, 57.54batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:02<00:00, 56.89batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:16,  2.08s/epoch, loss=0.000517, prev_loss=0.000519]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:15,  2.19s/epoch, loss=0.000517, prev_loss=0.000519]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:04, 29.97batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:03, 33.31batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:02, 36.70batch/s]

Training batches on cuda:  15%|█▍        | 18/123 [00:00<00:02, 37.84batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:02, 40.07batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:00<00:02, 42.96batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:00<00:02, 44.30batch/s]

Training batches on cuda:  33%|███▎      | 40/123 [00:00<00:01, 48.03batch/s]

Training batches on cuda:  37%|███▋      | 46/123 [00:00<00:01, 51.04batch/s]

Training batches on cuda:  42%|████▏     | 52/123 [00:01<00:01, 49.05batch/s]

Training batches on cuda:  47%|████▋     | 58/123 [00:01<00:01, 51.16batch/s]

Training batches on cuda:  52%|█████▏    | 64/123 [00:01<00:01, 52.33batch/s]

Training batches on cuda:  57%|█████▋    | 70/123 [00:01<00:00, 53.60batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:01<00:00, 55.52batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:01<00:00, 55.32batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:01<00:00, 56.01batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:01<00:00, 56.81batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:01<00:00, 57.44batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:02<00:00, 57.11batch/s]

Training batches on cuda:  93%|█████████▎| 114/123 [00:02<00:00, 58.37batch/s]

Training batches on cuda:  98%|█████████▊| 120/123 [00:02<00:00, 58.78batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:15,  2.19s/epoch, loss=0.000512, prev_loss=0.000517]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:13,  2.30s/epoch, loss=0.000512, prev_loss=0.000517]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 35.98batch/s]

Training batches on cuda:   8%|▊         | 10/123 [00:00<00:02, 40.58batch/s]

Training batches on cuda:  13%|█▎        | 16/123 [00:00<00:02, 44.21batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:02, 48.35batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:00<00:01, 51.26batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:00<00:01, 50.14batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:00<00:01, 54.05batch/s]

Training batches on cuda:  39%|███▉      | 48/123 [00:00<00:01, 56.52batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:00<00:01, 59.61batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:01<00:01, 58.19batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:01<00:00, 58.00batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:01<00:00, 57.77batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:01<00:00, 57.75batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:01<00:00, 59.84batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:01<00:00, 61.70batch/s]

Training batches on cuda:  81%|████████▏ | 100/123 [00:01<00:00, 62.83batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:01<00:00, 64.30batch/s]

Training batches on cuda:  93%|█████████▎| 114/123 [00:01<00:00, 65.36batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:02<00:00, 65.32batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:13,  2.30s/epoch, loss=0.000497, prev_loss=0.000512]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.28s/epoch, loss=0.000497, prev_loss=0.000512]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:03, 35.21batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:02, 40.54batch/s]

Training batches on cuda:  15%|█▍        | 18/123 [00:00<00:02, 45.08batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:00<00:02, 48.86batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:00<00:01, 51.96batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:00<00:01, 55.57batch/s]

Training batches on cuda:  37%|███▋      | 46/123 [00:00<00:01, 58.13batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:00<00:01, 60.95batch/s]

Training batches on cuda:  49%|████▉     | 60/123 [00:00<00:01, 61.53batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:01<00:00, 62.88batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:01<00:00, 65.63batch/s]

Training batches on cuda:  67%|██████▋   | 82/123 [00:01<00:00, 66.35batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:01<00:00, 66.11batch/s]

Training batches on cuda:  78%|███████▊  | 96/123 [00:01<00:00, 66.83batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:01<00:00, 67.34batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:01<00:00, 60.96batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:01<00:00, 62.59batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:11,  2.28s/epoch, loss=0.000487, prev_loss=0.000497]
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:09,  2.26s/epoch, loss=0.000487, prev_loss=0.000497]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:06, 19.00batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:04, 23.81batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:00<00:03, 29.57batch/s]

Training batches on cuda:  18%|█▊        | 22/123 [00:00<00:02, 35.40batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:00<00:02, 40.48batch/s]

Training batches on cuda:  29%|██▉       | 36/123 [00:00<00:01, 45.26batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:00<00:01, 50.64batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:00<00:01, 53.03batch/s]

Training batches on cuda:  47%|████▋     | 58/123 [00:00<00:01, 55.54batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:01<00:01, 57.99batch/s]

Training batches on cuda:  59%|█████▊    | 72/123 [00:01<00:00, 60.03batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:01<00:00, 61.89batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:01<00:00, 62.58batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:01<00:00, 62.88batch/s]

Training batches on cuda:  81%|████████▏ | 100/123 [00:01<00:00, 61.20batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:01<00:00, 60.18batch/s]

Training batches on cuda:  93%|█████████▎| 114/123 [00:01<00:00, 60.24batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:02<00:00, 60.52batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:09,  2.26s/epoch, loss=0.000478, prev_loss=0.000487]
Training epochs on cuda:  70%|███████   | 7/10 [00:15<00:06,  2.25s/epoch, loss=0.000478, prev_loss=0.000487]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▍         | 6/123 [00:00<00:02, 55.36batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:01, 59.73batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:00<00:01, 62.20batch/s]

Training batches on cuda:  23%|██▎       | 28/123 [00:00<00:01, 62.86batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:00<00:01, 64.41batch/s]

Training batches on cuda:  34%|███▍      | 42/123 [00:00<00:01, 65.33batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:00<00:01, 66.40batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:00<00:00, 67.05batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:00<00:00, 66.93batch/s]

Training batches on cuda:  57%|█████▋    | 70/123 [00:01<00:00, 67.25batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:01<00:00, 67.70batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:01<00:00, 68.57batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:01<00:00, 67.88batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:01<00:00, 68.19batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:01<00:00, 68.83batch/s]

Training batches on cuda:  93%|█████████▎| 114/123 [00:01<00:00, 68.94batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:01<00:00, 69.09batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:06,  2.25s/epoch, loss=0.000467, prev_loss=0.000478]
Training epochs on cuda:  80%|████████  | 8/10 [00:17<00:04,  2.18s/epoch, loss=0.000467, prev_loss=0.000478]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▍         | 6/123 [00:00<00:02, 53.62batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:01, 57.47batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:00<00:01, 60.70batch/s]

Training batches on cuda:  23%|██▎       | 28/123 [00:00<00:01, 62.97batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:00<00:01, 64.40batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:00<00:01, 66.37batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:00<00:01, 62.50batch/s]

Training batches on cuda:  47%|████▋     | 58/123 [00:00<00:01, 64.69batch/s]

Training batches on cuda:  54%|█████▎    | 66/123 [00:00<00:00, 67.03batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:01<00:00, 68.21batch/s]

Training batches on cuda:  67%|██████▋   | 82/123 [00:01<00:00, 69.28batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:01<00:00, 69.27batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:01<00:00, 69.76batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:01<00:00, 70.65batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:01<00:00, 70.72batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:01<00:00, 70.88batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:19<00:04,  2.18s/epoch, loss=0.000469, prev_loss=0.000467]
Training epochs on cuda:  90%|█████████ | 9/10 [00:19<00:02,  2.12s/epoch, loss=0.000469, prev_loss=0.000467]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:02, 49.82batch/s]

Training batches on cuda:  10%|▉         | 12/123 [00:00<00:02, 54.26batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:01, 58.46batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:00<00:01, 60.13batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:00<00:01, 63.13batch/s]

Training batches on cuda:  34%|███▍      | 42/123 [00:00<00:01, 64.94batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:00<00:01, 65.90batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:00<00:01, 65.38batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:00<00:00, 66.95batch/s]

Training batches on cuda:  59%|█████▊    | 72/123 [00:01<00:00, 66.10batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:01<00:00, 65.69batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:01<00:00, 67.71batch/s]

Training batches on cuda:  76%|███████▋  | 94/123 [00:01<00:00, 67.18batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:01<00:00, 61.02batch/s]

Training batches on cuda:  88%|████████▊ | 108/123 [00:01<00:00, 58.98batch/s]

Training batches on cuda:  93%|█████████▎| 114/123 [00:01<00:00, 58.51batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:01<00:00, 59.42batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:22<00:02,  2.12s/epoch, loss=0.000456, prev_loss=0.000469]2021-09-16:18:11:32,407 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:11:33,412 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:11:48,324 INFO     [evaluator.py:657] Evaluation took 14.91s seconds
2021-09-16:18:11:48,456 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:38<00:00,  6.95s/epoch, loss=0.000456, prev_loss=0.000469]
Training epochs on cuda: 100%|██████████| 10/10 [00:38<00:00,  3.81s/epoch, loss=0.000456, prev_loss=0.000469]
2021-09-16:18:11:48,457 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp_scve8v2'
2021-09-16:18:11:48,516 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_scve8v2' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.35ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.47ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:09, 2.53ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.56ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.59ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.61ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:12:03,388 INFO     [evaluator.py:657] Evaluation took 14.83s seconds
[I 2021-09-16 18:12:03,393] Trial 34 finished with value: 0.3811042288750114 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.016848728097173422, 'optimizer.lr': 0.05781292773033672, 'negative_sampler.num_negs_per_pos': 4, 'training.batch_size': 2532}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:12:03,414 WARNING  [api.py:823] No random seed is specified. Setting to 4279299836.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/103 [00:00<00:05, 19.31batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:04, 21.01batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:04, 19.80batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:04, 21.91batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:03, 24.08batch/s]

Training batches on cuda:  17%|█▋        | 18/103 [00:00<00:03, 24.93batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:00<00:03, 26.15batch/s]

Training batches on cuda:  23%|██▎       | 24/103 [00:00<00:02, 26.77batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:01<00:02, 27.09batch/s]

Training batches on cuda:  29%|██▉       | 30/103 [00:01<00:02, 27.38batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:01<00:02, 27.39batch/s]

Training batches on cuda:  35%|███▍      | 36/103 [00:01<00:02, 27.48batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:01<00:02, 27.55batch/s]

Training batches on cuda:  41%|████      | 42/103 [00:01<00:02, 27.65batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:01<00:02, 27.72batch/s]

Training batches on cuda:  47%|████▋     | 48/103 [00:01<00:01, 27.75batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:01<00:01, 28.00batch/s]

Training batches on cuda:  52%|█████▏    | 54/103 [00:01<00:01, 27.97batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:02<00:01, 27.77batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:02<00:01, 28.71batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:02<00:01, 29.22batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:02<00:01, 29.15batch/s]

Training batches on cuda:  70%|██████▉   | 72/103 [00:02<00:01, 29.35batch/s]

Training batches on cuda:  74%|███████▍  | 76/103 [00:02<00:00, 30.10batch/s]

Training batches on cuda:  78%|███████▊  | 80/103 [00:02<00:00, 30.42batch/s]

Training batches on cuda:  82%|████████▏ | 84/103 [00:02<00:00, 30.49batch/s]

Training batches on cuda:  85%|████████▌ | 88/103 [00:03<00:00, 30.65batch/s]

Training batches on cuda:  89%|████████▉ | 92/103 [00:03<00:00, 30.70batch/s]

Training batches on cuda:  93%|█████████▎| 96/103 [00:03<00:00, 30.77batch/s]

Training batches on cuda:  97%|█████████▋| 100/103 [00:03<00:00, 30.85batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000441, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:34,  3.82s/epoch, loss=0.000441, prev_loss=nan]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/103 [00:00<00:06, 16.32batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:05, 18.65batch/s]

Training batches on cuda:   8%|▊         | 8/103 [00:00<00:04, 20.72batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:04, 22.49batch/s]

Training batches on cuda:  14%|█▎        | 14/103 [00:00<00:03, 23.45batch/s]

Training batches on cuda:  17%|█▋        | 18/103 [00:00<00:03, 25.69batch/s]

Training batches on cuda:  21%|██▏       | 22/103 [00:00<00:02, 27.31batch/s]

Training batches on cuda:  25%|██▌       | 26/103 [00:00<00:02, 28.61batch/s]

Training batches on cuda:  29%|██▉       | 30/103 [00:01<00:02, 29.38batch/s]

Training batches on cuda:  33%|███▎      | 34/103 [00:01<00:02, 30.08batch/s]

Training batches on cuda:  37%|███▋      | 38/103 [00:01<00:02, 30.46batch/s]

Training batches on cuda:  41%|████      | 42/103 [00:01<00:01, 30.94batch/s]

Training batches on cuda:  45%|████▍     | 46/103 [00:01<00:01, 28.75batch/s]

Training batches on cuda:  49%|████▊     | 50/103 [00:01<00:01, 29.62batch/s]

Training batches on cuda:  52%|█████▏    | 54/103 [00:01<00:01, 30.18batch/s]

Training batches on cuda:  56%|█████▋    | 58/103 [00:01<00:01, 30.71batch/s]

Training batches on cuda:  60%|██████    | 62/103 [00:02<00:01, 30.94batch/s]

Training batches on cuda:  64%|██████▍   | 66/103 [00:02<00:01, 31.30batch/s]

Training batches on cuda:  68%|██████▊   | 70/103 [00:02<00:01, 31.45batch/s]

Training batches on cuda:  72%|███████▏  | 74/103 [00:02<00:00, 31.55batch/s]

Training batches on cuda:  76%|███████▌  | 78/103 [00:02<00:00, 31.75batch/s]

Training batches on cuda:  80%|███████▉  | 82/103 [00:02<00:00, 31.88batch/s]

Training batches on cuda:  83%|████████▎ | 86/103 [00:02<00:00, 31.71batch/s]

Training batches on cuda:  87%|████████▋ | 90/103 [00:02<00:00, 31.89batch/s]

Training batches on cuda:  91%|█████████▏| 94/103 [00:03<00:00, 31.94batch/s]

Training batches on cuda:  95%|█████████▌| 98/103 [00:03<00:00, 31.04batch/s]

Training batches on cuda:  99%|█████████▉| 102/103 [00:03<00:00, 31.51batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:07<00:34,  3.82s/epoch, loss=0.000437, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:30,  3.75s/epoch, loss=0.000437, prev_loss=nan]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:04, 23.62batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:03, 25.51batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:03, 27.18batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:03, 28.60batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:02, 29.59batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:00<00:02, 30.53batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:00<00:02, 31.06batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:00<00:02, 31.43batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:01<00:02, 31.33batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:01<00:02, 31.59batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:01<00:01, 31.79batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:01<00:01, 31.68batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:01<00:01, 31.84batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:01<00:01, 32.23batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:01<00:01, 32.34batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:01<00:01, 32.55batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:02<00:01, 32.49batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:02<00:00, 32.45batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:02<00:00, 32.38batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:02<00:00, 32.37batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:02<00:00, 32.50batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:02<00:00, 32.45batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:02<00:00, 32.23batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:03<00:00, 29.86batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:03<00:00, 30.59batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:03<00:00, 32.37batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:10<00:30,  3.75s/epoch, loss=0.000437, prev_loss=0.000437]
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:25,  3.68s/epoch, loss=0.000437, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:04, 24.45batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:03, 26.52batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:03, 28.02batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:03, 29.13batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:02, 29.85batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:00<00:02, 30.79batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:00<00:02, 31.11batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:00<00:02, 31.56batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:01<00:02, 31.70batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:01<00:02, 31.80batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:01<00:01, 32.02batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:01<00:01, 32.06batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:01<00:01, 32.06batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:01<00:01, 32.13batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:01<00:01, 31.98batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:01<00:01, 32.05batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:02<00:01, 32.13batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:02<00:00, 32.31batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:02<00:00, 32.06batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:02<00:00, 31.97batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:02<00:00, 32.27batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:02<00:00, 32.26batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:02<00:00, 32.24batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:02<00:00, 32.26batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:03<00:00, 32.22batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:03<00:00, 33.17batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:14<00:25,  3.68s/epoch, loss=0.000437, prev_loss=0.000437]
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:21,  3.61s/epoch, loss=0.000437, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/103 [00:00<00:05, 19.47batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:04, 21.48batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:03, 23.51batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:03, 25.29batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:00<00:03, 26.85batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:00<00:02, 27.97batch/s]

Training batches on cuda:  23%|██▎       | 24/103 [00:00<00:02, 28.07batch/s]

Training batches on cuda:  27%|██▋       | 28/103 [00:00<00:02, 28.64batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:01<00:02, 26.52batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:01<00:02, 27.58batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:01<00:02, 28.46batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:01<00:02, 29.11batch/s]

Training batches on cuda:  45%|████▍     | 46/103 [00:01<00:01, 28.88batch/s]

Training batches on cuda:  49%|████▊     | 50/103 [00:01<00:01, 29.24batch/s]

Training batches on cuda:  52%|█████▏    | 54/103 [00:01<00:01, 29.52batch/s]

Training batches on cuda:  56%|█████▋    | 58/103 [00:01<00:01, 29.79batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:02<00:01, 29.19batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:02<00:01, 29.71batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:02<00:01, 30.09batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:02<00:00, 30.14batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:02<00:00, 29.51batch/s]

Training batches on cuda:  78%|███████▊  | 80/103 [00:02<00:00, 29.00batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:02<00:00, 28.95batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:02<00:00, 29.50batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:03<00:00, 29.86batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:03<00:00, 30.01batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:03<00:00, 29.93batch/s]

Training batches on cuda:  99%|█████████▉| 102/103 [00:03<00:00, 29.93batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:18<00:21,  3.61s/epoch, loss=0.000438, prev_loss=0.000437]
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:18,  3.65s/epoch, loss=0.000438, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/103 [00:00<00:05, 16.97batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:05, 19.40batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:04, 21.82batch/s]

Training batches on cuda:  12%|█▏        | 12/103 [00:00<00:03, 23.73batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:03, 25.31batch/s]

Training batches on cuda:  17%|█▋        | 18/103 [00:00<00:03, 26.53batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:00<00:02, 27.43batch/s]

Training batches on cuda:  23%|██▎       | 24/103 [00:00<00:02, 27.92batch/s]

Training batches on cuda:  27%|██▋       | 28/103 [00:00<00:02, 28.79batch/s]

Training batches on cuda:  31%|███       | 32/103 [00:01<00:02, 29.56batch/s]

Training batches on cuda:  35%|███▍      | 36/103 [00:01<00:02, 29.43batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:01<00:02, 27.35batch/s]

Training batches on cuda:  41%|████      | 42/103 [00:01<00:02, 27.77batch/s]

Training batches on cuda:  45%|████▍     | 46/103 [00:01<00:02, 28.49batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:01<00:01, 27.90batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:01<00:01, 28.60batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:01<00:01, 29.23batch/s]

Training batches on cuda:  58%|█████▊    | 60/103 [00:02<00:01, 29.31batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:02<00:01, 29.31batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:02<00:01, 29.84batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:02<00:01, 30.11batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:02<00:00, 30.26batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:02<00:00, 30.54batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:02<00:00, 30.47batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:02<00:00, 29.52batch/s]

Training batches on cuda:  87%|████████▋ | 90/103 [00:03<00:00, 28.16batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:03<00:00, 28.14batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:03<00:00, 28.77batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:03<00:00, 29.40batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:21<00:18,  3.65s/epoch, loss=0.000438, prev_loss=0.000438]
Training epochs on cuda:  60%|██████    | 6/10 [00:21<00:14,  3.68s/epoch, loss=0.000438, prev_loss=0.000438]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:10,  9.60batch/s]

Training batches on cuda:   4%|▍         | 4/103 [00:00<00:08, 11.98batch/s]

Training batches on cuda:   8%|▊         | 8/103 [00:00<00:06, 14.66batch/s]

Training batches on cuda:  12%|█▏        | 12/103 [00:00<00:05, 17.35batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:04, 19.82batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:03, 22.11batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:00<00:03, 24.01batch/s]

Training batches on cuda:  25%|██▌       | 26/103 [00:00<00:03, 25.53batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:01<00:02, 26.69batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:01<00:02, 27.74batch/s]

Training batches on cuda:  35%|███▍      | 36/103 [00:01<00:02, 28.05batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:01<00:02, 28.38batch/s]

Training batches on cuda:  41%|████      | 42/103 [00:01<00:02, 28.83batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:01<00:01, 29.07batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:01<00:01, 29.19batch/s]

Training batches on cuda:  50%|█████     | 52/103 [00:01<00:01, 28.87batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:01<00:01, 28.92batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:02<00:01, 29.42batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:02<00:01, 29.83batch/s]

Training batches on cuda:  64%|██████▍   | 66/103 [00:02<00:01, 29.14batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:02<00:01, 28.88batch/s]

Training batches on cuda:  70%|██████▉   | 72/103 [00:02<00:01, 27.92batch/s]

Training batches on cuda:  74%|███████▍  | 76/103 [00:02<00:00, 29.22batch/s]

Training batches on cuda:  78%|███████▊  | 80/103 [00:02<00:00, 30.04batch/s]

Training batches on cuda:  82%|████████▏ | 84/103 [00:02<00:00, 30.68batch/s]

Training batches on cuda:  85%|████████▌ | 88/103 [00:03<00:00, 31.32batch/s]

Training batches on cuda:  89%|████████▉ | 92/103 [00:03<00:00, 30.12batch/s]

Training batches on cuda:  93%|█████████▎| 96/103 [00:03<00:00, 29.34batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:03<00:00, 27.79batch/s]

Training batches on cuda:  99%|█████████▉| 102/103 [00:03<00:00, 28.24batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:25<00:14,  3.68s/epoch, loss=0.000438, prev_loss=0.000438]
Training epochs on cuda:  70%|███████   | 7/10 [00:25<00:11,  3.73s/epoch, loss=0.000438, prev_loss=0.000438]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/103 [00:00<00:05, 18.06batch/s]

Training batches on cuda:   6%|▌         | 6/103 [00:00<00:04, 20.62batch/s]

Training batches on cuda:  10%|▉         | 10/103 [00:00<00:04, 22.77batch/s]

Training batches on cuda:  14%|█▎        | 14/103 [00:00<00:03, 24.42batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:00<00:03, 25.63batch/s]

Training batches on cuda:  19%|█▉        | 20/103 [00:00<00:03, 26.63batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:00<00:02, 27.52batch/s]

Training batches on cuda:  25%|██▌       | 26/103 [00:00<00:02, 28.13batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:01<00:02, 28.58batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:01<00:02, 29.17batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:01<00:02, 29.67batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:01<00:02, 29.91batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:01<00:01, 30.24batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:01<00:01, 30.57batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:01<00:01, 30.54batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:01<00:01, 30.72batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:02<00:01, 30.79batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:02<00:01, 30.53batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:02<00:01, 30.64batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:02<00:00, 30.84batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:02<00:00, 30.95batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:02<00:00, 31.00batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:02<00:00, 30.97batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:02<00:00, 29.72batch/s]

Training batches on cuda:  89%|████████▉ | 92/103 [00:03<00:00, 29.54batch/s]

Training batches on cuda:  93%|█████████▎| 96/103 [00:03<00:00, 30.04batch/s]

Training batches on cuda:  97%|█████████▋| 100/103 [00:03<00:00, 30.26batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:29<00:11,  3.73s/epoch, loss=0.000438, prev_loss=0.000438]
Training epochs on cuda:  80%|████████  | 8/10 [00:29<00:07,  3.70s/epoch, loss=0.000438, prev_loss=0.000438]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:04, 22.79batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:03, 24.79batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:03, 26.34batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:00<00:03, 27.51batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:02, 28.51batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:00<00:02, 29.23batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:00<00:02, 29.62batch/s]

Training batches on cuda:  29%|██▉       | 30/103 [00:01<00:02, 29.41batch/s]

Training batches on cuda:  33%|███▎      | 34/103 [00:01<00:02, 29.52batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:01<00:02, 29.64batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:01<00:02, 29.55batch/s]

Training batches on cuda:  43%|████▎     | 44/103 [00:01<00:01, 29.60batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:01<00:01, 29.60batch/s]

Training batches on cuda:  49%|████▊     | 50/103 [00:01<00:01, 29.61batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:01<00:01, 29.57batch/s]

Training batches on cuda:  54%|█████▍    | 56/103 [00:01<00:01, 29.69batch/s]

Training batches on cuda:  58%|█████▊    | 60/103 [00:02<00:01, 29.82batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:02<00:01, 29.77batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:02<00:01, 29.88batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:02<00:01, 30.13batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:02<00:00, 29.77batch/s]

Training batches on cuda:  76%|███████▌  | 78/103 [00:02<00:00, 29.65batch/s]

Training batches on cuda:  80%|███████▉  | 82/103 [00:02<00:00, 29.95batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:02<00:00, 29.93batch/s]

Training batches on cuda:  85%|████████▌ | 88/103 [00:02<00:00, 29.79batch/s]

Training batches on cuda:  89%|████████▉ | 92/103 [00:03<00:00, 30.08batch/s]

Training batches on cuda:  93%|█████████▎| 96/103 [00:03<00:00, 30.03batch/s]

Training batches on cuda:  97%|█████████▋| 100/103 [00:03<00:00, 29.92batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:33<00:07,  3.70s/epoch, loss=0.000438, prev_loss=0.000438]
Training epochs on cuda:  90%|█████████ | 9/10 [00:33<00:03,  3.69s/epoch, loss=0.000438, prev_loss=0.000438]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:03, 25.41batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:03, 26.81batch/s]

Training batches on cuda:  10%|▉         | 10/103 [00:00<00:03, 27.10batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:03, 27.48batch/s]

Training batches on cuda:  16%|█▌        | 16/103 [00:00<00:03, 27.26batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:00<00:03, 27.36batch/s]

Training batches on cuda:  21%|██▏       | 22/103 [00:00<00:02, 27.58batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:00<00:02, 27.65batch/s]

Training batches on cuda:  27%|██▋       | 28/103 [00:01<00:02, 27.72batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:01<00:02, 27.77batch/s]

Training batches on cuda:  33%|███▎      | 34/103 [00:01<00:02, 27.74batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:01<00:02, 27.86batch/s]

Training batches on cuda:  39%|███▉      | 40/103 [00:01<00:02, 28.09batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:01<00:02, 28.05batch/s]

Training batches on cuda:  45%|████▍     | 46/103 [00:01<00:02, 27.49batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:01<00:01, 27.68batch/s]

Training batches on cuda:  50%|█████     | 52/103 [00:01<00:01, 27.62batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:01<00:01, 28.05batch/s]

Training batches on cuda:  56%|█████▋    | 58/103 [00:02<00:01, 28.15batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:02<00:01, 28.30batch/s]

Training batches on cuda:  62%|██████▏   | 64/103 [00:02<00:01, 28.08batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:02<00:01, 28.15batch/s]

Training batches on cuda:  68%|██████▊   | 70/103 [00:02<00:01, 25.55batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:02<00:01, 26.23batch/s]

Training batches on cuda:  74%|███████▍  | 76/103 [00:02<00:01, 26.82batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:02<00:00, 27.17batch/s]

Training batches on cuda:  80%|███████▉  | 82/103 [00:02<00:00, 27.51batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:03<00:00, 27.92batch/s]

Training batches on cuda:  85%|████████▌ | 88/103 [00:03<00:00, 27.78batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:03<00:00, 27.99batch/s]

Training batches on cuda:  91%|█████████▏| 94/103 [00:03<00:00, 28.03batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:03<00:00, 28.32batch/s]

Training batches on cuda:  97%|█████████▋| 100/103 [00:03<00:00, 28.47batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:36<00:03,  3.69s/epoch, loss=0.000438, prev_loss=0.000438]2021-09-16:18:12:40,860 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:12:41,860 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:12:56,788 INFO     [evaluator.py:657] Evaluation took 14.93s seconds
2021-09-16:18:12:56,920 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00,  8.57s/epoch, loss=0.000438, prev_loss=0.000438]
Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00,  5.30s/epoch, loss=0.000438, prev_loss=0.000438]
2021-09-16:18:12:56,920 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpxcob2smy'
2021-09-16:18:12:56,987 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxcob2smy' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.16ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.29ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.39ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.56ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:13:11,819 INFO     [evaluator.py:657] Evaluation took 14.81s seconds
[I 2021-09-16 18:13:11,824] Trial 35 finished with value: -0.00421726687761792 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.08357970763050988, 'optimizer.lr': 0.07664380708788704, 'negative_sampler.num_negs_per_pos': 14, 'training.batch_size': 3032}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:13:11,849 WARNING  [api.py:823] No random seed is specified. Setting to 2862923998.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:03, 55.38batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:02, 61.96batch/s]

Training batches on cuda:  13%|█▎        | 24/191 [00:00<00:02, 67.68batch/s]

Training batches on cuda:  17%|█▋        | 33/191 [00:00<00:02, 72.14batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:00<00:01, 78.38batch/s]

Training batches on cuda:  28%|██▊       | 53/191 [00:00<00:01, 83.72batch/s]

Training batches on cuda:  33%|███▎      | 63/191 [00:00<00:01, 87.66batch/s]

Training batches on cuda:  39%|███▊      | 74/191 [00:00<00:01, 91.22batch/s]

Training batches on cuda:  44%|████▍     | 84/191 [00:00<00:01, 93.43batch/s]

Training batches on cuda:  49%|████▉     | 94/191 [00:01<00:01, 91.34batch/s]

Training batches on cuda:  54%|█████▍    | 104/191 [00:01<00:00, 89.53batch/s]

Training batches on cuda:  59%|█████▉    | 113/191 [00:01<00:00, 88.90batch/s]

Training batches on cuda:  64%|██████▍   | 123/191 [00:01<00:00, 90.86batch/s]

Training batches on cuda:  70%|██████▉   | 133/191 [00:01<00:00, 92.89batch/s]

Training batches on cuda:  75%|███████▍  | 143/191 [00:01<00:00, 94.82batch/s]

Training batches on cuda:  80%|████████  | 153/191 [00:01<00:00, 96.19batch/s]

Training batches on cuda:  85%|████████▌ | 163/191 [00:01<00:00, 93.23batch/s]

Training batches on cuda:  91%|█████████ | 173/191 [00:01<00:00, 89.72batch/s]

Training batches on cuda:  96%|█████████▌| 183/191 [00:02<00:00, 88.51batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000812, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:20,  2.30s/epoch, loss=0.000812, prev_loss=nan]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:02, 67.44batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:02, 74.61batch/s]

Training batches on cuda:  14%|█▍        | 27/191 [00:00<00:02, 80.56batch/s]

Training batches on cuda:  19%|█▉        | 37/191 [00:00<00:01, 85.45batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:00<00:01, 88.97batch/s]

Training batches on cuda:  30%|██▉       | 57/191 [00:00<00:01, 90.98batch/s]

Training batches on cuda:  35%|███▌      | 67/191 [00:00<00:01, 93.06batch/s]

Training batches on cuda:  40%|████      | 77/191 [00:00<00:01, 94.37batch/s]

Training batches on cuda:  46%|████▌     | 87/191 [00:00<00:01, 89.99batch/s]

Training batches on cuda:  50%|█████     | 96/191 [00:01<00:01, 88.75batch/s]

Training batches on cuda:  55%|█████▍    | 105/191 [00:01<00:00, 88.30batch/s]

Training batches on cuda:  60%|█████▉    | 114/191 [00:01<00:00, 88.08batch/s]

Training batches on cuda:  64%|██████▍   | 123/191 [00:01<00:00, 86.37batch/s]

Training batches on cuda:  69%|██████▉   | 132/191 [00:01<00:00, 86.99batch/s]

Training batches on cuda:  74%|███████▍  | 142/191 [00:01<00:00, 88.23batch/s]

Training batches on cuda:  79%|███████▉  | 151/191 [00:01<00:00, 81.99batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:01<00:00, 83.91batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:01<00:00, 86.31batch/s]

Training batches on cuda:  94%|█████████▍| 180/191 [00:02<00:00, 87.96batch/s]

Training batches on cuda:  99%|█████████▉| 189/191 [00:02<00:00, 88.35batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:20,  2.30s/epoch, loss=0.000809, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:18,  2.31s/epoch, loss=0.000809, prev_loss=nan]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:02, 63.93batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:02, 70.52batch/s]

Training batches on cuda:  14%|█▍        | 27/191 [00:00<00:02, 75.55batch/s]

Training batches on cuda:  19%|█▉        | 36/191 [00:00<00:01, 78.58batch/s]

Training batches on cuda:  24%|██▍       | 46/191 [00:00<00:01, 81.74batch/s]

Training batches on cuda:  29%|██▉       | 56/191 [00:00<00:01, 84.71batch/s]

Training batches on cuda:  34%|███▍      | 65/191 [00:00<00:01, 85.56batch/s]

Training batches on cuda:  39%|███▊      | 74/191 [00:00<00:01, 82.45batch/s]

Training batches on cuda:  43%|████▎     | 82/191 [00:00<00:01, 80.74batch/s]

Training batches on cuda:  48%|████▊     | 91/191 [00:01<00:01, 82.52batch/s]

Training batches on cuda:  53%|█████▎    | 101/191 [00:01<00:01, 85.63batch/s]

Training batches on cuda:  58%|█████▊    | 111/191 [00:01<00:00, 87.01batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:01<00:00, 85.10batch/s]

Training batches on cuda:  68%|██████▊   | 130/191 [00:01<00:00, 87.75batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:01<00:00, 90.19batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:01<00:00, 91.84batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:01<00:00, 92.83batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:01<00:00, 93.78batch/s]

Training batches on cuda:  94%|█████████▍| 180/191 [00:02<00:00, 94.82batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:02<00:00, 95.03batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:18,  2.31s/epoch, loss=0.000809, prev_loss=0.000809]
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:16,  2.34s/epoch, loss=0.000809, prev_loss=0.000809]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:02, 69.01batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:02, 75.86batch/s]

Training batches on cuda:  14%|█▍        | 27/191 [00:00<00:02, 81.60batch/s]

Training batches on cuda:  20%|█▉        | 38/191 [00:00<00:01, 86.43batch/s]

Training batches on cuda:  25%|██▌       | 48/191 [00:00<00:01, 88.98batch/s]

Training batches on cuda:  30%|███       | 58/191 [00:00<00:01, 91.44batch/s]

Training batches on cuda:  36%|███▌      | 68/191 [00:00<00:01, 93.53batch/s]

Training batches on cuda:  41%|████      | 78/191 [00:00<00:01, 94.52batch/s]

Training batches on cuda:  46%|████▌     | 88/191 [00:00<00:01, 95.96batch/s]

Training batches on cuda:  52%|█████▏    | 99/191 [00:01<00:00, 97.10batch/s]

Training batches on cuda:  57%|█████▋    | 109/191 [00:01<00:00, 96.97batch/s]

Training batches on cuda:  62%|██████▏   | 119/191 [00:01<00:00, 97.40batch/s]

Training batches on cuda:  68%|██████▊   | 130/191 [00:01<00:00, 98.24batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:01<00:00, 96.64batch/s]

Training batches on cuda:  79%|███████▉  | 151/191 [00:01<00:00, 97.74batch/s]

Training batches on cuda:  84%|████████▍ | 161/191 [00:01<00:00, 96.54batch/s]

Training batches on cuda:  90%|████████▉ | 171/191 [00:01<00:00, 97.05batch/s]

Training batches on cuda:  95%|█████████▍| 181/191 [00:01<00:00, 97.55batch/s]

Training batches on cuda: 100%|██████████| 191/191 [00:01<00:00, 97.38batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:16,  2.34s/epoch, loss=0.000809, prev_loss=0.000809]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:13,  2.29s/epoch, loss=0.000809, prev_loss=0.000809]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/191 [00:00<00:03, 48.80batch/s]

Training batches on cuda:   7%|▋         | 14/191 [00:00<00:03, 56.52batch/s]

Training batches on cuda:  13%|█▎        | 24/191 [00:00<00:02, 63.64batch/s]

Training batches on cuda:  18%|█▊        | 34/191 [00:00<00:02, 70.66batch/s]

Training batches on cuda:  23%|██▎       | 44/191 [00:00<00:01, 76.47batch/s]

Training batches on cuda:  28%|██▊       | 54/191 [00:00<00:01, 81.16batch/s]

Training batches on cuda:  34%|███▎      | 64/191 [00:00<00:01, 84.80batch/s]

Training batches on cuda:  39%|███▊      | 74/191 [00:00<00:01, 87.31batch/s]

Training batches on cuda:  44%|████▍     | 84/191 [00:00<00:01, 89.97batch/s]

Training batches on cuda:  49%|████▉     | 94/191 [00:01<00:01, 91.87batch/s]

Training batches on cuda:  54%|█████▍    | 104/191 [00:01<00:00, 92.75batch/s]

Training batches on cuda:  60%|█████▉    | 114/191 [00:01<00:00, 93.84batch/s]

Training batches on cuda:  65%|██████▍   | 124/191 [00:01<00:00, 94.65batch/s]

Training batches on cuda:  70%|███████   | 134/191 [00:01<00:00, 89.45batch/s]

Training batches on cuda:  75%|███████▌  | 144/191 [00:01<00:00, 90.58batch/s]

Training batches on cuda:  81%|████████  | 154/191 [00:01<00:00, 92.56batch/s]

Training batches on cuda:  86%|████████▌ | 164/191 [00:01<00:00, 94.49batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:01<00:00, 94.87batch/s]

Training batches on cuda:  96%|█████████▋| 184/191 [00:02<00:00, 95.53batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:13,  2.29s/epoch, loss=0.000809, prev_loss=0.000809]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.29s/epoch, loss=0.000809, prev_loss=0.000809]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:06, 29.79batch/s]

Training batches on cuda:   6%|▋         | 12/191 [00:00<00:04, 36.80batch/s]

Training batches on cuda:  10%|█         | 20/191 [00:00<00:03, 42.96batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:03, 49.20batch/s]

Training batches on cuda:  20%|█▉        | 38/191 [00:00<00:02, 56.96batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:00<00:02, 62.75batch/s]

Training batches on cuda:  28%|██▊       | 54/191 [00:00<00:02, 62.99batch/s]

Training batches on cuda:  34%|███▎      | 64/191 [00:00<00:01, 70.53batch/s]

Training batches on cuda:  39%|███▉      | 75/191 [00:00<00:01, 78.12batch/s]

Training batches on cuda:  45%|████▌     | 86/191 [00:01<00:01, 84.82batch/s]

Training batches on cuda:  50%|█████     | 96/191 [00:01<00:01, 79.91batch/s]

Training batches on cuda:  56%|█████▌    | 107/191 [00:01<00:00, 85.11batch/s]

Training batches on cuda:  61%|██████▏   | 117/191 [00:01<00:00, 88.54batch/s]

Training batches on cuda:  66%|██████▋   | 127/191 [00:01<00:00, 91.26batch/s]

Training batches on cuda:  72%|███████▏  | 138/191 [00:01<00:00, 94.04batch/s]

Training batches on cuda:  77%|███████▋  | 148/191 [00:01<00:00, 95.61batch/s]

Training batches on cuda:  83%|████████▎ | 158/191 [00:01<00:00, 95.81batch/s]

Training batches on cuda:  88%|████████▊ | 168/191 [00:01<00:00, 96.65batch/s]

Training batches on cuda:  93%|█████████▎| 178/191 [00:02<00:00, 96.13batch/s]

Training batches on cuda:  98%|█████████▊| 188/191 [00:02<00:00, 93.91batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:11,  2.29s/epoch, loss=0.000809, prev_loss=0.000809]
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:09,  2.34s/epoch, loss=0.000809, prev_loss=0.000809]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:02, 65.74batch/s]

Training batches on cuda:   8%|▊         | 16/191 [00:00<00:02, 71.22batch/s]

Training batches on cuda:  13%|█▎        | 25/191 [00:00<00:02, 74.76batch/s]

Training batches on cuda:  18%|█▊        | 34/191 [00:00<00:02, 78.26batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:00<00:01, 80.60batch/s]

Training batches on cuda:  27%|██▋       | 52/191 [00:00<00:01, 81.40batch/s]

Training batches on cuda:  32%|███▏      | 61/191 [00:00<00:01, 82.97batch/s]

Training batches on cuda:  37%|███▋      | 70/191 [00:00<00:01, 84.07batch/s]

Training batches on cuda:  41%|████▏     | 79/191 [00:00<00:01, 84.88batch/s]

Training batches on cuda:  47%|████▋     | 89/191 [00:01<00:01, 86.98batch/s]

Training batches on cuda:  52%|█████▏    | 99/191 [00:01<00:01, 87.79batch/s]

Training batches on cuda:  57%|█████▋    | 109/191 [00:01<00:00, 89.16batch/s]

Training batches on cuda:  62%|██████▏   | 119/191 [00:01<00:00, 89.70batch/s]

Training batches on cuda:  67%|██████▋   | 128/191 [00:01<00:00, 88.46batch/s]

Training batches on cuda:  72%|███████▏  | 137/191 [00:01<00:00, 88.62batch/s]

Training batches on cuda:  76%|███████▋  | 146/191 [00:01<00:00, 80.56batch/s]

Training batches on cuda:  81%|████████  | 155/191 [00:01<00:00, 82.14batch/s]

Training batches on cuda:  86%|████████▌ | 164/191 [00:01<00:00, 82.43batch/s]

Training batches on cuda:  91%|█████████ | 173/191 [00:02<00:00, 77.97batch/s]

Training batches on cuda:  95%|█████████▌| 182/191 [00:02<00:00, 80.05batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:16<00:09,  2.34s/epoch, loss=0.000809, prev_loss=0.000809]
Training epochs on cuda:  70%|███████   | 7/10 [00:16<00:07,  2.38s/epoch, loss=0.000809, prev_loss=0.000809]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:03, 55.02batch/s]

Training batches on cuda:   7%|▋         | 14/191 [00:00<00:02, 60.70batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:02, 66.40batch/s]

Training batches on cuda:  17%|█▋        | 32/191 [00:00<00:02, 70.94batch/s]

Training batches on cuda:  21%|██▏       | 41/191 [00:00<00:01, 75.12batch/s]

Training batches on cuda:  27%|██▋       | 51/191 [00:00<00:01, 80.28batch/s]

Training batches on cuda:  32%|███▏      | 61/191 [00:00<00:01, 84.47batch/s]

Training batches on cuda:  37%|███▋      | 71/191 [00:00<00:01, 87.60batch/s]

Training batches on cuda:  42%|████▏     | 81/191 [00:00<00:01, 90.41batch/s]

Training batches on cuda:  48%|████▊     | 91/191 [00:01<00:01, 92.54batch/s]

Training batches on cuda:  53%|█████▎    | 101/191 [00:01<00:00, 92.86batch/s]

Training batches on cuda:  58%|█████▊    | 111/191 [00:01<00:00, 94.68batch/s]

Training batches on cuda:  63%|██████▎   | 121/191 [00:01<00:00, 86.19batch/s]

Training batches on cuda:  69%|██████▊   | 131/191 [00:01<00:00, 89.51batch/s]

Training batches on cuda:  74%|███████▍  | 141/191 [00:01<00:00, 91.96batch/s]

Training batches on cuda:  80%|███████▉  | 152/191 [00:01<00:00, 94.25batch/s]

Training batches on cuda:  85%|████████▍ | 162/191 [00:01<00:00, 95.81batch/s]

Training batches on cuda:  90%|█████████ | 172/191 [00:01<00:00, 96.91batch/s]

Training batches on cuda:  95%|█████████▌| 182/191 [00:02<00:00, 96.35batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:18<00:07,  2.38s/epoch, loss=0.000809, prev_loss=0.000809]
Training epochs on cuda:  80%|████████  | 8/10 [00:18<00:04,  2.37s/epoch, loss=0.000809, prev_loss=0.000809]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:02, 68.08batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:02, 74.91batch/s]

Training batches on cuda:  14%|█▍        | 27/191 [00:00<00:02, 80.33batch/s]

Training batches on cuda:  19%|█▉        | 37/191 [00:00<00:01, 84.35batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:00<00:01, 87.73batch/s]

Training batches on cuda:  30%|██▉       | 57/191 [00:00<00:01, 90.49batch/s]

Training batches on cuda:  35%|███▌      | 67/191 [00:00<00:01, 92.18batch/s]

Training batches on cuda:  40%|███▉      | 76/191 [00:00<00:01, 89.89batch/s]

Training batches on cuda:  45%|████▍     | 85/191 [00:00<00:01, 87.44batch/s]

Training batches on cuda:  49%|████▉     | 94/191 [00:01<00:01, 83.74batch/s]

Training batches on cuda:  54%|█████▍    | 103/191 [00:01<00:01, 81.48batch/s]

Training batches on cuda:  59%|█████▊    | 112/191 [00:01<00:00, 80.37batch/s]

Training batches on cuda:  63%|██████▎   | 121/191 [00:01<00:00, 81.40batch/s]

Training batches on cuda:  68%|██████▊   | 130/191 [00:01<00:00, 80.58batch/s]

Training batches on cuda:  73%|███████▎  | 139/191 [00:01<00:00, 80.18batch/s]

Training batches on cuda:  77%|███████▋  | 148/191 [00:01<00:00, 79.94batch/s]

Training batches on cuda:  82%|████████▏ | 157/191 [00:01<00:00, 79.63batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:01<00:00, 79.23batch/s]

Training batches on cuda:  91%|█████████ | 173/191 [00:02<00:00, 78.80batch/s]

Training batches on cuda:  95%|█████████▍| 181/191 [00:02<00:00, 77.96batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:02<00:00, 79.75batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:21<00:04,  2.37s/epoch, loss=0.000809, prev_loss=0.000809]
Training epochs on cuda:  90%|█████████ | 9/10 [00:21<00:02,  2.40s/epoch, loss=0.000809, prev_loss=0.000809]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:03, 54.68batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:02, 61.57batch/s]

Training batches on cuda:  13%|█▎        | 25/191 [00:00<00:02, 68.65batch/s]

Training batches on cuda:  17%|█▋        | 33/191 [00:00<00:02, 70.60batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:00<00:01, 75.74batch/s]

Training batches on cuda:  28%|██▊       | 53/191 [00:00<00:01, 79.88batch/s]

Training batches on cuda:  33%|███▎      | 63/191 [00:00<00:01, 82.55batch/s]

Training batches on cuda:  38%|███▊      | 72/191 [00:00<00:01, 82.45batch/s]

Training batches on cuda:  42%|████▏     | 81/191 [00:00<00:01, 83.25batch/s]

Training batches on cuda:  48%|████▊     | 91/191 [00:01<00:01, 85.80batch/s]

Training batches on cuda:  53%|█████▎    | 101/191 [00:01<00:01, 87.71batch/s]

Training batches on cuda:  58%|█████▊    | 110/191 [00:01<00:00, 87.79batch/s]

Training batches on cuda:  62%|██████▏   | 119/191 [00:01<00:00, 86.79batch/s]

Training batches on cuda:  67%|██████▋   | 128/191 [00:01<00:00, 86.99batch/s]

Training batches on cuda:  72%|███████▏  | 137/191 [00:01<00:00, 86.28batch/s]

Training batches on cuda:  76%|███████▋  | 146/191 [00:01<00:00, 86.17batch/s]

Training batches on cuda:  81%|████████  | 155/191 [00:01<00:00, 87.14batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:01<00:00, 88.62batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:02<00:00, 87.93batch/s]

Training batches on cuda:  96%|█████████▌| 183/191 [00:02<00:00, 86.12batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:23<00:02,  2.40s/epoch, loss=0.000809, prev_loss=0.000809]2021-09-16:18:13:35,969 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:13:37,79 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:13:51,962 INFO     [evaluator.py:657] Evaluation took 14.88s seconds
2021-09-16:18:13:52,89 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  7.25s/epoch, loss=0.000809, prev_loss=0.000809]
Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  3.98s/epoch, loss=0.000809, prev_loss=0.000809]
2021-09-16:18:13:52,90 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpyw0zyqmk'
2021-09-16:18:13:52,164 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpyw0zyqmk' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.59ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.61ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:14:06,983 INFO     [evaluator.py:657] Evaluation took 14.81s seconds
[I 2021-09-16 18:14:06,988] Trial 36 finished with value: 0.027327681176892105 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.08061928589036754, 'optimizer.lr': 0.026880914428396255, 'negative_sampler.num_negs_per_pos': 5, 'training.batch_size': 1632}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:14:07,9 WARNING  [api.py:823] No random seed is specified. Setting to 2025912935.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:07, 18.62batch/s]

Training batches on cuda:   6%|▌         | 8/140 [00:00<00:05, 23.09batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 27.81batch/s]

Training batches on cuda:  15%|█▌        | 21/140 [00:00<00:03, 33.19batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 36.86batch/s]

Training batches on cuda:  24%|██▎       | 33/140 [00:00<00:02, 41.51batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:00<00:02, 46.96batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:00<00:01, 52.08batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:00<00:01, 56.61batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:01, 60.33batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:01<00:01, 63.17batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:01<00:00, 65.00batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:01<00:00, 66.71batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:01<00:00, 67.99batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:01<00:00, 69.03batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:01<00:00, 69.68batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:01<00:00, 70.20batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:01<00:00, 70.23batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:02<00:00, 70.50batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000464, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:21,  2.36s/epoch, loss=0.000464, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:02, 52.77batch/s]

Training batches on cuda:   9%|▉         | 13/140 [00:00<00:02, 56.75batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:01, 60.13batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:01, 62.90batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:00<00:01, 62.93batch/s]

Training batches on cuda:  29%|██▉       | 41/140 [00:00<00:01, 59.52batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:00<00:01, 58.61batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:00<00:01, 61.76batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:00<00:01, 64.01batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 65.76batch/s]

Training batches on cuda:  55%|█████▌    | 77/140 [00:01<00:00, 66.82batch/s]

Training batches on cuda:  61%|██████    | 85/140 [00:01<00:00, 67.77batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:01<00:00, 67.69batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:01<00:00, 67.52batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:01<00:00, 67.83batch/s]

Training batches on cuda:  81%|████████  | 113/140 [00:01<00:00, 62.20batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:01<00:00, 60.61batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:01<00:00, 60.89batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:02<00:00, 60.87batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:21,  2.36s/epoch, loss=0.000454, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:18,  2.36s/epoch, loss=0.000454, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 38.41batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:02, 43.29batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:02, 47.40batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:02, 52.58batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:01, 57.00batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:00<00:01, 60.44batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:00<00:01, 63.17batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:00<00:01, 65.12batch/s]

Training batches on cuda:  46%|████▋     | 65/140 [00:01<00:01, 66.44batch/s]

Training batches on cuda:  51%|█████▏    | 72/140 [00:01<00:01, 66.31batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:01<00:00, 61.91batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:01<00:00, 60.29batch/s]

Training batches on cuda:  66%|██████▋   | 93/140 [00:01<00:00, 58.91batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:01<00:00, 58.27batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:01<00:00, 57.59batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:01<00:00, 54.20batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:01<00:00, 56.41batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:02<00:00, 56.06batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:02<00:00, 56.81batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:02<00:00, 57.96batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:18,  2.36s/epoch, loss=0.000454, prev_loss=0.000454]
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:16,  2.41s/epoch, loss=0.000454, prev_loss=0.000454]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:04, 29.96batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:03, 35.93batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:03, 40.60batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:02, 44.61batch/s]

Training batches on cuda:  21%|██        | 29/140 [00:00<00:02, 48.88batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:00<00:02, 50.51batch/s]

Training batches on cuda:  29%|██▉       | 41/140 [00:00<00:01, 50.18batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:00<00:01, 48.57batch/s]

Training batches on cuda:  38%|███▊      | 53/140 [00:00<00:01, 49.85batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:01<00:01, 52.60batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:01, 55.38batch/s]

Training batches on cuda:  52%|█████▏    | 73/140 [00:01<00:01, 53.68batch/s]

Training batches on cuda:  57%|█████▋    | 80/140 [00:01<00:01, 55.77batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:01<00:00, 58.70batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:01<00:00, 61.32batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:01<00:00, 60.50batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:01<00:00, 59.48batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:02<00:00, 59.72batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:02<00:00, 58.97batch/s]

Training batches on cuda:  91%|█████████▏| 128/140 [00:02<00:00, 58.40batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:02<00:00, 58.84batch/s]

Training batches on cuda: 100%|██████████| 140/140 [00:02<00:00, 59.07batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:16,  2.41s/epoch, loss=0.000454, prev_loss=0.000454]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:14,  2.48s/epoch, loss=0.000454, prev_loss=0.000454]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:04, 29.37batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:03, 34.94batch/s]

Training batches on cuda:  12%|█▏        | 17/140 [00:00<00:03, 40.77batch/s]

Training batches on cuda:  17%|█▋        | 24/140 [00:00<00:02, 46.13batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:02, 50.51batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:00<00:01, 54.66batch/s]

Training batches on cuda:  32%|███▏      | 45/140 [00:00<00:01, 57.13batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:00<00:01, 55.95batch/s]

Training batches on cuda:  41%|████      | 57/140 [00:00<00:01, 56.45batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:01, 59.22batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:01<00:01, 61.51batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:01<00:00, 62.88batch/s]

Training batches on cuda:  61%|██████    | 85/140 [00:01<00:00, 64.21batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:01<00:00, 65.00batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:01<00:00, 65.00batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:01<00:00, 65.79batch/s]

Training batches on cuda:  81%|████████  | 113/140 [00:01<00:00, 64.48batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:01<00:00, 64.16batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:02<00:00, 61.64batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:02<00:00, 58.12batch/s]

Training batches on cuda: 100%|██████████| 140/140 [00:02<00:00, 56.89batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:14,  2.48s/epoch, loss=0.000454, prev_loss=0.000454]
Training epochs on cuda:  50%|█████     | 5/10 [00:12<00:12,  2.49s/epoch, loss=0.000454, prev_loss=0.000454]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 5/140 [00:00<00:02, 49.40batch/s]

Training batches on cuda:   9%|▉         | 13/140 [00:00<00:02, 54.58batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:02, 57.62batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:01, 61.41batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:00<00:01, 63.95batch/s]

Training batches on cuda:  31%|███▏      | 44/140 [00:00<00:01, 66.19batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:00<00:01, 67.92batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:00<00:01, 69.35batch/s]

Training batches on cuda:  49%|████▊     | 68/140 [00:00<00:01, 70.53batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:00, 71.40batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:01<00:00, 72.08batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:01<00:00, 72.31batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:01<00:00, 72.57batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:01<00:00, 72.77batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:01<00:00, 72.82batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:01<00:00, 71.01batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:01<00:00, 71.39batch/s]

Training batches on cuda: 100%|██████████| 140/140 [00:01<00:00, 72.49batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:12,  2.49s/epoch, loss=0.000454, prev_loss=0.000454]
Training epochs on cuda:  60%|██████    | 6/10 [00:14<00:09,  2.40s/epoch, loss=0.000454, prev_loss=0.000454]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 39.82batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:02, 45.43batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:02, 49.54batch/s]

Training batches on cuda:  17%|█▋        | 24/140 [00:00<00:02, 51.49batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:02, 54.13batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:00<00:01, 56.00batch/s]

Training batches on cuda:  32%|███▏      | 45/140 [00:00<00:01, 57.79batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:00<00:01, 60.74batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:00<00:01, 62.85batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:01, 60.08batch/s]

Training batches on cuda:  51%|█████▏    | 72/140 [00:01<00:01, 59.64batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:01<00:01, 59.26batch/s]

Training batches on cuda:  61%|██████    | 85/140 [00:01<00:00, 59.52batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:01<00:00, 59.94batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:01<00:00, 62.21batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:01<00:00, 63.95batch/s]

Training batches on cuda:  81%|████████  | 113/140 [00:01<00:00, 64.64batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:01<00:00, 65.07batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:02<00:00, 66.20batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:02<00:00, 65.70batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:09,  2.40s/epoch, loss=0.000455, prev_loss=0.000454]
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:07,  2.43s/epoch, loss=0.000455, prev_loss=0.000454]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/140 [00:00<00:13,  9.98batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:10, 13.21batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:07, 17.27batch/s]

Training batches on cuda:  15%|█▌        | 21/140 [00:00<00:05, 22.00batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:04, 27.48batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:00<00:03, 33.33batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:00<00:02, 39.36batch/s]

Training batches on cuda:  35%|███▌      | 49/140 [00:00<00:02, 44.78batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:00<00:01, 49.39batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:01, 48.24batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 52.31batch/s]

Training batches on cuda:  55%|█████▌    | 77/140 [00:01<00:01, 55.31batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:01<00:00, 57.19batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:01<00:00, 59.23batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:01<00:00, 60.29batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:01<00:00, 60.88batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:01<00:00, 61.57batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:02<00:00, 62.76batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:02<00:00, 62.20batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:02<00:00, 62.76batch/s]

Training batches on cuda: 100%|██████████| 140/140 [00:02<00:00, 64.19batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:19<00:07,  2.43s/epoch, loss=0.000455, prev_loss=0.000455]
Training epochs on cuda:  80%|████████  | 8/10 [00:19<00:04,  2.46s/epoch, loss=0.000455, prev_loss=0.000455]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/140 [00:00<00:03, 39.81batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:02, 45.29batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:02, 50.12batch/s]

Training batches on cuda:  18%|█▊        | 25/140 [00:00<00:02, 53.56batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:01, 56.77batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:00<00:01, 59.12batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:00<00:01, 59.66batch/s]

Training batches on cuda:  38%|███▊      | 53/140 [00:00<00:01, 61.59batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:00<00:01, 63.15batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:01, 63.58batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:01<00:01, 64.31batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:01<00:00, 65.32batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:01<00:00, 66.01batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:01<00:00, 66.23batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:01<00:00, 66.77batch/s]

Training batches on cuda:  78%|███████▊  | 109/140 [00:01<00:00, 67.01batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:01<00:00, 67.10batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:01<00:00, 67.61batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:01<00:00, 68.02batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:02<00:00, 68.47batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:21<00:04,  2.46s/epoch, loss=0.000455, prev_loss=0.000455]
Training epochs on cuda:  90%|█████████ | 9/10 [00:21<00:02,  2.42s/epoch, loss=0.000455, prev_loss=0.000455]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 5/140 [00:00<00:03, 44.24batch/s]

Training batches on cuda:   9%|▊         | 12/140 [00:00<00:02, 49.59batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:02, 54.38batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:01, 58.25batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:01, 61.07batch/s]

Training batches on cuda:  29%|██▉       | 41/140 [00:00<00:01, 63.19batch/s]

Training batches on cuda:  34%|███▍      | 48/140 [00:00<00:01, 64.55batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:00<00:01, 65.55batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:00<00:01, 66.32batch/s]

Training batches on cuda:  49%|████▉     | 69/140 [00:01<00:01, 67.36batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:00, 67.47batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:01<00:00, 60.86batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:01<00:00, 63.32batch/s]

Training batches on cuda:  69%|██████▉   | 97/140 [00:01<00:00, 64.76batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:01<00:00, 65.56batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:01<00:00, 64.01batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:01<00:00, 63.15batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:01<00:00, 64.76batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:02<00:00, 65.82batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:24<00:02,  2.42s/epoch, loss=0.000455, prev_loss=0.000455]2021-09-16:18:14:31,798 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:14:32,828 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:14:47,721 INFO     [evaluator.py:657] Evaluation took 14.89s seconds
2021-09-16:18:14:47,857 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00,  7.21s/epoch, loss=0.000455, prev_loss=0.000455]
Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00,  4.03s/epoch, loss=0.000455, prev_loss=0.000455]
2021-09-16:18:14:47,858 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpxgw2z1sg'
2021-09-16:18:14:47,938 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpxgw2z1sg' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.52ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.65ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
2021-09-16:18:15:02,687 INFO     [evaluator.py:657] Evaluation took 14.73s seconds
[I 2021-09-16 18:15:02,692] Trial 37 finished with value: 0.002182967835189231 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.23176718449060174, 'optimizer.lr': 0.03784885406003079, 'negative_sampler.num_negs_per_pos': 7, 'training.batch_size': 2232}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:15:02,715 WARNING  [api.py:823] No random seed is specified. Setting to 2069384859.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   6%|▌         | 5/88 [00:00<00:01, 43.31batch/s]

Training batches on cuda:  12%|█▎        | 11/88 [00:00<00:01, 46.85batch/s]

Training batches on cuda:  19%|█▉        | 17/88 [00:00<00:01, 49.62batch/s]

Training batches on cuda:  26%|██▌       | 23/88 [00:00<00:01, 51.61batch/s]

Training batches on cuda:  33%|███▎      | 29/88 [00:00<00:01, 53.70batch/s]

Training batches on cuda:  40%|███▉      | 35/88 [00:00<00:00, 55.21batch/s]

Training batches on cuda:  47%|████▋     | 41/88 [00:00<00:00, 55.55batch/s]

Training batches on cuda:  53%|█████▎    | 47/88 [00:00<00:00, 56.36batch/s]

Training batches on cuda:  60%|██████    | 53/88 [00:00<00:00, 56.78batch/s]

Training batches on cuda:  67%|██████▋   | 59/88 [00:01<00:00, 56.25batch/s]

Training batches on cuda:  74%|███████▍  | 65/88 [00:01<00:00, 56.35batch/s]

Training batches on cuda:  81%|████████  | 71/88 [00:01<00:00, 56.82batch/s]

Training batches on cuda:  88%|████████▊ | 77/88 [00:01<00:00, 56.88batch/s]

Training batches on cuda:  94%|█████████▍| 83/88 [00:01<00:00, 56.69batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:01<?, ?epoch/s, loss=0.000204, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:01<00:15,  1.75s/epoch, loss=0.000204, prev_loss=nan]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/88 [00:00<00:03, 26.02batch/s]

Training batches on cuda:  10%|█         | 9/88 [00:00<00:02, 31.34batch/s]

Training batches on cuda:  17%|█▋        | 15/88 [00:00<00:02, 36.35batch/s]

Training batches on cuda:  24%|██▍       | 21/88 [00:00<00:01, 40.82batch/s]

Training batches on cuda:  32%|███▏      | 28/88 [00:00<00:01, 45.14batch/s]

Training batches on cuda:  39%|███▊      | 34/88 [00:00<00:01, 48.33batch/s]

Training batches on cuda:  45%|████▌     | 40/88 [00:00<00:00, 50.23batch/s]

Training batches on cuda:  52%|█████▏    | 46/88 [00:00<00:00, 52.43batch/s]

Training batches on cuda:  59%|█████▉    | 52/88 [00:00<00:00, 54.05batch/s]

Training batches on cuda:  66%|██████▌   | 58/88 [00:01<00:00, 54.98batch/s]

Training batches on cuda:  73%|███████▎  | 64/88 [00:01<00:00, 49.99batch/s]

Training batches on cuda:  80%|███████▉  | 70/88 [00:01<00:00, 51.76batch/s]

Training batches on cuda:  86%|████████▋ | 76/88 [00:01<00:00, 52.50batch/s]

Training batches on cuda:  93%|█████████▎| 82/88 [00:01<00:00, 53.63batch/s]

Training batches on cuda: 100%|██████████| 88/88 [00:01<00:00, 54.26batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:15,  1.75s/epoch, loss=0.000198, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:03<00:14,  1.77s/epoch, loss=0.000198, prev_loss=nan]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▍         | 4/88 [00:00<00:02, 38.65batch/s]

Training batches on cuda:  11%|█▏        | 10/88 [00:00<00:01, 42.69batch/s]

Training batches on cuda:  18%|█▊        | 16/88 [00:00<00:01, 45.46batch/s]

Training batches on cuda:  25%|██▌       | 22/88 [00:00<00:01, 48.61batch/s]

Training batches on cuda:  32%|███▏      | 28/88 [00:00<00:01, 50.49batch/s]

Training batches on cuda:  39%|███▊      | 34/88 [00:00<00:01, 51.66batch/s]

Training batches on cuda:  45%|████▌     | 40/88 [00:00<00:00, 52.42batch/s]

Training batches on cuda:  52%|█████▏    | 46/88 [00:00<00:00, 53.92batch/s]

Training batches on cuda:  59%|█████▉    | 52/88 [00:00<00:00, 53.90batch/s]

Training batches on cuda:  66%|██████▌   | 58/88 [00:01<00:00, 53.86batch/s]

Training batches on cuda:  73%|███████▎  | 64/88 [00:01<00:00, 54.76batch/s]

Training batches on cuda:  80%|███████▉  | 70/88 [00:01<00:00, 55.48batch/s]

Training batches on cuda:  86%|████████▋ | 76/88 [00:01<00:00, 54.99batch/s]

Training batches on cuda:  93%|█████████▎| 82/88 [00:01<00:00, 54.85batch/s]

Training batches on cuda: 100%|██████████| 88/88 [00:01<00:00, 54.25batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:14,  1.77s/epoch, loss=0.000198, prev_loss=0.000198]
Training epochs on cuda:  30%|███       | 3/10 [00:05<00:12,  1.79s/epoch, loss=0.000198, prev_loss=0.000198]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/88 [00:00<00:03, 28.04batch/s]

Training batches on cuda:   9%|▉         | 8/88 [00:00<00:02, 32.29batch/s]

Training batches on cuda:  16%|█▌        | 14/88 [00:00<00:02, 35.77batch/s]

Training batches on cuda:  23%|██▎       | 20/88 [00:00<00:01, 39.17batch/s]

Training batches on cuda:  28%|██▊       | 25/88 [00:00<00:01, 41.80batch/s]

Training batches on cuda:  34%|███▍      | 30/88 [00:00<00:01, 43.93batch/s]

Training batches on cuda:  41%|████      | 36/88 [00:00<00:01, 46.40batch/s]

Training batches on cuda:  48%|████▊     | 42/88 [00:00<00:00, 47.26batch/s]

Training batches on cuda:  53%|█████▎    | 47/88 [00:01<00:00, 44.53batch/s]

Training batches on cuda:  59%|█████▉    | 52/88 [00:01<00:00, 46.04batch/s]

Training batches on cuda:  66%|██████▌   | 58/88 [00:01<00:00, 46.84batch/s]

Training batches on cuda:  72%|███████▏  | 63/88 [00:01<00:00, 46.18batch/s]

Training batches on cuda:  77%|███████▋  | 68/88 [00:01<00:00, 45.01batch/s]

Training batches on cuda:  83%|████████▎ | 73/88 [00:01<00:00, 42.02batch/s]

Training batches on cuda:  89%|████████▊ | 78/88 [00:01<00:00, 43.49batch/s]

Training batches on cuda:  94%|█████████▍| 83/88 [00:01<00:00, 41.81batch/s]

Training batches on cuda: 100%|██████████| 88/88 [00:01<00:00, 41.94batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:12,  1.79s/epoch, loss=0.000198, prev_loss=0.000198]
Training epochs on cuda:  40%|████      | 4/10 [00:07<00:11,  1.92s/epoch, loss=0.000198, prev_loss=0.000198]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/88 [00:00<00:02, 28.34batch/s]

Training batches on cuda:   9%|▉         | 8/88 [00:00<00:02, 31.69batch/s]

Training batches on cuda:  14%|█▎        | 12/88 [00:00<00:02, 32.19batch/s]

Training batches on cuda:  20%|██        | 18/88 [00:00<00:01, 36.13batch/s]

Training batches on cuda:  27%|██▋       | 24/88 [00:00<00:01, 40.05batch/s]

Training batches on cuda:  34%|███▍      | 30/88 [00:00<00:01, 43.99batch/s]

Training batches on cuda:  41%|████      | 36/88 [00:00<00:01, 47.23batch/s]

Training batches on cuda:  48%|████▊     | 42/88 [00:00<00:00, 49.70batch/s]

Training batches on cuda:  55%|█████▍    | 48/88 [00:00<00:00, 51.53batch/s]

Training batches on cuda:  61%|██████▏   | 54/88 [00:01<00:00, 53.12batch/s]

Training batches on cuda:  68%|██████▊   | 60/88 [00:01<00:00, 54.75batch/s]

Training batches on cuda:  75%|███████▌  | 66/88 [00:01<00:00, 55.83batch/s]

Training batches on cuda:  82%|████████▏ | 72/88 [00:01<00:00, 55.12batch/s]

Training batches on cuda:  89%|████████▊ | 78/88 [00:01<00:00, 54.83batch/s]

Training batches on cuda:  95%|█████████▌| 84/88 [00:01<00:00, 55.88batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:11,  1.92s/epoch, loss=0.000198, prev_loss=0.000198]
Training epochs on cuda:  50%|█████     | 5/10 [00:09<00:09,  1.92s/epoch, loss=0.000198, prev_loss=0.000198]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▍         | 4/88 [00:00<00:02, 39.62batch/s]

Training batches on cuda:  11%|█▏        | 10/88 [00:00<00:01, 43.72batch/s]

Training batches on cuda:  18%|█▊        | 16/88 [00:00<00:01, 46.94batch/s]

Training batches on cuda:  25%|██▌       | 22/88 [00:00<00:01, 49.90batch/s]

Training batches on cuda:  32%|███▏      | 28/88 [00:00<00:01, 51.75batch/s]

Training batches on cuda:  39%|███▊      | 34/88 [00:00<00:01, 51.28batch/s]

Training batches on cuda:  45%|████▌     | 40/88 [00:00<00:00, 51.20batch/s]

Training batches on cuda:  52%|█████▏    | 46/88 [00:00<00:00, 50.89batch/s]

Training batches on cuda:  59%|█████▉    | 52/88 [00:00<00:00, 50.78batch/s]

Training batches on cuda:  66%|██████▌   | 58/88 [00:01<00:00, 50.66batch/s]

Training batches on cuda:  73%|███████▎  | 64/88 [00:01<00:00, 52.21batch/s]

Training batches on cuda:  80%|███████▉  | 70/88 [00:01<00:00, 53.60batch/s]

Training batches on cuda:  86%|████████▋ | 76/88 [00:01<00:00, 54.83batch/s]

Training batches on cuda:  93%|█████████▎| 82/88 [00:01<00:00, 55.48batch/s]

Training batches on cuda: 100%|██████████| 88/88 [00:01<00:00, 55.44batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:09,  1.92s/epoch, loss=0.000198, prev_loss=0.000198]
Training epochs on cuda:  60%|██████    | 6/10 [00:11<00:07,  1.89s/epoch, loss=0.000198, prev_loss=0.000198]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▍         | 4/88 [00:00<00:02, 37.67batch/s]

Training batches on cuda:  11%|█▏        | 10/88 [00:00<00:01, 41.74batch/s]

Training batches on cuda:  17%|█▋        | 15/88 [00:00<00:01, 42.34batch/s]

Training batches on cuda:  23%|██▎       | 20/88 [00:00<00:01, 43.38batch/s]

Training batches on cuda:  28%|██▊       | 25/88 [00:00<00:01, 44.58batch/s]

Training batches on cuda:  34%|███▍      | 30/88 [00:00<00:01, 46.05batch/s]

Training batches on cuda:  41%|████      | 36/88 [00:00<00:01, 48.02batch/s]

Training batches on cuda:  48%|████▊     | 42/88 [00:00<00:00, 50.00batch/s]

Training batches on cuda:  55%|█████▍    | 48/88 [00:00<00:00, 51.59batch/s]

Training batches on cuda:  61%|██████▏   | 54/88 [00:01<00:00, 52.54batch/s]

Training batches on cuda:  68%|██████▊   | 60/88 [00:01<00:00, 52.49batch/s]

Training batches on cuda:  75%|███████▌  | 66/88 [00:01<00:00, 52.41batch/s]

Training batches on cuda:  82%|████████▏ | 72/88 [00:01<00:00, 50.62batch/s]

Training batches on cuda:  89%|████████▊ | 78/88 [00:01<00:00, 49.44batch/s]

Training batches on cuda:  94%|█████████▍| 83/88 [00:01<00:00, 48.05batch/s]

Training batches on cuda: 100%|██████████| 88/88 [00:01<00:00, 47.08batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:07,  1.89s/epoch, loss=0.000198, prev_loss=0.000198]
Training epochs on cuda:  70%|███████   | 7/10 [00:13<00:05,  1.92s/epoch, loss=0.000198, prev_loss=0.000198]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/88 [00:00<00:02, 29.57batch/s]

Training batches on cuda:   9%|▉         | 8/88 [00:00<00:02, 32.92batch/s]

Training batches on cuda:  14%|█▎        | 12/88 [00:00<00:02, 34.66batch/s]

Training batches on cuda:  19%|█▉        | 17/88 [00:00<00:01, 37.86batch/s]

Training batches on cuda:  25%|██▌       | 22/88 [00:00<00:01, 40.35batch/s]

Training batches on cuda:  32%|███▏      | 28/88 [00:00<00:01, 44.34batch/s]

Training batches on cuda:  39%|███▊      | 34/88 [00:00<00:01, 46.99batch/s]

Training batches on cuda:  45%|████▌     | 40/88 [00:00<00:00, 49.41batch/s]

Training batches on cuda:  52%|█████▏    | 46/88 [00:00<00:00, 51.81batch/s]

Training batches on cuda:  59%|█████▉    | 52/88 [00:01<00:00, 53.18batch/s]

Training batches on cuda:  66%|██████▌   | 58/88 [00:01<00:00, 53.05batch/s]

Training batches on cuda:  73%|███████▎  | 64/88 [00:01<00:00, 54.50batch/s]

Training batches on cuda:  80%|███████▉  | 70/88 [00:01<00:00, 54.96batch/s]

Training batches on cuda:  86%|████████▋ | 76/88 [00:01<00:00, 55.17batch/s]

Training batches on cuda:  93%|█████████▎| 82/88 [00:01<00:00, 56.04batch/s]

Training batches on cuda: 100%|██████████| 88/88 [00:01<00:00, 55.61batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [00:15<00:05,  1.92s/epoch, loss=0.000198, prev_loss=0.000198]
Training epochs on cuda:  80%|████████  | 8/10 [00:15<00:03,  1.92s/epoch, loss=0.000198, prev_loss=0.000198]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/88 [00:00<00:03, 26.61batch/s]

Training batches on cuda:   8%|▊         | 7/88 [00:00<00:02, 29.00batch/s]

Training batches on cuda:  14%|█▎        | 12/88 [00:00<00:02, 32.48batch/s]

Training batches on cuda:  20%|██        | 18/88 [00:00<00:01, 37.50batch/s]

Training batches on cuda:  27%|██▋       | 24/88 [00:00<00:01, 40.68batch/s]

Training batches on cuda:  34%|███▍      | 30/88 [00:00<00:01, 43.25batch/s]

Training batches on cuda:  41%|████      | 36/88 [00:00<00:01, 46.27batch/s]

Training batches on cuda:  48%|████▊     | 42/88 [00:00<00:00, 49.11batch/s]

Training batches on cuda:  53%|█████▎    | 47/88 [00:01<00:00, 46.62batch/s]

Training batches on cuda:  59%|█████▉    | 52/88 [00:01<00:00, 45.40batch/s]

Training batches on cuda:  65%|██████▍   | 57/88 [00:01<00:00, 44.58batch/s]

Training batches on cuda:  70%|███████   | 62/88 [00:01<00:00, 43.31batch/s]

Training batches on cuda:  76%|███████▌  | 67/88 [00:01<00:00, 42.17batch/s]

Training batches on cuda:  82%|████████▏ | 72/88 [00:01<00:00, 41.80batch/s]

Training batches on cuda:  88%|████████▊ | 77/88 [00:01<00:00, 42.38batch/s]

Training batches on cuda:  93%|█████████▎| 82/88 [00:01<00:00, 44.04batch/s]

Training batches on cuda:  99%|█████████▉| 87/88 [00:01<00:00, 45.05batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [00:17<00:03,  1.92s/epoch, loss=0.000198, prev_loss=0.000198]
Training epochs on cuda:  90%|█████████ | 9/10 [00:17<00:01,  1.99s/epoch, loss=0.000198, prev_loss=0.000198]

Training batches on cuda:   0%|          | 0/88 [00:00<?, ?batch/s]

Training batches on cuda:   5%|▍         | 4/88 [00:00<00:02, 36.25batch/s]

Training batches on cuda:  11%|█▏        | 10/88 [00:00<00:01, 39.57batch/s]

Training batches on cuda:  18%|█▊        | 16/88 [00:00<00:01, 43.52batch/s]

Training batches on cuda:  25%|██▌       | 22/88 [00:00<00:01, 45.89batch/s]

Training batches on cuda:  32%|███▏      | 28/88 [00:00<00:01, 48.43batch/s]

Training batches on cuda:  38%|███▊      | 33/88 [00:00<00:01, 45.08batch/s]

Training batches on cuda:  43%|████▎     | 38/88 [00:00<00:01, 46.36batch/s]

Training batches on cuda:  49%|████▉     | 43/88 [00:00<00:01, 44.90batch/s]

Training batches on cuda:  55%|█████▍    | 48/88 [00:01<00:00, 45.55batch/s]

Training batches on cuda:  60%|██████    | 53/88 [00:01<00:00, 46.78batch/s]

Training batches on cuda:  66%|██████▌   | 58/88 [00:01<00:00, 46.72batch/s]

Training batches on cuda:  73%|███████▎  | 64/88 [00:01<00:00, 49.46batch/s]

Training batches on cuda:  80%|███████▉  | 70/88 [00:01<00:00, 51.15batch/s]

Training batches on cuda:  86%|████████▋ | 76/88 [00:01<00:00, 50.60batch/s]

Training batches on cuda:  93%|█████████▎| 82/88 [00:01<00:00, 50.52batch/s]

Training batches on cuda: 100%|██████████| 88/88 [00:01<00:00, 50.10batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [00:19<00:01,  1.99s/epoch, loss=0.000198, prev_loss=0.000198]2021-09-16:18:15:22,554 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:15:23,516 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:15:38,494 INFO     [evaluator.py:657] Evaluation took 14.98s seconds
2021-09-16:18:15:38,639 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:35<00:00,  6.81s/epoch, loss=0.000198, prev_loss=0.000198]
Training epochs on cuda: 100%|██████████| 10/10 [00:35<00:00,  3.55s/epoch, loss=0.000198, prev_loss=0.000198]
2021-09-16:18:15:38,639 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp6yot038t'
2021-09-16:18:15:38,708 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6yot038t' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.66ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.66ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.66ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.66ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.66ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:06, 2.54ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.57ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.60ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.66ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.66ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:18:15:53,496 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 18:15:53,501] Trial 38 finished with value: -0.0006768879774121661 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.14564575431045393, 'optimizer.lr': 0.04547063995860642, 'negative_sampler.num_negs_per_pos': 2, 'training.batch_size': 3532}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:15:53,521 WARNING  [api.py:823] No random seed is specified. Setting to 1062933755.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 17.44batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 19.65batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 21.60batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 23.36batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.65batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 25.35batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 26.03batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.48batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 26.72batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 27.17batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:02, 27.88batch/s]

Training batches on cuda:  32%|███▏      | 36/114 [00:01<00:02, 28.69batch/s]

Training batches on cuda:  35%|███▌      | 40/114 [00:01<00:02, 29.32batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 29.75batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 29.81batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 29.79batch/s]

Training batches on cuda:  47%|████▋     | 54/114 [00:01<00:02, 29.98batch/s]

Training batches on cuda:  51%|█████     | 58/114 [00:02<00:01, 30.07batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 30.23batch/s]

Training batches on cuda:  58%|█████▊    | 66/114 [00:02<00:01, 30.31batch/s]

Training batches on cuda:  61%|██████▏   | 70/114 [00:02<00:01, 30.34batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 30.17batch/s]

Training batches on cuda:  68%|██████▊   | 78/114 [00:02<00:01, 29.89batch/s]

Training batches on cuda:  72%|███████▏  | 82/114 [00:02<00:01, 28.77batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:02<00:00, 29.21batch/s]

Training batches on cuda:  79%|███████▉  | 90/114 [00:03<00:00, 29.57batch/s]

Training batches on cuda:  82%|████████▏ | 94/114 [00:03<00:00, 29.89batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 30.18batch/s]

Training batches on cuda:  89%|████████▉ | 102/114 [00:03<00:00, 29.12batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:03<00:00, 29.15batch/s]

Training batches on cuda:  95%|█████████▍| 108/114 [00:03<00:00, 29.27batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:03<00:00, 29.43batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000483, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:37,  4.14s/epoch, loss=0.000483, prev_loss=nan]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:05, 19.70batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 21.55batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 23.40batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 24.85batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:03, 25.54batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 25.68batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 26.01batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.19batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 26.66batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 27.05batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.62batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.92batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.76batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 26.98batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 26.65batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 27.01batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.10batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 26.87batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 26.87batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 26.71batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.67batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.90batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.90batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.10batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.00batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.38batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:02<00:01, 27.49batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.21batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.51batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.34batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.63batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 27.83batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 27.81batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 27.74batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 27.76batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 27.72batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.92batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 27.82batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:37,  4.14s/epoch, loss=0.000481, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:33,  4.23s/epoch, loss=0.000481, prev_loss=nan]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:07, 15.77batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:06, 18.01batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 19.80batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 21.85batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 23.28batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.54batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.68batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.28batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 26.47batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 27.00batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 27.28batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 26.31batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.90batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.29batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.48batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 27.68batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.71batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 27.81batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 27.67batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 27.44batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 27.30batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 27.13batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 27.01batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.00batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.24batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.43batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:02<00:01, 27.35batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.31batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.33batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.34batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.37batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.56batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 26.18batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 26.79batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 26.01batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 26.28batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 26.03batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 25.76batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:33,  4.23s/epoch, loss=0.000475, prev_loss=0.000481]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:30,  4.31s/epoch, loss=0.000475, prev_loss=0.000481]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 16.18batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 18.57batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 20.68batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.44batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 21.92batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:04, 22.97batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 23.75batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 24.33batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 24.96batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 24.87batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 25.70batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 25.48batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.26batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 26.88batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.26batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 26.46batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.03batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 27.39batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 27.29batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 27.39batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 27.62batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 27.88batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 27.84batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.68batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.15batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.25batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 27.43batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.11batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.01batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.01batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 26.85batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.48batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 26.38batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 26.36batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 26.35batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 26.24batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 26.29batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 25.72batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:30,  4.31s/epoch, loss=0.000464, prev_loss=0.000475]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:26,  4.38s/epoch, loss=0.000464, prev_loss=0.000475]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 17.97batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 19.88batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 21.67batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.39batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 23.77batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:04, 24.12batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.25batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.68batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 23.74batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 25.01batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 25.67batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 26.27batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.89batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.28batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.56batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 27.44batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.77batch/s]

Training batches on cuda:  47%|████▋     | 54/114 [00:02<00:02, 28.43batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:02<00:01, 28.67batch/s]

Training batches on cuda:  53%|█████▎    | 60/114 [00:02<00:01, 28.07batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:02<00:01, 27.30batch/s]

Training batches on cuda:  58%|█████▊    | 66/114 [00:02<00:01, 26.61batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:02<00:01, 26.17batch/s]

Training batches on cuda:  63%|██████▎   | 72/114 [00:02<00:01, 25.88batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:02<00:01, 25.77batch/s]

Training batches on cuda:  68%|██████▊   | 78/114 [00:02<00:01, 25.65batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:03<00:01, 25.41batch/s]

Training batches on cuda:  74%|███████▎  | 84/114 [00:03<00:01, 25.89batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:03<00:01, 26.66batch/s]

Training batches on cuda:  79%|███████▉  | 90/114 [00:03<00:00, 27.22batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:03<00:00, 27.23batch/s]

Training batches on cuda:  84%|████████▍ | 96/114 [00:03<00:00, 26.72batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:03<00:00, 27.11batch/s]

Training batches on cuda:  89%|████████▉ | 102/114 [00:03<00:00, 27.49batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:03<00:00, 27.45batch/s]

Training batches on cuda:  95%|█████████▍| 108/114 [00:04<00:00, 26.44batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:04<00:00, 26.25batch/s]

Training batches on cuda: 100%|██████████| 114/114 [00:04<00:00, 25.79batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:26,  4.38s/epoch, loss=0.000401, prev_loss=0.000464]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:22,  4.45s/epoch, loss=0.000401, prev_loss=0.000464]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:11,  9.88batch/s]

Training batches on cuda:   4%|▎         | 4/114 [00:00<00:09, 12.11batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:07, 14.51batch/s]

Training batches on cuda:   9%|▉         | 10/114 [00:00<00:06, 16.80batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:00<00:05, 18.87batch/s]

Training batches on cuda:  14%|█▍        | 16/114 [00:00<00:04, 20.72batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:00<00:04, 21.82batch/s]

Training batches on cuda:  19%|█▉        | 22/114 [00:00<00:03, 23.02batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:01<00:03, 24.31batch/s]

Training batches on cuda:  25%|██▍       | 28/114 [00:01<00:03, 24.62batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:01<00:03, 25.79batch/s]

Training batches on cuda:  30%|██▉       | 34/114 [00:01<00:03, 26.47batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:01<00:03, 23.88batch/s]

Training batches on cuda:  35%|███▌      | 40/114 [00:01<00:03, 24.44batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:01<00:02, 25.35batch/s]

Training batches on cuda:  40%|████      | 46/114 [00:01<00:02, 26.18batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:01<00:02, 26.08batch/s]

Training batches on cuda:  46%|████▌     | 52/114 [00:02<00:02, 26.44batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:02<00:02, 26.10batch/s]

Training batches on cuda:  51%|█████     | 58/114 [00:02<00:02, 26.42batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:02<00:02, 26.33batch/s]

Training batches on cuda:  56%|█████▌    | 64/114 [00:02<00:01, 26.58batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:02<00:01, 27.06batch/s]

Training batches on cuda:  61%|██████▏   | 70/114 [00:02<00:01, 27.74batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:02<00:01, 28.20batch/s]

Training batches on cuda:  67%|██████▋   | 76/114 [00:02<00:01, 28.19batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:03<00:01, 28.31batch/s]

Training batches on cuda:  72%|███████▏  | 82/114 [00:03<00:01, 28.67batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:03<00:01, 28.67batch/s]

Training batches on cuda:  77%|███████▋  | 88/114 [00:03<00:00, 28.31batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:03<00:00, 27.58batch/s]

Training batches on cuda:  82%|████████▏ | 94/114 [00:03<00:00, 27.38batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:03<00:00, 27.00batch/s]

Training batches on cuda:  88%|████████▊ | 100/114 [00:03<00:00, 26.21batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:03<00:00, 26.73batch/s]

Training batches on cuda:  93%|█████████▎| 106/114 [00:04<00:00, 26.70batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:04<00:00, 26.76batch/s]

Training batches on cuda:  98%|█████████▊| 112/114 [00:04<00:00, 26.85batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:26<00:22,  4.45s/epoch, loss=0.00032, prev_loss=0.000401] 
Training epochs on cuda:  60%|██████    | 6/10 [00:26<00:17,  4.46s/epoch, loss=0.00032, prev_loss=0.000401]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 16.35batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 18.69batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 20.76batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.71batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.21batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 25.48batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 26.34batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.63batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 26.75batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.99batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.90batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 27.18batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 27.46batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.73batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.98batch/s]

Training batches on cuda:  42%|████▏     | 48/114 [00:01<00:02, 28.64batch/s]

Training batches on cuda:  46%|████▌     | 52/114 [00:01<00:02, 28.33batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:02<00:02, 28.11batch/s]

Training batches on cuda:  51%|█████     | 58/114 [00:02<00:02, 27.94batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:02<00:01, 27.97batch/s]

Training batches on cuda:  56%|█████▌    | 64/114 [00:02<00:01, 27.80batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:02<00:01, 28.07batch/s]

Training batches on cuda:  61%|██████▏   | 70/114 [00:02<00:01, 28.52batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:02<00:01, 28.78batch/s]

Training batches on cuda:  67%|██████▋   | 76/114 [00:02<00:01, 28.99batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:02<00:01, 29.14batch/s]

Training batches on cuda:  72%|███████▏  | 82/114 [00:02<00:01, 28.98batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:03<00:01, 28.73batch/s]

Training batches on cuda:  77%|███████▋  | 88/114 [00:03<00:00, 28.64batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:03<00:00, 28.54batch/s]

Training batches on cuda:  82%|████████▏ | 94/114 [00:03<00:00, 28.46batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:03<00:00, 28.31batch/s]

Training batches on cuda:  88%|████████▊ | 100/114 [00:03<00:00, 28.22batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:03<00:00, 27.70batch/s]

Training batches on cuda:  93%|█████████▎| 106/114 [00:03<00:00, 27.21batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:03<00:00, 26.97batch/s]

Training batches on cuda:  98%|█████████▊| 112/114 [00:04<00:00, 27.46batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:31<00:17,  4.46s/epoch, loss=0.000285, prev_loss=0.00032]
Training epochs on cuda:  70%|███████   | 7/10 [00:31<00:13,  4.42s/epoch, loss=0.000285, prev_loss=0.00032]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 16.76batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 19.00batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 21.06batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.71batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.20batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 25.23batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 26.00batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.70batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 27.01batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 27.39batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.89batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 27.34batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 27.65batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.85batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 28.05batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 28.16batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 28.28batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 28.25batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 27.92batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 27.10batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.80batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 27.18batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 27.22batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.51batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.12batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.47batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:02<00:01, 27.64batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.83batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.27batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.50batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.76batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 27.98batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 28.13batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 28.18batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 28.20batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 28.21batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:03<00:00, 28.24batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 27.93batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:13,  4.42s/epoch, loss=0.000258, prev_loss=0.000285]
Training epochs on cuda:  80%|████████  | 8/10 [00:35<00:08,  4.40s/epoch, loss=0.000258, prev_loss=0.000285]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:04, 22.31batch/s]

Training batches on cuda:   5%|▌         | 6/114 [00:00<00:04, 23.48batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:04, 24.67batch/s]

Training batches on cuda:  11%|█         | 12/114 [00:00<00:03, 25.58batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:00<00:03, 24.75batch/s]

Training batches on cuda:  16%|█▌        | 18/114 [00:00<00:03, 25.62batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:00<00:03, 25.77batch/s]

Training batches on cuda:  21%|██        | 24/114 [00:00<00:03, 26.31batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:01<00:03, 26.71batch/s]

Training batches on cuda:  26%|██▋       | 30/114 [00:01<00:03, 26.69batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:01<00:03, 26.84batch/s]

Training batches on cuda:  32%|███▏      | 36/114 [00:01<00:02, 26.95batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:01<00:02, 27.01batch/s]

Training batches on cuda:  37%|███▋      | 42/114 [00:01<00:02, 27.20batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:01<00:02, 27.30batch/s]

Training batches on cuda:  42%|████▏     | 48/114 [00:01<00:02, 26.64batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:01<00:02, 26.89batch/s]

Training batches on cuda:  47%|████▋     | 54/114 [00:02<00:02, 27.05batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:02<00:02, 27.23batch/s]

Training batches on cuda:  53%|█████▎    | 60/114 [00:02<00:01, 27.36batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:02<00:01, 27.46batch/s]

Training batches on cuda:  58%|█████▊    | 66/114 [00:02<00:01, 27.59batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:02<00:01, 27.76batch/s]

Training batches on cuda:  63%|██████▎   | 72/114 [00:02<00:01, 27.79batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:02<00:01, 27.79batch/s]

Training batches on cuda:  68%|██████▊   | 78/114 [00:02<00:01, 27.79batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:03<00:01, 27.75batch/s]

Training batches on cuda:  74%|███████▎  | 84/114 [00:03<00:01, 27.60batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:03<00:01, 26.80batch/s]

Training batches on cuda:  79%|███████▉  | 90/114 [00:03<00:00, 27.14batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:03<00:00, 27.30batch/s]

Training batches on cuda:  84%|████████▍ | 96/114 [00:03<00:00, 27.44batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:03<00:00, 27.27batch/s]

Training batches on cuda:  89%|████████▉ | 102/114 [00:03<00:00, 26.55batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:03<00:00, 26.10batch/s]

Training batches on cuda:  95%|█████████▍| 108/114 [00:04<00:00, 26.63batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:04<00:00, 26.98batch/s]

Training batches on cuda: 100%|██████████| 114/114 [00:04<00:00, 27.15batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:39<00:08,  4.40s/epoch, loss=0.000239, prev_loss=0.000258]
Training epochs on cuda:  90%|█████████ | 9/10 [00:39<00:04,  4.42s/epoch, loss=0.000239, prev_loss=0.000258]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 17.51batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 19.24batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 20.99batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.68batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.01batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 25.07batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.85batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.48batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 26.62batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.59batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.83batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.68batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 27.09batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.14batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.26batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 26.89batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 26.67batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 26.42batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 26.51batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 26.98batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 27.45batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 27.38batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 27.48batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.67batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.80batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.74batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:02<00:01, 27.82batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.89batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.92batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 28.00batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.96batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 27.84batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 27.86batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 27.82batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 27.79batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 27.84batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.92batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 27.88batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:44<00:04,  4.42s/epoch, loss=0.00023, prev_loss=0.000239] 2021-09-16:18:16:38,360 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:16:39,400 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:16:54,384 INFO     [evaluator.py:657] Evaluation took 14.98s seconds
2021-09-16:18:16:54,562 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00,  9.30s/epoch, loss=0.00023, prev_loss=0.000239]
Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00,  6.05s/epoch, loss=0.00023, prev_loss=0.000239]
2021-09-16:18:16:54,563 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp6vrhc8fa'
2021-09-16:18:16:54,627 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp6vrhc8fa' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.52ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:18:17:09,451 INFO     [evaluator.py:657] Evaluation took 14.81s seconds
[I 2021-09-16 18:17:09,457] Trial 39 finished with value: 0.9411152490588105 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.01668886352668359, 'optimizer.lr': 0.07136433923689836, 'negative_sampler.num_negs_per_pos': 21, 'training.batch_size': 2732}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:17:09,479 WARNING  [api.py:823] No random seed is specified. Setting to 1687563882.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/153 [00:00<00:03, 43.28batch/s]

Training batches on cuda:   7%|▋         | 11/153 [00:00<00:03, 46.43batch/s]

Training batches on cuda:  11%|█         | 17/153 [00:00<00:02, 48.92batch/s]

Training batches on cuda:  15%|█▌        | 23/153 [00:00<00:02, 51.60batch/s]

Training batches on cuda:  19%|█▉        | 29/153 [00:00<00:02, 51.94batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:00<00:02, 49.86batch/s]

Training batches on cuda:  26%|██▌       | 40/153 [00:00<00:02, 51.43batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:00<00:02, 52.29batch/s]

Training batches on cuda:  34%|███▍      | 52/153 [00:00<00:01, 52.67batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:01, 53.15batch/s]

Training batches on cuda:  42%|████▏     | 64/153 [00:01<00:01, 52.63batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:01, 52.10batch/s]

Training batches on cuda:  50%|████▉     | 76/153 [00:01<00:01, 53.00batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 53.61batch/s]

Training batches on cuda:  58%|█████▊    | 88/153 [00:01<00:01, 54.02batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:01<00:01, 54.13batch/s]

Training batches on cuda:  65%|██████▌   | 100/153 [00:01<00:00, 54.13batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:01<00:00, 54.55batch/s]

Training batches on cuda:  73%|███████▎  | 112/153 [00:02<00:00, 54.47batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:02<00:00, 54.26batch/s]

Training batches on cuda:  81%|████████  | 124/153 [00:02<00:00, 51.64batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:02<00:00, 48.14batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:02<00:00, 48.23batch/s]

Training batches on cuda:  92%|█████████▏| 140/153 [00:02<00:00, 48.37batch/s]

Training batches on cuda:  95%|█████████▍| 145/153 [00:02<00:00, 48.45batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:02<00:00, 48.64batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000997, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:28,  3.15s/epoch, loss=0.000997, prev_loss=nan]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 28.13batch/s]

Training batches on cuda:   5%|▌         | 8/153 [00:00<00:04, 31.79batch/s]

Training batches on cuda:   8%|▊         | 13/153 [00:00<00:03, 35.49batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:03, 38.65batch/s]

Training batches on cuda:  15%|█▌        | 23/153 [00:00<00:03, 40.82batch/s]

Training batches on cuda:  18%|█▊        | 28/153 [00:00<00:02, 42.50batch/s]

Training batches on cuda:  22%|██▏       | 33/153 [00:00<00:02, 44.18batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:00<00:02, 45.32batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:00<00:02, 46.22batch/s]

Training batches on cuda:  31%|███▏      | 48/153 [00:01<00:02, 46.41batch/s]

Training batches on cuda:  35%|███▍      | 53/153 [00:01<00:02, 46.32batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:02, 46.77batch/s]

Training batches on cuda:  42%|████▏     | 64/153 [00:01<00:01, 48.75batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:01, 49.60batch/s]

Training batches on cuda:  50%|████▉     | 76/153 [00:01<00:01, 50.30batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 51.43batch/s]

Training batches on cuda:  58%|█████▊    | 88/153 [00:01<00:01, 52.34batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:01<00:01, 53.42batch/s]

Training batches on cuda:  65%|██████▌   | 100/153 [00:02<00:00, 54.15batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:02<00:00, 54.69batch/s]

Training batches on cuda:  73%|███████▎  | 112/153 [00:02<00:00, 54.97batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:02<00:00, 54.70batch/s]

Training batches on cuda:  81%|████████  | 124/153 [00:02<00:00, 54.73batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:02<00:00, 54.71batch/s]

Training batches on cuda:  89%|████████▉ | 136/153 [00:02<00:00, 54.20batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:02<00:00, 52.64batch/s]

Training batches on cuda:  97%|█████████▋| 148/153 [00:02<00:00, 50.48batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:28,  3.15s/epoch, loss=0.000987, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:25,  3.18s/epoch, loss=0.000987, prev_loss=nan]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 28.89batch/s]

Training batches on cuda:   5%|▌         | 8/153 [00:00<00:04, 32.85batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:03, 36.75batch/s]

Training batches on cuda:  13%|█▎        | 20/153 [00:00<00:03, 40.33batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:02, 43.52batch/s]

Training batches on cuda:  21%|██        | 32/153 [00:00<00:02, 45.33batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:00<00:02, 47.28batch/s]

Training batches on cuda:  29%|██▉       | 44/153 [00:00<00:02, 48.46batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:01<00:02, 49.60batch/s]

Training batches on cuda:  36%|███▌      | 55/153 [00:01<00:02, 48.94batch/s]

Training batches on cuda:  40%|███▉      | 61/153 [00:01<00:01, 49.54batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:01, 49.96batch/s]

Training batches on cuda:  48%|████▊     | 73/153 [00:01<00:01, 50.31batch/s]

Training batches on cuda:  52%|█████▏    | 79/153 [00:01<00:01, 50.61batch/s]

Training batches on cuda:  56%|█████▌    | 85/153 [00:01<00:01, 50.95batch/s]

Training batches on cuda:  59%|█████▉    | 91/153 [00:01<00:01, 52.63batch/s]

Training batches on cuda:  63%|██████▎   | 97/153 [00:01<00:01, 52.98batch/s]

Training batches on cuda:  67%|██████▋   | 103/153 [00:02<00:00, 53.18batch/s]

Training batches on cuda:  71%|███████   | 109/153 [00:02<00:00, 53.35batch/s]

Training batches on cuda:  75%|███████▌  | 115/153 [00:02<00:00, 53.44batch/s]

Training batches on cuda:  79%|███████▉  | 121/153 [00:02<00:00, 53.79batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:02<00:00, 53.56batch/s]

Training batches on cuda:  87%|████████▋ | 133/153 [00:02<00:00, 53.33batch/s]

Training batches on cuda:  91%|█████████ | 139/153 [00:02<00:00, 53.27batch/s]

Training batches on cuda:  95%|█████████▍| 145/153 [00:02<00:00, 53.62batch/s]

Training batches on cuda:  99%|█████████▊| 151/153 [00:02<00:00, 53.70batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:25,  3.18s/epoch, loss=0.000987, prev_loss=0.000987]
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:22,  3.17s/epoch, loss=0.000987, prev_loss=0.000987]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 36.85batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:03, 39.04batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:03, 38.87batch/s]

Training batches on cuda:  12%|█▏        | 19/153 [00:00<00:03, 41.06batch/s]

Training batches on cuda:  16%|█▌        | 24/153 [00:00<00:03, 41.85batch/s]

Training batches on cuda:  19%|█▉        | 29/153 [00:00<00:02, 42.61batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:00<00:02, 43.92batch/s]

Training batches on cuda:  26%|██▌       | 40/153 [00:00<00:02, 45.70batch/s]

Training batches on cuda:  29%|██▉       | 45/153 [00:01<00:02, 40.13batch/s]

Training batches on cuda:  33%|███▎      | 51/153 [00:01<00:02, 43.42batch/s]

Training batches on cuda:  37%|███▋      | 57/153 [00:01<00:02, 45.95batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:01, 47.57batch/s]

Training batches on cuda:  45%|████▌     | 69/153 [00:01<00:01, 49.20batch/s]

Training batches on cuda:  49%|████▉     | 75/153 [00:01<00:01, 49.75batch/s]

Training batches on cuda:  53%|█████▎    | 81/153 [00:01<00:01, 49.38batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:01<00:01, 48.76batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:01<00:01, 49.63batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:02<00:01, 49.86batch/s]

Training batches on cuda:  68%|██████▊   | 104/153 [00:02<00:00, 50.29batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:02<00:00, 50.18batch/s]

Training batches on cuda:  76%|███████▌  | 116/153 [00:02<00:00, 50.77batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:02<00:00, 50.71batch/s]

Training batches on cuda:  84%|████████▎ | 128/153 [00:02<00:00, 51.95batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:02<00:00, 52.45batch/s]

Training batches on cuda:  92%|█████████▏| 140/153 [00:02<00:00, 52.14batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:03<00:00, 52.74batch/s]

Training batches on cuda:  99%|█████████▉| 152/153 [00:03<00:00, 53.42batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:22,  3.17s/epoch, loss=0.000987, prev_loss=0.000987]
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:19,  3.24s/epoch, loss=0.000987, prev_loss=0.000987]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 26.45batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:04, 31.01batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:03, 35.53batch/s]

Training batches on cuda:  14%|█▎        | 21/153 [00:00<00:03, 39.37batch/s]

Training batches on cuda:  18%|█▊        | 27/153 [00:00<00:03, 41.62batch/s]

Training batches on cuda:  22%|██▏       | 33/153 [00:00<00:02, 44.26batch/s]

Training batches on cuda:  25%|██▌       | 39/153 [00:00<00:02, 46.68batch/s]

Training batches on cuda:  29%|██▉       | 45/153 [00:00<00:02, 48.51batch/s]

Training batches on cuda:  33%|███▎      | 51/153 [00:01<00:02, 49.37batch/s]

Training batches on cuda:  37%|███▋      | 57/153 [00:01<00:01, 50.16batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:01, 50.30batch/s]

Training batches on cuda:  45%|████▌     | 69/153 [00:01<00:01, 50.94batch/s]

Training batches on cuda:  49%|████▉     | 75/153 [00:01<00:01, 48.67batch/s]

Training batches on cuda:  53%|█████▎    | 81/153 [00:01<00:01, 50.10batch/s]

Training batches on cuda:  57%|█████▋    | 87/153 [00:01<00:01, 50.98batch/s]

Training batches on cuda:  61%|██████    | 93/153 [00:01<00:01, 51.68batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:01<00:01, 52.37batch/s]

Training batches on cuda:  69%|██████▊   | 105/153 [00:02<00:00, 52.27batch/s]

Training batches on cuda:  73%|███████▎  | 111/153 [00:02<00:00, 50.42batch/s]

Training batches on cuda:  76%|███████▋  | 117/153 [00:02<00:00, 49.72batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:02<00:00, 47.80batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:02<00:00, 46.31batch/s]

Training batches on cuda:  86%|████████▋ | 132/153 [00:02<00:00, 43.12batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:02<00:00, 45.32batch/s]

Training batches on cuda:  93%|█████████▎| 143/153 [00:02<00:00, 45.41batch/s]

Training batches on cuda:  97%|█████████▋| 148/153 [00:03<00:00, 46.44batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:03<00:00, 47.44batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:16<00:19,  3.24s/epoch, loss=0.000987, prev_loss=0.000987]
Training epochs on cuda:  50%|█████     | 5/10 [00:16<00:16,  3.27s/epoch, loss=0.000987, prev_loss=0.000987]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/153 [00:00<00:08, 18.10batch/s]

Training batches on cuda:   5%|▌         | 8/153 [00:00<00:06, 22.39batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:05, 27.02batch/s]

Training batches on cuda:  13%|█▎        | 20/153 [00:00<00:04, 31.40batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:03, 35.89batch/s]

Training batches on cuda:  21%|██        | 32/153 [00:00<00:03, 39.73batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:00<00:02, 42.65batch/s]

Training batches on cuda:  29%|██▉       | 44/153 [00:00<00:02, 45.05batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:01<00:02, 47.05batch/s]

Training batches on cuda:  37%|███▋      | 56/153 [00:01<00:01, 48.54batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:01, 49.79batch/s]

Training batches on cuda:  44%|████▍     | 68/153 [00:01<00:01, 50.07batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 50.55batch/s]

Training batches on cuda:  52%|█████▏    | 80/153 [00:01<00:01, 51.39batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:01<00:01, 49.28batch/s]

Training batches on cuda:  59%|█████▉    | 91/153 [00:01<00:01, 46.28batch/s]

Training batches on cuda:  63%|██████▎   | 97/153 [00:01<00:01, 47.91batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:02<00:01, 46.71batch/s]

Training batches on cuda:  70%|██████▉   | 107/153 [00:02<00:00, 47.12batch/s]

Training batches on cuda:  74%|███████▍  | 113/153 [00:02<00:00, 48.59batch/s]

Training batches on cuda:  78%|███████▊  | 119/153 [00:02<00:00, 49.78batch/s]

Training batches on cuda:  82%|████████▏ | 125/153 [00:02<00:00, 50.96batch/s]

Training batches on cuda:  86%|████████▌ | 131/153 [00:02<00:00, 51.62batch/s]

Training batches on cuda:  90%|████████▉ | 137/153 [00:02<00:00, 52.05batch/s]

Training batches on cuda:  93%|█████████▎| 143/153 [00:02<00:00, 51.91batch/s]

Training batches on cuda:  97%|█████████▋| 149/153 [00:02<00:00, 52.46batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:19<00:16,  3.27s/epoch, loss=0.000987, prev_loss=0.000987]
Training epochs on cuda:  60%|██████    | 6/10 [00:19<00:13,  3.28s/epoch, loss=0.000987, prev_loss=0.000987]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 32.69batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:03, 36.42batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:03, 39.34batch/s]

Training batches on cuda:  13%|█▎        | 20/153 [00:00<00:03, 42.06batch/s]

Training batches on cuda:  16%|█▌        | 24/153 [00:00<00:03, 41.33batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:02, 43.77batch/s]

Training batches on cuda:  23%|██▎       | 35/153 [00:00<00:02, 44.81batch/s]

Training batches on cuda:  27%|██▋       | 41/153 [00:00<00:02, 46.60batch/s]

Training batches on cuda:  31%|███       | 47/153 [00:01<00:02, 48.52batch/s]

Training batches on cuda:  34%|███▍      | 52/153 [00:01<00:02, 44.34batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:02, 46.62batch/s]

Training batches on cuda:  42%|████▏     | 64/153 [00:01<00:01, 47.83batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:01, 49.09batch/s]

Training batches on cuda:  50%|████▉     | 76/153 [00:01<00:01, 50.12batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 50.92batch/s]

Training batches on cuda:  58%|█████▊    | 88/153 [00:01<00:01, 51.31batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:01<00:01, 51.85batch/s]

Training batches on cuda:  65%|██████▌   | 100/153 [00:02<00:01, 52.44batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:02<00:00, 52.58batch/s]

Training batches on cuda:  73%|███████▎  | 112/153 [00:02<00:00, 52.87batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:02<00:00, 52.69batch/s]

Training batches on cuda:  81%|████████  | 124/153 [00:02<00:00, 49.50batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:02<00:00, 49.82batch/s]

Training batches on cuda:  89%|████████▉ | 136/153 [00:02<00:00, 50.03batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:02<00:00, 50.04batch/s]

Training batches on cuda:  97%|█████████▋| 148/153 [00:03<00:00, 49.83batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:03<00:00, 49.69batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:22<00:13,  3.28s/epoch, loss=0.000987, prev_loss=0.000987]
Training epochs on cuda:  70%|███████   | 7/10 [00:22<00:09,  3.30s/epoch, loss=0.000987, prev_loss=0.000987]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/153 [00:00<00:17,  8.79batch/s]

Training batches on cuda:   4%|▍         | 6/153 [00:00<00:12, 11.64batch/s]

Training batches on cuda:   8%|▊         | 12/153 [00:00<00:09, 15.24batch/s]

Training batches on cuda:  11%|█         | 17/153 [00:00<00:07, 19.13batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:05, 23.38batch/s]

Training batches on cuda:  18%|█▊        | 27/153 [00:00<00:04, 27.44batch/s]

Training batches on cuda:  21%|██        | 32/153 [00:00<00:03, 31.62batch/s]

Training batches on cuda:  24%|██▍       | 37/153 [00:00<00:03, 34.99batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:00<00:02, 38.18batch/s]

Training batches on cuda:  31%|███       | 47/153 [00:01<00:02, 40.33batch/s]

Training batches on cuda:  34%|███▍      | 52/153 [00:01<00:02, 41.78batch/s]

Training batches on cuda:  37%|███▋      | 57/153 [00:01<00:02, 43.67batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:02, 44.71batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:01, 45.28batch/s]

Training batches on cuda:  47%|████▋     | 72/153 [00:01<00:01, 46.37batch/s]

Training batches on cuda:  50%|█████     | 77/153 [00:01<00:01, 46.18batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 46.80batch/s]

Training batches on cuda:  57%|█████▋    | 87/153 [00:01<00:01, 47.14batch/s]

Training batches on cuda:  61%|██████    | 93/153 [00:02<00:01, 48.39batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:02<00:01, 49.46batch/s]

Training batches on cuda:  68%|██████▊   | 104/153 [00:02<00:01, 48.70batch/s]

Training batches on cuda:  71%|███████   | 109/153 [00:02<00:00, 46.75batch/s]

Training batches on cuda:  75%|███████▌  | 115/153 [00:02<00:00, 48.01batch/s]

Training batches on cuda:  79%|███████▉  | 121/153 [00:02<00:00, 48.99batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:02<00:00, 49.83batch/s]

Training batches on cuda:  87%|████████▋ | 133/153 [00:02<00:00, 48.87batch/s]

Training batches on cuda:  91%|█████████ | 139/153 [00:02<00:00, 50.13batch/s]

Training batches on cuda:  95%|█████████▍| 145/153 [00:03<00:00, 51.68batch/s]

Training batches on cuda:  99%|█████████▊| 151/153 [00:03<00:00, 51.84batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:26<00:09,  3.30s/epoch, loss=0.000987, prev_loss=0.000987]
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:06,  3.35s/epoch, loss=0.000987, prev_loss=0.000987]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 25.75batch/s]

Training batches on cuda:   5%|▌         | 8/153 [00:00<00:04, 29.69batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:04, 33.87batch/s]

Training batches on cuda:  13%|█▎        | 20/153 [00:00<00:03, 38.29batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:03, 41.21batch/s]

Training batches on cuda:  21%|██        | 32/153 [00:00<00:02, 44.11batch/s]

Training batches on cuda:  24%|██▍       | 37/153 [00:00<00:02, 42.74batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:00<00:02, 43.98batch/s]

Training batches on cuda:  31%|███       | 47/153 [00:01<00:02, 45.05batch/s]

Training batches on cuda:  34%|███▍      | 52/153 [00:01<00:02, 45.93batch/s]

Training batches on cuda:  37%|███▋      | 57/153 [00:01<00:02, 46.11batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:02, 44.01batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:01, 44.55batch/s]

Training batches on cuda:  47%|████▋     | 72/153 [00:01<00:01, 45.01batch/s]

Training batches on cuda:  50%|█████     | 77/153 [00:01<00:01, 45.37batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 45.93batch/s]

Training batches on cuda:  57%|█████▋    | 87/153 [00:01<00:01, 46.27batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:02<00:01, 46.03batch/s]

Training batches on cuda:  63%|██████▎   | 97/153 [00:02<00:01, 46.68batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:02<00:01, 47.20batch/s]

Training batches on cuda:  70%|██████▉   | 107/153 [00:02<00:00, 47.54batch/s]

Training batches on cuda:  73%|███████▎  | 112/153 [00:02<00:00, 47.29batch/s]

Training batches on cuda:  76%|███████▋  | 117/153 [00:02<00:00, 47.66batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:02<00:00, 47.08batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:02<00:00, 46.72batch/s]

Training batches on cuda:  86%|████████▋ | 132/153 [00:02<00:00, 46.87batch/s]

Training batches on cuda:  90%|████████▉ | 137/153 [00:02<00:00, 47.08batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:03<00:00, 47.18batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:03<00:00, 47.16batch/s]

Training batches on cuda:  99%|█████████▉| 152/153 [00:03<00:00, 47.63batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:29<00:06,  3.35s/epoch, loss=0.000987, prev_loss=0.000987]
Training epochs on cuda:  90%|█████████ | 9/10 [00:29<00:03,  3.40s/epoch, loss=0.000987, prev_loss=0.000987]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/153 [00:00<00:09, 15.94batch/s]

Training batches on cuda:   5%|▍         | 7/153 [00:00<00:07, 19.84batch/s]

Training batches on cuda:   8%|▊         | 12/153 [00:00<00:05, 23.90batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:04, 28.69batch/s]

Training batches on cuda:  16%|█▌        | 24/153 [00:00<00:03, 33.23batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:03, 37.45batch/s]

Training batches on cuda:  23%|██▎       | 35/153 [00:00<00:02, 39.91batch/s]

Training batches on cuda:  26%|██▌       | 40/153 [00:00<00:02, 40.11batch/s]

Training batches on cuda:  29%|██▉       | 45/153 [00:01<00:02, 42.19batch/s]

Training batches on cuda:  33%|███▎      | 51/153 [00:01<00:02, 44.14batch/s]

Training batches on cuda:  37%|███▋      | 57/153 [00:01<00:02, 46.44batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:01, 47.64batch/s]

Training batches on cuda:  44%|████▍     | 68/153 [00:01<00:01, 47.68batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 49.22batch/s]

Training batches on cuda:  52%|█████▏    | 80/153 [00:01<00:01, 50.25batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:01<00:01, 49.38batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:01<00:01, 50.53batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:02<00:01, 51.04batch/s]

Training batches on cuda:  68%|██████▊   | 104/153 [00:02<00:00, 51.80batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:02<00:00, 51.73batch/s]

Training batches on cuda:  76%|███████▌  | 116/153 [00:02<00:00, 53.55batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:02<00:00, 53.56batch/s]

Training batches on cuda:  84%|████████▎ | 128/153 [00:02<00:00, 53.46batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:02<00:00, 54.77batch/s]

Training batches on cuda:  92%|█████████▏| 140/153 [00:02<00:00, 55.57batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:02<00:00, 55.27batch/s]

Training batches on cuda:  99%|█████████▉| 152/153 [00:03<00:00, 54.00batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:33<00:03,  3.40s/epoch, loss=0.000987, prev_loss=0.000987]2021-09-16:18:17:43,117 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:17:44,109 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:17:59,72 INFO     [evaluator.py:657] Evaluation took 14.96s seconds
2021-09-16:18:17:59,200 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:49<00:00,  8.19s/epoch, loss=0.000987, prev_loss=0.000987]
Training epochs on cuda: 100%|██████████| 10/10 [00:49<00:00,  4.93s/epoch, loss=0.000987, prev_loss=0.000987]
2021-09-16:18:17:59,201 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp_l9wtbo7'
2021-09-16:18:17:59,255 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_l9wtbo7' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.55ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.58ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:18:14,124 INFO     [evaluator.py:657] Evaluation took 14.85s seconds
[I 2021-09-16 18:18:14,129] Trial 40 finished with value: -0.005275884938920283 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.20163925607115324, 'optimizer.lr': 0.021487113985745487, 'negative_sampler.num_negs_per_pos': 13, 'training.batch_size': 2032}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:18:14,151 WARNING  [api.py:823] No random seed is specified. Setting to 3762636234.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:05, 19.49batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 21.58batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 23.56batch/s]

Training batches on cuda:  11%|█         | 12/114 [00:00<00:04, 25.29batch/s]

Training batches on cuda:  14%|█▍        | 16/114 [00:00<00:03, 26.76batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 27.76batch/s]

Training batches on cuda:  21%|██        | 24/114 [00:00<00:03, 28.55batch/s]

Training batches on cuda:  25%|██▍       | 28/114 [00:00<00:02, 29.07batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:02, 29.54batch/s]

Training batches on cuda:  32%|███▏      | 36/114 [00:01<00:02, 29.76batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:01<00:02, 29.55batch/s]

Training batches on cuda:  37%|███▋      | 42/114 [00:01<00:02, 29.46batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:01<00:02, 29.23batch/s]

Training batches on cuda:  42%|████▏     | 48/114 [00:01<00:02, 29.18batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:01<00:02, 29.15batch/s]

Training batches on cuda:  47%|████▋     | 54/114 [00:01<00:02, 29.14batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:01<00:01, 28.99batch/s]

Training batches on cuda:  53%|█████▎    | 60/114 [00:02<00:01, 28.80batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:02<00:01, 28.53batch/s]

Training batches on cuda:  58%|█████▊    | 66/114 [00:02<00:01, 27.84batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:02<00:01, 28.22batch/s]

Training batches on cuda:  63%|██████▎   | 72/114 [00:02<00:01, 28.46batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:02<00:01, 28.71batch/s]

Training batches on cuda:  68%|██████▊   | 78/114 [00:02<00:01, 28.81batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:02<00:01, 28.79batch/s]

Training batches on cuda:  74%|███████▎  | 84/114 [00:02<00:01, 28.86batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:02<00:00, 28.97batch/s]

Training batches on cuda:  79%|███████▉  | 90/114 [00:03<00:00, 29.25batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:03<00:00, 29.24batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:03<00:00, 29.71batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 29.87batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 29.77batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 29.63batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:03<00:00, 29.69batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000486, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:37,  4.12s/epoch, loss=0.000486, prev_loss=nan]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:07, 15.57batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:06, 17.84batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 20.15batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.23batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 23.70batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 25.08batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 26.13batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.76batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 27.44batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 27.75batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.55batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 26.03batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 25.87batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 25.53batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 25.61batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 25.49batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 25.75batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 25.92batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 25.85batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 25.66batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:02, 25.48batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 25.18batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 25.13batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 25.10batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 22.19batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:03<00:01, 22.98batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 23.82batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 24.92batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 25.82batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 26.58batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 26.92batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 27.41batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 27.81batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 27.88batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 27.96batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 27.95batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.52batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 27.28batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:37,  4.12s/epoch, loss=0.000484, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:34,  4.26s/epoch, loss=0.000484, prev_loss=nan]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 18.57batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 20.63batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 22.62batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 24.16batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:03, 25.32batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 26.03batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 24.97batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 24.00batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 24.58batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 25.60batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.06batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 26.00batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.34batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 26.49batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.16batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 27.77batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.87batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 27.89batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 27.64batch/s]

Training batches on cuda:  53%|█████▎    | 60/114 [00:02<00:01, 28.30batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:02<00:01, 28.40batch/s]

Training batches on cuda:  58%|█████▊    | 66/114 [00:02<00:01, 28.79batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:02<00:01, 28.58batch/s]

Training batches on cuda:  63%|██████▎   | 72/114 [00:02<00:01, 28.00batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:02<00:01, 27.57batch/s]

Training batches on cuda:  68%|██████▊   | 78/114 [00:02<00:01, 27.78batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:02<00:01, 27.56batch/s]

Training batches on cuda:  74%|███████▎  | 84/114 [00:03<00:01, 27.46batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:03<00:00, 27.67batch/s]

Training batches on cuda:  79%|███████▉  | 90/114 [00:03<00:00, 24.91batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:03<00:00, 25.72batch/s]

Training batches on cuda:  84%|████████▍ | 96/114 [00:03<00:00, 26.29batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:03<00:00, 27.11batch/s]

Training batches on cuda:  89%|████████▉ | 102/114 [00:03<00:00, 27.41batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:03<00:00, 27.65batch/s]

Training batches on cuda:  95%|█████████▍| 108/114 [00:03<00:00, 27.87batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:04<00:00, 28.22batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:34,  4.26s/epoch, loss=0.000484, prev_loss=0.000484]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:30,  4.31s/epoch, loss=0.000484, prev_loss=0.000484]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 18.63batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 20.48batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 22.39batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 23.58batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.30batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.83batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.28batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.02batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 26.52batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.89batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 27.08batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.92batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 27.21batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.51batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.81batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 28.07batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 28.11batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 28.09batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 28.09batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:01, 28.03batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 27.86batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 27.86batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 27.92batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.79batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.89batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 28.05batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:02<00:01, 28.08batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.98batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.94batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.83batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.99batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 28.08batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 28.10batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 28.19batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 28.28batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 24.98batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 25.69batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 26.51batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:30,  4.31s/epoch, loss=0.000484, prev_loss=0.000484]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:26,  4.33s/epoch, loss=0.000484, prev_loss=0.000484]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 16.56batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 18.54batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 20.46batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 21.99batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 23.34batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.29batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.04batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.55batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 25.96batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.21batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.26batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.37batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.45batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 26.71batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 26.83batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 26.69batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 25.99batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 25.95batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 26.28batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 26.26batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.38batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.52batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.63batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 26.72batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 26.61batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 26.67batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 26.63batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 26.79batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 26.80batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 26.81batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 26.86batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.81batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 26.17batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 26.23batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 26.47batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 26.68batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.33batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 27.82batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:26,  4.33s/epoch, loss=0.000484, prev_loss=0.000484]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:21,  4.39s/epoch, loss=0.000484, prev_loss=0.000484]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 16.91batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 18.87batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 20.70batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.16batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 23.45batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.33batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.08batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.67batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 26.11batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.24batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.83batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 27.45batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 27.81batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 28.12batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 28.39batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 28.56batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 28.71batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 28.74batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 28.65batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:01, 28.49batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 28.06batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.78batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.69batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 26.07batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.03batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 26.26batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:02<00:01, 26.03batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 26.47batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 26.37batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 26.93batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.32batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 27.28batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 27.30batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 27.12batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 26.77batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 27.31batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.79batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 28.07batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:26<00:21,  4.39s/epoch, loss=0.000484, prev_loss=0.000484]
Training epochs on cuda:  60%|██████    | 6/10 [00:26<00:17,  4.44s/epoch, loss=0.000484, prev_loss=0.000484]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:04, 22.71batch/s]

Training batches on cuda:   5%|▌         | 6/114 [00:00<00:04, 23.55batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 22.00batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 23.03batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:00<00:04, 21.62batch/s]

Training batches on cuda:  14%|█▍        | 16/114 [00:00<00:04, 22.83batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:00<00:03, 23.87batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.55batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 26.38batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.57batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.27batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.71batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.39batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 26.82batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 26.67batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 27.34batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.38batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 27.48batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 27.66batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:01, 28.05batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 28.06batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 28.08batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 28.28batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.85batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 26.86batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.35batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 26.60batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 26.87batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.12batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.53batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.64batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 28.04batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:03<00:00, 28.47batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:03<00:00, 28.81batch/s]

Training batches on cuda:  93%|█████████▎| 106/114 [00:03<00:00, 28.97batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:04<00:00, 29.02batch/s]

Training batches on cuda:  98%|█████████▊| 112/114 [00:04<00:00, 29.03batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:17,  4.44s/epoch, loss=0.000484, prev_loss=0.000484]
Training epochs on cuda:  70%|███████   | 7/10 [00:30<00:13,  4.43s/epoch, loss=0.000484, prev_loss=0.000484]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:05, 20.93batch/s]

Training batches on cuda:   5%|▌         | 6/114 [00:00<00:04, 22.18batch/s]

Training batches on cuda:   9%|▉         | 10/114 [00:00<00:04, 24.22batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:03, 25.92batch/s]

Training batches on cuda:  16%|█▌        | 18/114 [00:00<00:03, 27.16batch/s]

Training batches on cuda:  19%|█▉        | 22/114 [00:00<00:03, 27.94batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:00<00:03, 28.45batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:02, 29.19batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:01<00:02, 29.76batch/s]

Training batches on cuda:  32%|███▏      | 36/114 [00:01<00:02, 26.60batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:01<00:02, 27.53batch/s]

Training batches on cuda:  37%|███▋      | 42/114 [00:01<00:02, 27.64batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:01<00:02, 27.53batch/s]

Training batches on cuda:  42%|████▏     | 48/114 [00:01<00:02, 27.89batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:01<00:02, 27.77batch/s]

Training batches on cuda:  47%|████▋     | 54/114 [00:01<00:02, 27.39batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:02<00:02, 27.40batch/s]

Training batches on cuda:  53%|█████▎    | 60/114 [00:02<00:01, 27.82batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:02<00:01, 27.53batch/s]

Training batches on cuda:  58%|█████▊    | 66/114 [00:02<00:01, 27.54batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:02<00:01, 27.89batch/s]

Training batches on cuda:  63%|██████▎   | 72/114 [00:02<00:01, 28.08batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:02<00:01, 28.32batch/s]

Training batches on cuda:  68%|██████▊   | 78/114 [00:02<00:01, 28.51batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:02<00:01, 28.15batch/s]

Training batches on cuda:  74%|███████▎  | 84/114 [00:02<00:01, 28.26batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:03<00:00, 28.08batch/s]

Training batches on cuda:  79%|███████▉  | 90/114 [00:03<00:00, 27.83batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:03<00:00, 27.78batch/s]

Training batches on cuda:  84%|████████▍ | 96/114 [00:03<00:00, 27.76batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:03<00:00, 28.22batch/s]

Training batches on cuda:  89%|████████▉ | 102/114 [00:03<00:00, 28.27batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:03<00:00, 28.37batch/s]

Training batches on cuda:  95%|█████████▍| 108/114 [00:03<00:00, 28.36batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:03<00:00, 28.62batch/s]

Training batches on cuda: 100%|██████████| 114/114 [00:04<00:00, 28.41batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:13,  4.43s/epoch, loss=0.000484, prev_loss=0.000484]
Training epochs on cuda:  80%|████████  | 8/10 [00:35<00:08,  4.39s/epoch, loss=0.000484, prev_loss=0.000484]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:05, 18.98batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 21.03batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 22.52batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 24.24batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:03, 25.45batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 26.35batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 26.70batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 27.09batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:00<00:03, 27.43batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 27.54batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:02, 27.73batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 27.99batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 27.48batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.38batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.51batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 27.11batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.46batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 27.23batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 24.14batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 25.39batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.15batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.90batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 27.01batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 26.75batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 26.86batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.03batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:02<00:01, 26.93batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 26.78batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.11batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.54batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.95batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 27.93batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 27.86batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 28.17batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 27.95batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 28.08batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.49batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 26.79batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:39<00:08,  4.39s/epoch, loss=0.000484, prev_loss=0.000484]
Training epochs on cuda:  90%|█████████ | 9/10 [00:39<00:04,  4.40s/epoch, loss=0.000484, prev_loss=0.000484]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:07, 15.14batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:06, 17.18batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 19.20batch/s]

Training batches on cuda:  11%|█         | 12/114 [00:00<00:04, 21.54batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:00<00:04, 22.84batch/s]

Training batches on cuda:  16%|█▌        | 18/114 [00:00<00:04, 23.82batch/s]

Training batches on cuda:  19%|█▉        | 22/114 [00:00<00:03, 25.06batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:00<00:03, 25.93batch/s]

Training batches on cuda:  25%|██▍       | 28/114 [00:01<00:03, 26.72batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:01<00:03, 27.35batch/s]

Training batches on cuda:  30%|██▉       | 34/114 [00:01<00:02, 27.94batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:01<00:02, 28.30batch/s]

Training batches on cuda:  35%|███▌      | 40/114 [00:01<00:02, 27.82batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:01<00:02, 28.01batch/s]

Training batches on cuda:  40%|████      | 46/114 [00:01<00:02, 27.90batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:01<00:02, 28.09batch/s]

Training batches on cuda:  46%|████▌     | 52/114 [00:01<00:02, 28.09batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:02<00:02, 28.14batch/s]

Training batches on cuda:  51%|█████     | 58/114 [00:02<00:01, 28.30batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:02<00:01, 28.39batch/s]

Training batches on cuda:  56%|█████▌    | 64/114 [00:02<00:01, 28.60batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:02<00:01, 28.70batch/s]

Training batches on cuda:  61%|██████▏   | 70/114 [00:02<00:01, 28.78batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:02<00:01, 24.99batch/s]

Training batches on cuda:  67%|██████▋   | 76/114 [00:02<00:01, 25.27batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:02<00:01, 25.24batch/s]

Training batches on cuda:  72%|███████▏  | 82/114 [00:03<00:01, 25.18batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:03<00:01, 25.37batch/s]

Training batches on cuda:  77%|███████▋  | 88/114 [00:03<00:01, 25.80batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:03<00:00, 26.41batch/s]

Training batches on cuda:  82%|████████▏ | 94/114 [00:03<00:00, 26.20batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:03<00:00, 26.83batch/s]

Training batches on cuda:  88%|████████▊ | 100/114 [00:03<00:00, 27.49batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:03<00:00, 27.92batch/s]

Training batches on cuda:  93%|█████████▎| 106/114 [00:03<00:00, 28.04batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:04<00:00, 28.17batch/s]

Training batches on cuda:  98%|█████████▊| 112/114 [00:04<00:00, 28.22batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:44<00:04,  4.40s/epoch, loss=0.000484, prev_loss=0.000484]2021-09-16:18:18:58,962 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:19:00,77 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:19:15,106 INFO     [evaluator.py:657] Evaluation took 15.03s seconds
2021-09-16:18:19:15,276 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00,  9.33s/epoch, loss=0.000484, prev_loss=0.000484]
Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00,  6.06s/epoch, loss=0.000484, prev_loss=0.000484]
2021-09-16:18:19:15,276 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp10smzs4c'
2021-09-16:18:19:15,330 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp10smzs4c' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.32ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.41ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.54ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.56ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.52ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.56ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:07, 2.58ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.60ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:19:30,212 INFO     [evaluator.py:657] Evaluation took 14.87s seconds
[I 2021-09-16 18:19:30,217] Trial 41 finished with value: -0.0014562022984110357 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.058662504832084836, 'optimizer.lr': 0.08204099275029648, 'negative_sampler.num_negs_per_pos': 20, 'training.batch_size': 2732}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:19:30,240 WARNING  [api.py:823] No random seed is specified. Setting to 3403035415.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:07, 13.83batch/s]

Training batches on cuda:   4%|▍         | 4/106 [00:00<00:06, 15.02batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:06, 16.17batch/s]

Training batches on cuda:   8%|▊         | 8/106 [00:00<00:05, 17.03batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:05, 17.34batch/s]

Training batches on cuda:  11%|█▏        | 12/106 [00:00<00:05, 17.90batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:04, 18.40batch/s]

Training batches on cuda:  15%|█▌        | 16/106 [00:00<00:04, 18.69batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:00<00:04, 18.77batch/s]

Training batches on cuda:  19%|█▉        | 20/106 [00:01<00:04, 18.84batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:01<00:04, 18.90batch/s]

Training batches on cuda:  23%|██▎       | 24/106 [00:01<00:04, 18.93batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:01<00:04, 18.98batch/s]

Training batches on cuda:  26%|██▋       | 28/106 [00:01<00:04, 19.13batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:01<00:04, 18.95batch/s]

Training batches on cuda:  30%|███       | 32/106 [00:01<00:03, 18.69batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:04, 17.96batch/s]

Training batches on cuda:  34%|███▍      | 36/106 [00:01<00:03, 17.73batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:02<00:03, 17.66batch/s]

Training batches on cuda:  38%|███▊      | 40/106 [00:02<00:03, 17.70batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:02<00:03, 17.88batch/s]

Training batches on cuda:  42%|████▏     | 44/106 [00:02<00:03, 17.95batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:02<00:03, 17.94batch/s]

Training batches on cuda:  45%|████▌     | 48/106 [00:02<00:03, 17.94batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:02<00:03, 18.02batch/s]

Training batches on cuda:  49%|████▉     | 52/106 [00:02<00:03, 17.95batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:02<00:02, 17.97batch/s]

Training batches on cuda:  53%|█████▎    | 56/106 [00:03<00:02, 17.99batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:03<00:02, 17.90batch/s]

Training batches on cuda:  57%|█████▋    | 60/106 [00:03<00:02, 17.74batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:03<00:02, 17.98batch/s]

Training batches on cuda:  60%|██████    | 64/106 [00:03<00:02, 18.25batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:03<00:02, 18.11batch/s]

Training batches on cuda:  64%|██████▍   | 68/106 [00:03<00:02, 18.13batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:03<00:01, 18.10batch/s]

Training batches on cuda:  68%|██████▊   | 72/106 [00:03<00:01, 18.00batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:04<00:01, 17.82batch/s]

Training batches on cuda:  72%|███████▏  | 76/106 [00:04<00:01, 17.75batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:04<00:01, 17.76batch/s]

Training batches on cuda:  75%|███████▌  | 80/106 [00:04<00:01, 17.53batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:04<00:01, 17.57batch/s]

Training batches on cuda:  79%|███████▉  | 84/106 [00:04<00:01, 17.71batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:04<00:01, 17.72batch/s]

Training batches on cuda:  83%|████████▎ | 88/106 [00:04<00:01, 17.69batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:04<00:00, 17.61batch/s]

Training batches on cuda:  87%|████████▋ | 92/106 [00:05<00:00, 17.66batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:05<00:00, 17.69batch/s]

Training batches on cuda:  91%|█████████ | 96/106 [00:05<00:00, 17.75batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:05<00:00, 17.74batch/s]

Training batches on cuda:  94%|█████████▍| 100/106 [00:05<00:00, 17.64batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:05<00:00, 17.67batch/s]

Training batches on cuda:  98%|█████████▊| 104/106 [00:05<00:00, 17.58batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:05<00:00, 17.65batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:06<?, ?epoch/s, loss=0.000449, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:54,  6.09s/epoch, loss=0.000449, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:14,  7.47batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:11,  8.79batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:09, 10.27batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:08, 11.83batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:07, 13.37batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:06, 14.74batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:00<00:05, 15.82batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:05, 16.56batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:05, 16.98batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:05, 17.16batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:04, 17.59batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:04, 17.55batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:04, 17.59batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:01<00:04, 17.93batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:01<00:04, 18.16batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:01<00:04, 18.25batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:03, 18.38batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:03, 18.44batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:03, 18.59batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:02<00:03, 18.27batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:02<00:03, 18.33batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:02<00:03, 18.47batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:02<00:03, 18.54batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:02<00:03, 18.57batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:02<00:03, 18.02batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:02<00:03, 17.96batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:02<00:02, 17.99batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:03<00:02, 18.17batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:03<00:02, 17.98batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:03<00:02, 17.82batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:03<00:02, 18.03batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:03<00:02, 18.28batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:03<00:02, 18.04batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:03<00:02, 17.99batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:03<00:02, 18.06batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:04<00:01, 17.81batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:04<00:01, 18.14batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:04<00:01, 18.49batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:04<00:01, 18.63batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:04<00:01, 18.81batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:04<00:01, 18.68batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:04<00:01, 18.62batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:04<00:01, 18.24batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:04<00:01, 17.80batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:04<00:00, 17.88batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:05<00:00, 17.85batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:05<00:00, 17.68batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:05<00:00, 18.06batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:05<00:00, 17.65batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:05<00:00, 17.46batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:05<00:00, 17.52batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:05<00:00, 17.41batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:05<00:00, 17.76batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:12<00:54,  6.09s/epoch, loss=0.000445, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:48,  6.11s/epoch, loss=0.000445, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:12,  8.56batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:10, 10.17batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:08, 11.82batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:07, 13.35batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:06, 14.69batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:05, 15.90batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:00<00:05, 16.58batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:05, 17.15batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:00<00:05, 17.36batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:04, 17.55batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:04, 17.62batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:04, 17.73batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:04, 17.97batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:01<00:04, 18.19batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:01<00:04, 18.35batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:01<00:04, 18.48batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:03, 18.63batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:03, 18.63batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:03, 18.20batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:02<00:03, 18.53batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:02<00:03, 18.47batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:02<00:03, 18.78batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:02<00:03, 18.96batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:02<00:03, 18.86batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:02<00:03, 18.73batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:02<00:02, 18.43batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:02<00:02, 18.05batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:03<00:02, 18.27batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:03<00:02, 18.26batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:03<00:02, 18.49batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:03<00:02, 18.19batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:03<00:02, 18.19batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:03<00:02, 18.38batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:03<00:02, 18.31batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:03<00:02, 18.29batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:03<00:01, 18.41batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:04<00:01, 18.48batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:04<00:01, 18.49batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:04<00:01, 18.62batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:04<00:01, 18.61batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:04<00:01, 18.68batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:04<00:01, 17.94batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:04<00:01, 18.14batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:04<00:01, 17.79batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:04<00:01, 16.05batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:05<00:00, 16.91batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:05<00:00, 17.26batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:05<00:00, 17.31batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:05<00:00, 17.68batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:05<00:00, 17.65batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:05<00:00, 17.97batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:05<00:00, 18.23batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:05<00:00, 18.35batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:18<00:48,  6.11s/epoch, loss=0.000432, prev_loss=0.000445]
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:42,  6.10s/epoch, loss=0.000432, prev_loss=0.000445]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:12,  8.08batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:10,  9.69batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:08, 11.28batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:07, 12.64batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:06, 13.95batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:06, 15.00batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:00<00:05, 15.87batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:05, 16.52batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:05, 16.86batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:05, 17.24batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:04, 17.56batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:04, 17.77batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:04, 17.45batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:01<00:04, 17.95batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:01<00:04, 17.93batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:01<00:04, 18.10batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:04, 18.20batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:03, 18.45batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:03, 18.20batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:02<00:03, 18.32batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:02<00:03, 18.48batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:02<00:03, 18.55batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:02<00:03, 18.60batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:02<00:03, 18.61batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:02<00:03, 18.66batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:02<00:02, 18.59batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:02<00:02, 18.68batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:03<00:02, 18.54batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:03<00:02, 18.46batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:03<00:02, 18.52batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:03<00:02, 18.56batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:03<00:02, 18.57batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:03<00:02, 18.62batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:03<00:02, 18.31batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:03<00:02, 16.67batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:03<00:02, 17.07batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:04<00:01, 17.58batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:04<00:01, 17.81batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:04<00:01, 17.80batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:04<00:01, 18.03batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:04<00:01, 18.04batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:04<00:01, 18.15batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:04<00:01, 18.12batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:04<00:01, 18.02batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:04<00:00, 18.06batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:05<00:00, 17.97batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:05<00:00, 18.24batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:05<00:00, 18.31batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:05<00:00, 18.05batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:05<00:00, 18.25batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:05<00:00, 18.01batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:05<00:00, 18.18batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:05<00:00, 18.22batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:24<00:42,  6.10s/epoch, loss=0.000421, prev_loss=0.000432]
Training epochs on cuda:  40%|████      | 4/10 [00:24<00:36,  6.11s/epoch, loss=0.000421, prev_loss=0.000432]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:11,  9.28batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:09, 10.83batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:08, 12.41batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:07, 13.79batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:06, 14.92batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:05, 15.88batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:00<00:05, 16.63batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:05, 17.18batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:00<00:05, 17.60batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:04, 17.88batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:04, 18.13batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:04, 18.33batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:04, 18.07batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:01<00:04, 18.14batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:01<00:04, 18.07batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:01<00:04, 17.99batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:04, 17.86batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:03, 17.88batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:03, 17.88batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:02<00:03, 17.99batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:02<00:03, 17.95batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:02<00:03, 17.83batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:02<00:03, 17.69batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:02<00:03, 17.74batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:02<00:03, 16.09batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:02<00:03, 16.63batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:03<00:03, 17.03batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:03<00:02, 17.12batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:03<00:02, 17.62batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:03<00:02, 17.94batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:03<00:02, 18.28batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:03<00:02, 18.47batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:03<00:02, 18.37batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:03<00:02, 18.52batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:03<00:01, 18.79batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:03<00:01, 18.85batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:04<00:01, 18.94batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:04<00:01, 18.99batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:04<00:01, 19.03batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:04<00:01, 19.16batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:04<00:01, 19.17batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:04<00:01, 19.20batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:04<00:01, 19.18batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:04<00:00, 19.23batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:04<00:00, 19.21batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:05<00:00, 18.98batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:05<00:00, 19.14batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:05<00:00, 19.13batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:05<00:00, 19.19batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:05<00:00, 19.14batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:05<00:00, 19.21batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:05<00:00, 19.27batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:05<00:00, 19.13batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:30<00:36,  6.11s/epoch, loss=0.000353, prev_loss=0.000421]
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:30,  6.08s/epoch, loss=0.000353, prev_loss=0.000421]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:07, 13.44batch/s]

Training batches on cuda:   4%|▍         | 4/106 [00:00<00:06, 14.66batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:06, 15.75batch/s]

Training batches on cuda:   8%|▊         | 8/106 [00:00<00:05, 16.65batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:05, 17.34batch/s]

Training batches on cuda:  11%|█▏        | 12/106 [00:00<00:05, 17.95batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:05, 17.99batch/s]

Training batches on cuda:  15%|█▌        | 16/106 [00:00<00:05, 17.85batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:01<00:04, 17.89batch/s]

Training batches on cuda:  19%|█▉        | 20/106 [00:01<00:04, 18.07batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:01<00:04, 18.23batch/s]

Training batches on cuda:  23%|██▎       | 24/106 [00:01<00:04, 18.36batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:01<00:04, 18.31batch/s]

Training batches on cuda:  26%|██▋       | 28/106 [00:01<00:04, 18.27batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:01<00:04, 18.23batch/s]

Training batches on cuda:  30%|███       | 32/106 [00:01<00:04, 17.59batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:04, 17.86batch/s]

Training batches on cuda:  34%|███▍      | 36/106 [00:02<00:03, 17.61batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:02<00:03, 17.59batch/s]

Training batches on cuda:  38%|███▊      | 40/106 [00:02<00:03, 17.17batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:02<00:03, 17.24batch/s]

Training batches on cuda:  42%|████▏     | 44/106 [00:02<00:03, 17.29batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:02<00:03, 17.08batch/s]

Training batches on cuda:  45%|████▌     | 48/106 [00:02<00:03, 17.33batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:02<00:03, 17.51batch/s]

Training batches on cuda:  49%|████▉     | 52/106 [00:02<00:03, 17.78batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:03<00:02, 17.94batch/s]

Training batches on cuda:  53%|█████▎    | 56/106 [00:03<00:02, 18.08batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:03<00:02, 18.10batch/s]

Training batches on cuda:  57%|█████▋    | 60/106 [00:03<00:02, 18.12batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:03<00:02, 18.21batch/s]

Training batches on cuda:  60%|██████    | 64/106 [00:03<00:02, 18.28batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:03<00:02, 18.36batch/s]

Training batches on cuda:  64%|██████▍   | 68/106 [00:03<00:02, 18.50batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:03<00:01, 18.56batch/s]

Training batches on cuda:  68%|██████▊   | 72/106 [00:04<00:01, 18.68batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:04<00:01, 18.82batch/s]

Training batches on cuda:  72%|███████▏  | 76/106 [00:04<00:01, 18.87batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:04<00:01, 18.92batch/s]

Training batches on cuda:  75%|███████▌  | 80/106 [00:04<00:01, 19.14batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:04<00:01, 19.03batch/s]

Training batches on cuda:  79%|███████▉  | 84/106 [00:04<00:01, 18.49batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:04<00:01, 18.71batch/s]

Training batches on cuda:  83%|████████▎ | 88/106 [00:04<00:00, 18.09batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:04<00:00, 18.42batch/s]

Training batches on cuda:  87%|████████▋ | 92/106 [00:05<00:00, 18.03batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:05<00:00, 18.38batch/s]

Training batches on cuda:  91%|█████████ | 96/106 [00:05<00:00, 18.28batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:05<00:00, 18.10batch/s]

Training batches on cuda:  94%|█████████▍| 100/106 [00:05<00:00, 18.10batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:05<00:00, 17.87batch/s]

Training batches on cuda:  98%|█████████▊| 104/106 [00:05<00:00, 18.13batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:05<00:00, 18.06batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:36<00:30,  6.08s/epoch, loss=0.000266, prev_loss=0.000353]
Training epochs on cuda:  60%|██████    | 6/10 [00:36<00:24,  6.10s/epoch, loss=0.000266, prev_loss=0.000353]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:07, 13.62batch/s]

Training batches on cuda:   4%|▍         | 4/106 [00:00<00:06, 14.75batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:06, 15.80batch/s]

Training batches on cuda:   8%|▊         | 8/106 [00:00<00:05, 16.67batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:05, 17.27batch/s]

Training batches on cuda:  11%|█▏        | 12/106 [00:00<00:05, 16.10batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:05, 16.85batch/s]

Training batches on cuda:  15%|█▌        | 16/106 [00:00<00:05, 17.43batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:01<00:04, 17.85batch/s]

Training batches on cuda:  19%|█▉        | 20/106 [00:01<00:04, 18.19batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:01<00:04, 18.42batch/s]

Training batches on cuda:  23%|██▎       | 24/106 [00:01<00:04, 18.53batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:01<00:04, 18.70batch/s]

Training batches on cuda:  26%|██▋       | 28/106 [00:01<00:04, 18.78batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:01<00:04, 18.75batch/s]

Training batches on cuda:  30%|███       | 32/106 [00:01<00:03, 18.73batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:03, 18.84batch/s]

Training batches on cuda:  34%|███▍      | 36/106 [00:01<00:03, 18.82batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:02<00:03, 18.87batch/s]

Training batches on cuda:  38%|███▊      | 40/106 [00:02<00:03, 18.93batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:02<00:03, 18.92batch/s]

Training batches on cuda:  42%|████▏     | 44/106 [00:02<00:03, 18.84batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:02<00:03, 18.87batch/s]

Training batches on cuda:  45%|████▌     | 48/106 [00:02<00:03, 18.77batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:02<00:02, 18.75batch/s]

Training batches on cuda:  49%|████▉     | 52/106 [00:02<00:02, 18.80batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:02<00:02, 18.81batch/s]

Training batches on cuda:  53%|█████▎    | 56/106 [00:03<00:02, 18.75batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:03<00:02, 18.73batch/s]

Training batches on cuda:  57%|█████▋    | 60/106 [00:03<00:02, 18.78batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:03<00:02, 18.74batch/s]

Training batches on cuda:  60%|██████    | 64/106 [00:03<00:02, 18.75batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:03<00:02, 18.56batch/s]

Training batches on cuda:  64%|██████▍   | 68/106 [00:03<00:02, 18.70batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:03<00:01, 18.68batch/s]

Training batches on cuda:  68%|██████▊   | 72/106 [00:03<00:01, 18.80batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:04<00:01, 18.89batch/s]

Training batches on cuda:  72%|███████▏  | 76/106 [00:04<00:01, 18.88batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:04<00:01, 18.90batch/s]

Training batches on cuda:  75%|███████▌  | 80/106 [00:04<00:01, 18.84batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:04<00:01, 18.85batch/s]

Training batches on cuda:  79%|███████▉  | 84/106 [00:04<00:01, 18.61batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:04<00:01, 18.75batch/s]

Training batches on cuda:  83%|████████▎ | 88/106 [00:04<00:00, 18.71batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:04<00:00, 18.77batch/s]

Training batches on cuda:  87%|████████▋ | 92/106 [00:04<00:00, 18.84batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:05<00:00, 18.72batch/s]

Training batches on cuda:  91%|█████████ | 96/106 [00:05<00:00, 18.78batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:05<00:00, 18.82batch/s]

Training batches on cuda:  94%|█████████▍| 100/106 [00:05<00:00, 18.88batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:05<00:00, 18.90batch/s]

Training batches on cuda:  98%|█████████▊| 104/106 [00:05<00:00, 18.77batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:05<00:00, 16.89batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:42<00:24,  6.10s/epoch, loss=0.000224, prev_loss=0.000266]
Training epochs on cuda:  70%|███████   | 7/10 [00:42<00:18,  6.06s/epoch, loss=0.000224, prev_loss=0.000266]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:07, 13.57batch/s]

Training batches on cuda:   4%|▍         | 4/106 [00:00<00:06, 14.66batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:06, 15.49batch/s]

Training batches on cuda:   8%|▊         | 8/106 [00:00<00:05, 16.58batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:05, 17.36batch/s]

Training batches on cuda:  11%|█▏        | 12/106 [00:00<00:05, 17.44batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:05, 17.77batch/s]

Training batches on cuda:  15%|█▌        | 16/106 [00:00<00:04, 18.23batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:01<00:04, 18.41batch/s]

Training batches on cuda:  19%|█▉        | 20/106 [00:01<00:04, 18.54batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:01<00:04, 18.59batch/s]

Training batches on cuda:  23%|██▎       | 24/106 [00:01<00:04, 18.67batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:01<00:04, 18.56batch/s]

Training batches on cuda:  26%|██▋       | 28/106 [00:01<00:04, 18.52batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:01<00:04, 18.52batch/s]

Training batches on cuda:  30%|███       | 32/106 [00:01<00:03, 18.55batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:03, 18.41batch/s]

Training batches on cuda:  34%|███▍      | 36/106 [00:01<00:03, 18.50batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:02<00:03, 18.28batch/s]

Training batches on cuda:  38%|███▊      | 40/106 [00:02<00:03, 18.25batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:02<00:03, 18.36batch/s]

Training batches on cuda:  42%|████▏     | 44/106 [00:02<00:03, 18.45batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:02<00:03, 18.57batch/s]

Training batches on cuda:  45%|████▌     | 48/106 [00:02<00:03, 18.48batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:02<00:03, 18.49batch/s]

Training batches on cuda:  49%|████▉     | 52/106 [00:02<00:02, 18.54batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:02<00:02, 18.34batch/s]

Training batches on cuda:  53%|█████▎    | 56/106 [00:03<00:02, 18.34batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:03<00:02, 18.73batch/s]

Training batches on cuda:  57%|█████▋    | 60/106 [00:03<00:02, 18.73batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:03<00:02, 18.78batch/s]

Training batches on cuda:  60%|██████    | 64/106 [00:03<00:02, 19.05batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:03<00:02, 18.88batch/s]

Training batches on cuda:  64%|██████▍   | 68/106 [00:03<00:02, 18.74batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:03<00:01, 18.73batch/s]

Training batches on cuda:  68%|██████▊   | 72/106 [00:03<00:01, 18.71batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:04<00:01, 18.59batch/s]

Training batches on cuda:  72%|███████▏  | 76/106 [00:04<00:01, 18.62batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:04<00:01, 18.78batch/s]

Training batches on cuda:  75%|███████▌  | 80/106 [00:04<00:01, 18.37batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:04<00:01, 17.78batch/s]

Training batches on cuda:  79%|███████▉  | 84/106 [00:04<00:01, 17.66batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:04<00:01, 17.84batch/s]

Training batches on cuda:  83%|████████▎ | 88/106 [00:04<00:01, 16.13batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:04<00:00, 16.62batch/s]

Training batches on cuda:  87%|████████▋ | 92/106 [00:05<00:00, 16.94batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:05<00:00, 17.22batch/s]

Training batches on cuda:  91%|█████████ | 96/106 [00:05<00:00, 17.32batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:05<00:00, 16.67batch/s]

Training batches on cuda:  94%|█████████▍| 100/106 [00:05<00:00, 17.19batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:05<00:00, 17.65batch/s]

Training batches on cuda:  98%|█████████▊| 104/106 [00:05<00:00, 18.04batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:05<00:00, 18.42batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:48<00:18,  6.06s/epoch, loss=0.000204, prev_loss=0.000224]
Training epochs on cuda:  80%|████████  | 8/10 [00:48<00:12,  6.06s/epoch, loss=0.000204, prev_loss=0.000224]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:07, 13.74batch/s]

Training batches on cuda:   4%|▍         | 4/106 [00:00<00:06, 14.95batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:06, 16.03batch/s]

Training batches on cuda:   8%|▊         | 8/106 [00:00<00:05, 16.76batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:05, 17.22batch/s]

Training batches on cuda:  11%|█▏        | 12/106 [00:00<00:05, 17.27batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:05, 17.35batch/s]

Training batches on cuda:  15%|█▌        | 16/106 [00:00<00:05, 17.87batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:01<00:04, 17.96batch/s]

Training batches on cuda:  19%|█▉        | 20/106 [00:01<00:04, 17.69batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:01<00:04, 18.09batch/s]

Training batches on cuda:  23%|██▎       | 24/106 [00:01<00:04, 17.91batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:01<00:04, 18.15batch/s]

Training batches on cuda:  26%|██▋       | 28/106 [00:01<00:04, 18.12batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:01<00:04, 18.03batch/s]

Training batches on cuda:  30%|███       | 32/106 [00:01<00:04, 18.22batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:03, 18.29batch/s]

Training batches on cuda:  34%|███▍      | 36/106 [00:02<00:03, 18.49batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:02<00:03, 18.59batch/s]

Training batches on cuda:  38%|███▊      | 40/106 [00:02<00:03, 18.64batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:02<00:03, 18.76batch/s]

Training batches on cuda:  42%|████▏     | 44/106 [00:02<00:03, 18.95batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:02<00:03, 19.00batch/s]

Training batches on cuda:  45%|████▌     | 48/106 [00:02<00:03, 18.81batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:02<00:02, 18.87batch/s]

Training batches on cuda:  49%|████▉     | 52/106 [00:02<00:02, 18.85batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:02<00:02, 18.69batch/s]

Training batches on cuda:  53%|█████▎    | 56/106 [00:03<00:02, 18.18batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:03<00:02, 17.94batch/s]

Training batches on cuda:  57%|█████▋    | 60/106 [00:03<00:02, 18.03batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:03<00:02, 18.11batch/s]

Training batches on cuda:  60%|██████    | 64/106 [00:03<00:02, 17.85batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:03<00:02, 18.04batch/s]

Training batches on cuda:  64%|██████▍   | 68/106 [00:03<00:02, 16.54batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:03<00:02, 17.04batch/s]

Training batches on cuda:  68%|██████▊   | 72/106 [00:04<00:01, 17.07batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:04<00:01, 17.45batch/s]

Training batches on cuda:  72%|███████▏  | 76/106 [00:04<00:01, 17.65batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:04<00:01, 17.99batch/s]

Training batches on cuda:  75%|███████▌  | 80/106 [00:04<00:01, 18.40batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:04<00:01, 17.91batch/s]

Training batches on cuda:  79%|███████▉  | 84/106 [00:04<00:01, 18.27batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:04<00:01, 18.42batch/s]

Training batches on cuda:  83%|████████▎ | 88/106 [00:04<00:00, 18.56batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:04<00:00, 18.46batch/s]

Training batches on cuda:  87%|████████▋ | 92/106 [00:05<00:00, 18.60batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:05<00:00, 18.63batch/s]

Training batches on cuda:  91%|█████████ | 96/106 [00:05<00:00, 18.73batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:05<00:00, 18.68batch/s]

Training batches on cuda:  94%|█████████▍| 100/106 [00:05<00:00, 18.67batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:05<00:00, 18.68batch/s]

Training batches on cuda:  98%|█████████▊| 104/106 [00:05<00:00, 18.73batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:05<00:00, 18.55batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:54<00:12,  6.06s/epoch, loss=0.000201, prev_loss=0.000204]
Training epochs on cuda:  90%|█████████ | 9/10 [00:54<00:06,  6.06s/epoch, loss=0.000201, prev_loss=0.000204]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:11,  9.35batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:09, 10.98batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:08, 12.59batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:07, 14.08batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:06, 15.25batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:06, 15.78batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:00<00:05, 16.30batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:05, 16.93batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:00<00:05, 17.07batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:04, 17.58batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:04, 17.37batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:04, 17.70batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:04, 17.95batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:01<00:04, 18.05batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:01<00:04, 18.26batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:01<00:04, 18.04batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:04, 18.23batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:03, 18.49batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:03, 18.21batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:02<00:03, 17.82batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:02<00:03, 18.00batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:02<00:03, 18.09batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:02<00:03, 17.25batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:02<00:03, 17.61batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:02<00:03, 16.33batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:02<00:03, 16.99batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:03<00:03, 17.36batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:03<00:02, 17.67batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:03<00:02, 17.43batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:03<00:02, 17.84batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:03<00:02, 17.90batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:03<00:02, 17.91batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:03<00:02, 17.85batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:03<00:02, 17.94batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:03<00:02, 17.87batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:04<00:01, 17.86batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:04<00:01, 17.93batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:04<00:01, 17.97batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:04<00:01, 17.76batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:04<00:01, 17.51batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:04<00:01, 17.72batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:04<00:01, 17.79batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:04<00:01, 17.93batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:04<00:01, 17.64batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:05<00:00, 17.25batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:05<00:00, 16.64batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:05<00:00, 17.04batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:05<00:00, 17.31batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:05<00:00, 17.54batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:05<00:00, 17.50batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:05<00:00, 17.77batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:05<00:00, 17.96batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:05<00:00, 18.15batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:01<00:06,  6.06s/epoch, loss=0.000189, prev_loss=0.000201]2021-09-16:18:20:31,828 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:20:32,883 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:20:47,927 INFO     [evaluator.py:657] Evaluation took 15.04s seconds
2021-09-16:18:20:48,64 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00, 11.01s/epoch, loss=0.000189, prev_loss=0.000201]
Training epochs on cuda: 100%|██████████| 10/10 [01:17<00:00,  7.73s/epoch, loss=0.000189, prev_loss=0.000201]
2021-09-16:18:20:48,65 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpf2gpvh7r'
2021-09-16:18:20:48,120 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpf2gpvh7r' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.26ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.36ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.49ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.56ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.52ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.55ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.58ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.60ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.61ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:21:02,990 INFO     [evaluator.py:657] Evaluation took 14.86s seconds
[I 2021-09-16 18:21:03,035] Trial 42 finished with value: 0.9587108845768406 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.01249533079390941, 'optimizer.lr': 0.05099385011058832, 'negative_sampler.num_negs_per_pos': 33, 'training.batch_size': 2932}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:21:03,59 WARNING  [api.py:823] No random seed is specified. Setting to 1776216214.
2021-09-16:18:21:03,476 INFO     [training_loop.py:972] Starting sub_batch_size search for training now...
2021-09-16:18:21:03,995 INFO     [training_loop.py:1004] Concluded search with sub_batch_size 1516.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:13,  7.68batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:11,  8.87batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:09,  9.97batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 10.90batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:08, 11.65batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 12.20batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:07, 12.64batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:06, 12.89batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 13.21batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 13.23batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 13.22batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:06, 13.26batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.27batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.33batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.39batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.22batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.36batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:05, 13.46batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:04, 13.53batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.61batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 13.66batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 13.75batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.69batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.65batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:03, 13.72batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 13.57batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:03<00:03, 13.58batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.64batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.67batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.56batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.62batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 13.51batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.51batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:04<00:02, 13.65batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.72batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.74batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.84batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.78batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.81batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.75batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:05<00:01, 13.68batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 13.70batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.67batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.72batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.80batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.78batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.91batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 14.01batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.90batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.74batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.70batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.82batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:07<?, ?epoch/s, loss=0.000439, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:07<01:09,  7.77s/epoch, loss=0.000439, prev_loss=nan]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:10,  9.40batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:09, 10.30batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:08, 11.03batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 11.67batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:07, 12.22batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 12.59batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:07, 12.82batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:06, 13.11batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 13.23batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 13.32batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 13.35batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:06, 13.29batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.22batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.47batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.61batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.66batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.55batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:05, 13.51batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:04, 13.52batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.56batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 13.65batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 13.59batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.59batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.44batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:04, 13.33batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 13.22batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:03<00:03, 13.22batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.28batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.38batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.38batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.34batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 13.53batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.41batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:05<00:02, 13.40batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.46batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.44batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.69batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.87batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.93batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.85batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:06<00:01, 13.93batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 14.03batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.64batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.43batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.35batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.47batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.46batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 13.54batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.62batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.69batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.81batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.62batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:15<01:09,  7.77s/epoch, loss=0.000437, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:15<01:02,  7.79s/epoch, loss=0.000437, prev_loss=nan]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:12,  8.13batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:10,  9.14batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:09, 10.08batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 10.93batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:08, 11.62batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 11.96batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:01<00:07, 12.12batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:07, 12.54batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 12.86batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 13.04batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 13.18batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:06, 13.30batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.36batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.49batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.59batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.62batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.66batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:04, 13.66batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:04, 13.52batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.60batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 13.48batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 13.47batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.59batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.66batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:03, 13.60batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:04, 13.00batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:04<00:03, 13.19batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.25batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.37batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.33batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.38batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 13.45batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.55batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:05<00:02, 13.46batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.58batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.66batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.62batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.77batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.81batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.55batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:06<00:01, 13.61batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 13.69batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.63batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.38batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.43batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.32batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.10batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 13.13batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.15batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.30batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.40batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.38batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:23<01:02,  7.79s/epoch, loss=0.000437, prev_loss=0.000437]
Training epochs on cuda:  30%|███       | 3/10 [00:23<00:55,  7.86s/epoch, loss=0.000437, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:15,  6.58batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:12,  7.77batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:10,  8.95batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:09, 10.00batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:08, 10.86batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:08, 11.15batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:01<00:07, 11.71batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:07, 12.23batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 12.69batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 12.90batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 12.96batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:06, 13.17batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.31batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.47batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.45batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.44batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.57batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:04, 13.61batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:05, 13.19batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.33batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 13.50batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 13.62batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.64batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.68batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:03, 13.62batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 13.68batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:04<00:03, 13.67batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.63batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.65batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.67batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.73batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 13.73batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.74batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:05<00:02, 13.79batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.77batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.75batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.75batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.53batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.30batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 12.52batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:06<00:01, 12.77batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 13.07batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.21batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.19batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.37batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.38batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.52batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 13.64batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.50batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.57batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.71batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.77batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:31<00:55,  7.86s/epoch, loss=0.000437, prev_loss=0.000437]
Training epochs on cuda:  40%|████      | 4/10 [00:31<00:47,  7.87s/epoch, loss=0.000437, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:12,  8.03batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:10,  9.10batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:09, 10.13batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 10.70batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:08, 11.52batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 12.12batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:01<00:07, 12.59batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:06, 12.69batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 13.02batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 12.99batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 12.90batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:06, 13.16batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.11batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.36batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.47batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.53batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.59batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:05, 13.51batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:04, 13.54batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.50batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 12.53batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 12.86batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.11batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.28batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:04, 13.33batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 13.25batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:04<00:03, 13.20batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.13batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.11batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.35batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.49batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:03, 13.32batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.42batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:05<00:02, 13.08batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.40batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.56batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.68batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.73batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.80batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.81batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:06<00:01, 13.80batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 13.83batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.89batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.93batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.86batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.86batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.78batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 13.85batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.88batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.89batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.84batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.81batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:39<00:47,  7.87s/epoch, loss=0.000437, prev_loss=0.000437]
Training epochs on cuda:  50%|█████     | 5/10 [00:39<00:39,  7.89s/epoch, loss=0.000437, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:12,  8.00batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:10,  9.10batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:09, 10.10batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 10.90batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:08, 11.51batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 12.11batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:01<00:07, 12.48batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:06, 12.81batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 13.06batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 13.28batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 13.16batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:06, 13.24batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.40batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.58batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.75batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.66batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.67batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:04, 13.73batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:04, 13.84batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.90batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 13.89batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 13.89batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.96batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.94batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:03, 13.95batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 13.98batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:03<00:03, 13.98batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.76batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.70batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.79batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.72batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 13.54batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.54batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:04<00:02, 13.59batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.71batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.09batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.30batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.49batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.65batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.59batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:05<00:01, 13.64batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 13.49batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.43batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.51batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.44batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.40batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.30batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 13.28batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.42batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.54batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.55batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.70batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:47<00:39,  7.89s/epoch, loss=0.000437, prev_loss=0.000437]
Training epochs on cuda:  60%|██████    | 6/10 [00:47<00:31,  7.87s/epoch, loss=0.000437, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:12,  8.46batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:10,  9.56batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:09, 10.54batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 11.34batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:07, 12.03batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 12.46batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:07, 12.83batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:06, 13.19batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 13.01batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 13.02batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 12.90batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:06, 13.02batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.10batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.26batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.23batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.30batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.34batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:05, 13.23batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:05, 13.14batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.27batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 13.18batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 13.31batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.49batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.51batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:03, 13.53batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 13.35batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:03<00:03, 13.41batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.48batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.39batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.45batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.50batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 13.56batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.45batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:05<00:02, 13.51batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.40batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.38batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.44batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.19batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.36batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.13batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:06<00:01, 13.37batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 13.55batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.66batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.71batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.76batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.79batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.71batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 13.77batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.75batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.46batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.65batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.75batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:55<00:31,  7.87s/epoch, loss=0.000437, prev_loss=0.000437]
Training epochs on cuda:  70%|███████   | 7/10 [00:55<00:23,  7.88s/epoch, loss=0.000437, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:12,  8.04batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:11,  9.09batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:09, 10.09batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 10.95batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:08, 11.68batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 12.24batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:01<00:07, 12.70batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:06, 13.00batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 13.29batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 13.38batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 13.42batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:05, 13.38batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.37batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.36batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.25batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.43batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.56batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:04, 13.67batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:04, 13.70batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.67batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 13.66batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 13.64batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.41batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.26batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:04, 13.04batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 13.05batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:03<00:03, 13.34batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.58batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.71batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.85batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.62batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 13.70batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.75batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:04<00:02, 13.75batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.87batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.69batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.55batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.63batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.55batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.63batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:06<00:01, 13.48batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 13.64batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.73batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.79batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.70batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.81batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.83batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 13.59batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.67batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.74batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.81batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.80batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [01:03<00:23,  7.88s/epoch, loss=0.000437, prev_loss=0.000437]
Training epochs on cuda:  80%|████████  | 8/10 [01:03<00:15,  7.88s/epoch, loss=0.000437, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:13,  7.78batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:11,  8.99batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:09, 10.18batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 11.22batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:07, 12.08batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 12.78batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:06, 13.31batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:06, 13.68batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 13.94batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:05, 14.15batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:05, 14.29batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:05, 14.41batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.31batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:01<00:05, 13.70batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.96batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 14.16batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:04, 14.28batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:04, 14.39batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:04, 14.47batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 14.52batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:02<00:04, 14.42batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 14.48batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:03, 14.53batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:03, 14.60batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:03, 14.61batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 14.60batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:03<00:03, 14.62batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:03<00:03, 14.64batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:03<00:03, 14.65batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 14.49batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:02, 14.36batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 14.22batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 14.11batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:04<00:02, 14.02batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:04<00:02, 13.98batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.86batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.86batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.84batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.84batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.69batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:05<00:01, 13.75batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:05<00:01, 13.59batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.47batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.41batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.37batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.35batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.38batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:06<00:00, 12.60batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:06<00:00, 12.86batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 12.99batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.20batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.46batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:10<00:15,  7.88s/epoch, loss=0.000436, prev_loss=0.000437]
Training epochs on cuda:  90%|█████████ | 9/10 [01:10<00:07,  7.80s/epoch, loss=0.000436, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/103 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/103 [00:00<00:11,  8.78batch/s]

Training batches on cuda:   3%|▎         | 3/103 [00:00<00:10,  9.78batch/s]

Training batches on cuda:   5%|▍         | 5/103 [00:00<00:09, 10.63batch/s]

Training batches on cuda:   7%|▋         | 7/103 [00:00<00:08, 11.40batch/s]

Training batches on cuda:   9%|▊         | 9/103 [00:00<00:07, 12.03batch/s]

Training batches on cuda:  11%|█         | 11/103 [00:00<00:07, 12.43batch/s]

Training batches on cuda:  13%|█▎        | 13/103 [00:00<00:07, 12.85batch/s]

Training batches on cuda:  15%|█▍        | 15/103 [00:01<00:06, 13.05batch/s]

Training batches on cuda:  17%|█▋        | 17/103 [00:01<00:06, 13.29batch/s]

Training batches on cuda:  18%|█▊        | 19/103 [00:01<00:06, 13.45batch/s]

Training batches on cuda:  20%|██        | 21/103 [00:01<00:06, 13.50batch/s]

Training batches on cuda:  22%|██▏       | 23/103 [00:01<00:05, 13.51batch/s]

Training batches on cuda:  24%|██▍       | 25/103 [00:01<00:05, 13.60batch/s]

Training batches on cuda:  26%|██▌       | 27/103 [00:02<00:05, 13.72batch/s]

Training batches on cuda:  28%|██▊       | 29/103 [00:02<00:05, 13.69batch/s]

Training batches on cuda:  30%|███       | 31/103 [00:02<00:05, 13.78batch/s]

Training batches on cuda:  32%|███▏      | 33/103 [00:02<00:05, 13.49batch/s]

Training batches on cuda:  34%|███▍      | 35/103 [00:02<00:05, 13.42batch/s]

Training batches on cuda:  36%|███▌      | 37/103 [00:02<00:04, 13.53batch/s]

Training batches on cuda:  38%|███▊      | 39/103 [00:02<00:04, 13.61batch/s]

Training batches on cuda:  40%|███▉      | 41/103 [00:03<00:04, 13.67batch/s]

Training batches on cuda:  42%|████▏     | 43/103 [00:03<00:04, 13.74batch/s]

Training batches on cuda:  44%|████▎     | 45/103 [00:03<00:04, 13.75batch/s]

Training batches on cuda:  46%|████▌     | 47/103 [00:03<00:04, 13.74batch/s]

Training batches on cuda:  48%|████▊     | 49/103 [00:03<00:03, 13.72batch/s]

Training batches on cuda:  50%|████▉     | 51/103 [00:03<00:03, 13.73batch/s]

Training batches on cuda:  51%|█████▏    | 53/103 [00:03<00:03, 13.65batch/s]

Training batches on cuda:  53%|█████▎    | 55/103 [00:04<00:03, 13.59batch/s]

Training batches on cuda:  55%|█████▌    | 57/103 [00:04<00:03, 13.05batch/s]

Training batches on cuda:  57%|█████▋    | 59/103 [00:04<00:03, 13.21batch/s]

Training batches on cuda:  59%|█████▉    | 61/103 [00:04<00:03, 13.41batch/s]

Training batches on cuda:  61%|██████    | 63/103 [00:04<00:02, 13.49batch/s]

Training batches on cuda:  63%|██████▎   | 65/103 [00:04<00:02, 13.59batch/s]

Training batches on cuda:  65%|██████▌   | 67/103 [00:04<00:02, 13.60batch/s]

Training batches on cuda:  67%|██████▋   | 69/103 [00:05<00:02, 13.68batch/s]

Training batches on cuda:  69%|██████▉   | 71/103 [00:05<00:02, 13.79batch/s]

Training batches on cuda:  71%|███████   | 73/103 [00:05<00:02, 13.77batch/s]

Training batches on cuda:  73%|███████▎  | 75/103 [00:05<00:02, 13.78batch/s]

Training batches on cuda:  75%|███████▍  | 77/103 [00:05<00:01, 13.78batch/s]

Training batches on cuda:  77%|███████▋  | 79/103 [00:05<00:01, 13.62batch/s]

Training batches on cuda:  79%|███████▊  | 81/103 [00:05<00:01, 13.69batch/s]

Training batches on cuda:  81%|████████  | 83/103 [00:06<00:01, 13.73batch/s]

Training batches on cuda:  83%|████████▎ | 85/103 [00:06<00:01, 13.75batch/s]

Training batches on cuda:  84%|████████▍ | 87/103 [00:06<00:01, 13.79batch/s]

Training batches on cuda:  86%|████████▋ | 89/103 [00:06<00:01, 13.60batch/s]

Training batches on cuda:  88%|████████▊ | 91/103 [00:06<00:00, 13.70batch/s]

Training batches on cuda:  90%|█████████ | 93/103 [00:06<00:00, 13.75batch/s]

Training batches on cuda:  92%|█████████▏| 95/103 [00:07<00:00, 13.73batch/s]

Training batches on cuda:  94%|█████████▍| 97/103 [00:07<00:00, 13.67batch/s]

Training batches on cuda:  96%|█████████▌| 99/103 [00:07<00:00, 13.55batch/s]

Training batches on cuda:  98%|█████████▊| 101/103 [00:07<00:00, 13.62batch/s]

Training batches on cuda: 100%|██████████| 103/103 [00:07<00:00, 14.45batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:18<00:07,  7.80s/epoch, loss=0.000434, prev_loss=0.000436]2021-09-16:18:22:22,535 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:22:23,644 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:22:38,665 INFO     [evaluator.py:657] Evaluation took 15.02s seconds
2021-09-16:18:22:38,830 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:34<00:00, 12.71s/epoch, loss=0.000434, prev_loss=0.000436]
Training epochs on cuda: 100%|██████████| 10/10 [01:34<00:00,  9.48s/epoch, loss=0.000434, prev_loss=0.000436]
2021-09-16:18:22:38,831 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpv135yece'
2021-09-16:18:22:38,875 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpv135yece' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.51ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:22:53,749 INFO     [evaluator.py:657] Evaluation took 14.85s seconds
[I 2021-09-16 18:22:53,754] Trial 43 finished with value: 0.22144956972524954 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.06688293722775768, 'optimizer.lr': 0.0588932490643993, 'negative_sampler.num_negs_per_pos': 81, 'training.batch_size': 3032}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:22:53,776 WARNING  [api.py:823] No random seed is specified. Setting to 4105435867.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:11,  8.14batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:09,  9.51batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:07, 10.93batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:06, 12.12batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 13.08batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:05, 13.81batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 14.26batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:00<00:05, 14.84batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:04, 15.18batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 15.48batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 15.72batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 15.61batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.82batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.90batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:03, 15.87batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:01<00:03, 15.90batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 15.92batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 15.89batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.50batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.71batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 15.90batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.65batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:02<00:02, 15.56batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.67batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.55batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.56batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.53batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.55batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.56batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.48batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:03<00:01, 15.60batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.66batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.68batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.31batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.47batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 15.56batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 15.55batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 15.52batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:04<00:00, 15.67batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.62batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.59batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.67batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.73batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.73batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 15.76batch/s]

Training batches on cuda: 100%|██████████| 91/91 [00:05<00:00, 16.78batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:06<?, ?epoch/s, loss=0.00021, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:54,  6.05s/epoch, loss=0.00021, prev_loss=nan]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:10,  8.30batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:09,  9.47batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:08, 10.41batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:07, 11.04batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 11.99batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:06, 12.46batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 13.26batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:01<00:05, 13.77batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:05, 14.24batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 14.50batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 14.60batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 14.85batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.00batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 14.82batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:02<00:04, 15.07batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 15.38batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 15.35batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 15.58batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.65batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.37batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 15.35batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.18batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:03<00:03, 15.28batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.29batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.40batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.39batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.39batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.42batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.33batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.34batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:04<00:01, 15.35batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.50batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.57batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.62batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.67batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 15.81batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 15.72batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 15.74batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:05<00:00, 15.63batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.28batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.16batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.01batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.07batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.10batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 14.93batch/s]

Training batches on cuda: 100%|██████████| 91/91 [00:06<00:00, 15.92batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:12<00:54,  6.05s/epoch, loss=0.000205, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:49,  6.13s/epoch, loss=0.000205, prev_loss=nan]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:11,  7.59batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:09,  8.97batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:08, 10.33batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:07, 11.53batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 12.55batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:05, 13.42batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 13.86batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:01<00:05, 14.60batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:04, 14.98batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 15.36batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 15.60batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 15.54batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.56batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.46batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:04, 15.48batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 15.49batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 15.57batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 15.67batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.70batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.73batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 15.76batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.77batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:02<00:02, 15.81batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.89batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 16.02batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 16.07batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 16.00batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.92batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.86batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.81batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:03<00:01, 15.81batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.77batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.81batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.80batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.79batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 15.84batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 15.51batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 15.30batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:04<00:00, 15.47batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.57batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.51batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.70batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.61batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.50batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 15.75batch/s]

Training batches on cuda: 100%|██████████| 91/91 [00:05<00:00, 16.64batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:18<00:49,  6.13s/epoch, loss=0.000205, prev_loss=0.000205]
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:42,  6.11s/epoch, loss=0.000205, prev_loss=0.000205]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:11,  8.09batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:09,  9.52batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:07, 10.90batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:07, 11.92batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 13.00batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:05, 13.87batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 14.55batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:00<00:05, 14.97batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:04, 15.31batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 15.40batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 15.34batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 15.43batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.57batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.63batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:03, 15.70batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 15.77batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 15.77batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 15.73batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.76batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.81batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 15.67batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.73batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:02<00:02, 15.52batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.74batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.87batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.91batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.72batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.78batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.80batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.81batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:03<00:01, 15.81batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.81batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.83batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.92batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.66batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 15.51batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 15.55batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 15.58batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:04<00:00, 15.62batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.50batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.49batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.50batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.52batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.53batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 15.53batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:24<00:42,  6.11s/epoch, loss=0.000205, prev_loss=0.000205]
Training epochs on cuda:  40%|████      | 4/10 [00:24<00:36,  6.11s/epoch, loss=0.000205, prev_loss=0.000205]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:11,  7.52batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:09,  8.86batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:08, 10.02batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:07, 11.22batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 12.20batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:06, 13.09batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 13.83batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:01<00:05, 14.38batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:05, 14.67batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 15.00batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 15.10batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 15.32batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.47batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.53batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:03, 15.64batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 15.71batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 15.64batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 15.46batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.57batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.68batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 15.76batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.73batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:02<00:02, 15.78batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.81batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.81batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.76batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.79batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.80batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.82batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.83batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:03<00:01, 15.84batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.79batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.58batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.62batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.50batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 15.54batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 15.65batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 15.65batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:04<00:00, 15.60batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.64batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.58batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.63batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.63batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.39batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 15.49batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:30<00:36,  6.11s/epoch, loss=0.000205, prev_loss=0.000205]
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:30,  6.11s/epoch, loss=0.000205, prev_loss=0.000205]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:11,  7.96batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:09,  9.26batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:08, 10.53batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:07, 11.68batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 12.75batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:05, 13.71batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 14.47batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:01<00:05, 14.86batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:04, 15.09batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 15.30batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 15.44batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 15.42batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.64batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.84batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:03, 16.10batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 16.22batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 16.22batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 16.12batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 16.00batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.92batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 16.09batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:02, 16.19batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:02<00:02, 16.29batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.89batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.83batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.36batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.33batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.49batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.20batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 14.91batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:03<00:02, 13.92batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 14.25batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 14.52batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 14.44batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 14.28batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 14.16batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 14.51batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 14.88batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:05<00:00, 15.15batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.45batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.28batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 14.97batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 14.80batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 14.83batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 14.87batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:36<00:30,  6.11s/epoch, loss=0.000205, prev_loss=0.000205]
Training epochs on cuda:  60%|██████    | 6/10 [00:36<00:24,  6.15s/epoch, loss=0.000205, prev_loss=0.000205]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:14,  6.18batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:11,  7.52batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:09,  8.94batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:08, 10.30batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:07, 11.51batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:06, 12.54batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 13.31batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:01<00:05, 13.95batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:05, 14.60batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 15.00batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 15.33batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 15.29batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.15batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.30batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:04, 15.45batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 15.28batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 15.43batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 15.53batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.62batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.70batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 14.63batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.00batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:02<00:03, 15.21batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.29batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.44batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.59batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.64batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.69batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.69batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.64batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:04<00:01, 15.69batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.72batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.56batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.58batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.25batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 15.45batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 14.81batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 14.80batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:05<00:00, 14.96batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.18batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.31batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.38batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.52batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.60batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 15.71batch/s]

Training batches on cuda: 100%|██████████| 91/91 [00:05<00:00, 16.52batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [00:43<00:24,  6.15s/epoch, loss=0.000205, prev_loss=0.000205]
Training epochs on cuda:  70%|███████   | 7/10 [00:43<00:18,  6.17s/epoch, loss=0.000205, prev_loss=0.000205]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:12,  7.22batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:10,  8.55batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:08,  9.83batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:07, 11.01batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 12.07batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:06, 12.99batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 13.63batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:01<00:05, 14.18batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:05, 14.66batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 14.90batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 14.76batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 14.98batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.18batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.33batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:04, 15.34batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 15.43batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 15.49batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 15.44batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.29batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.46batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 15.45batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.41batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:03<00:03, 14.99batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.22batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.37batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.49batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.52batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.39batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.42batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.40batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:04<00:01, 15.41batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.45batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.39batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.45batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.33batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 15.68batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 15.80batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 15.86batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:05<00:00, 15.95batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.69batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.69batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.48batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.55batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.57batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 15.54batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [00:49<00:18,  6.17s/epoch, loss=0.000205, prev_loss=0.000205]
Training epochs on cuda:  80%|████████  | 8/10 [00:49<00:12,  6.19s/epoch, loss=0.000205, prev_loss=0.000205]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:12,  7.31batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:10,  8.66batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:08,  9.97batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:07, 11.12batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 12.02batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:06, 12.89batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 13.51batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:01<00:05, 13.94batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:05, 14.27batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 14.59batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 14.86batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 14.96batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.00batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.14batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:04, 15.18batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 15.33batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 14.87batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 14.98batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.19batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.24batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 15.14batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.18batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:03<00:02, 15.39batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.48batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.57batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.65batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.77batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.61batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.77batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.71batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:04<00:01, 15.64batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.76batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.37batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.21batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.27batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 14.87batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 14.96batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 15.09batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:05<00:00, 15.14batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.08batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.19batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.25batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.26batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.28batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 15.25batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [00:55<00:12,  6.19s/epoch, loss=0.000205, prev_loss=0.000205]
Training epochs on cuda:  90%|█████████ | 9/10 [00:55<00:06,  6.19s/epoch, loss=0.000205, prev_loss=0.000205]

Training batches on cuda:   0%|          | 0/91 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/91 [00:00<00:11,  7.91batch/s]

Training batches on cuda:   3%|▎         | 3/91 [00:00<00:09,  9.18batch/s]

Training batches on cuda:   5%|▌         | 5/91 [00:00<00:08, 10.45batch/s]

Training batches on cuda:   8%|▊         | 7/91 [00:00<00:07, 11.57batch/s]

Training batches on cuda:  10%|▉         | 9/91 [00:00<00:06, 12.54batch/s]

Training batches on cuda:  12%|█▏        | 11/91 [00:00<00:05, 13.37batch/s]

Training batches on cuda:  14%|█▍        | 13/91 [00:00<00:05, 13.98batch/s]

Training batches on cuda:  16%|█▋        | 15/91 [00:01<00:05, 14.47batch/s]

Training batches on cuda:  19%|█▊        | 17/91 [00:01<00:04, 14.84batch/s]

Training batches on cuda:  21%|██        | 19/91 [00:01<00:04, 15.07batch/s]

Training batches on cuda:  23%|██▎       | 21/91 [00:01<00:04, 15.45batch/s]

Training batches on cuda:  25%|██▌       | 23/91 [00:01<00:04, 15.48batch/s]

Training batches on cuda:  27%|██▋       | 25/91 [00:01<00:04, 15.53batch/s]

Training batches on cuda:  30%|██▉       | 27/91 [00:01<00:04, 15.51batch/s]

Training batches on cuda:  32%|███▏      | 29/91 [00:01<00:03, 15.53batch/s]

Training batches on cuda:  34%|███▍      | 31/91 [00:02<00:03, 15.60batch/s]

Training batches on cuda:  36%|███▋      | 33/91 [00:02<00:03, 15.62batch/s]

Training batches on cuda:  38%|███▊      | 35/91 [00:02<00:03, 15.57batch/s]

Training batches on cuda:  41%|████      | 37/91 [00:02<00:03, 15.59batch/s]

Training batches on cuda:  43%|████▎     | 39/91 [00:02<00:03, 15.62batch/s]

Training batches on cuda:  45%|████▌     | 41/91 [00:02<00:03, 15.45batch/s]

Training batches on cuda:  47%|████▋     | 43/91 [00:02<00:03, 15.47batch/s]

Training batches on cuda:  49%|████▉     | 45/91 [00:02<00:02, 15.62batch/s]

Training batches on cuda:  52%|█████▏    | 47/91 [00:03<00:02, 15.69batch/s]

Training batches on cuda:  54%|█████▍    | 49/91 [00:03<00:02, 15.75batch/s]

Training batches on cuda:  56%|█████▌    | 51/91 [00:03<00:02, 15.78batch/s]

Training batches on cuda:  58%|█████▊    | 53/91 [00:03<00:02, 15.81batch/s]

Training batches on cuda:  60%|██████    | 55/91 [00:03<00:02, 15.81batch/s]

Training batches on cuda:  63%|██████▎   | 57/91 [00:03<00:02, 15.82batch/s]

Training batches on cuda:  65%|██████▍   | 59/91 [00:03<00:02, 15.84batch/s]

Training batches on cuda:  67%|██████▋   | 61/91 [00:03<00:01, 15.87batch/s]

Training batches on cuda:  69%|██████▉   | 63/91 [00:04<00:01, 15.92batch/s]

Training batches on cuda:  71%|███████▏  | 65/91 [00:04<00:01, 15.93batch/s]

Training batches on cuda:  74%|███████▎  | 67/91 [00:04<00:01, 15.91batch/s]

Training batches on cuda:  76%|███████▌  | 69/91 [00:04<00:01, 15.92batch/s]

Training batches on cuda:  78%|███████▊  | 71/91 [00:04<00:01, 15.89batch/s]

Training batches on cuda:  80%|████████  | 73/91 [00:04<00:01, 15.81batch/s]

Training batches on cuda:  82%|████████▏ | 75/91 [00:04<00:01, 15.78batch/s]

Training batches on cuda:  85%|████████▍ | 77/91 [00:04<00:00, 15.74batch/s]

Training batches on cuda:  87%|████████▋ | 79/91 [00:05<00:00, 15.47batch/s]

Training batches on cuda:  89%|████████▉ | 81/91 [00:05<00:00, 15.47batch/s]

Training batches on cuda:  91%|█████████ | 83/91 [00:05<00:00, 15.47batch/s]

Training batches on cuda:  93%|█████████▎| 85/91 [00:05<00:00, 15.48batch/s]

Training batches on cuda:  96%|█████████▌| 87/91 [00:05<00:00, 15.31batch/s]

Training batches on cuda:  98%|█████████▊| 89/91 [00:05<00:00, 15.22batch/s]

Training batches on cuda: 100%|██████████| 91/91 [00:05<00:00, 16.23batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [01:01<00:06,  6.19s/epoch, loss=0.000205, prev_loss=0.000205]2021-09-16:18:23:56,194 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:23:57,505 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:24:12,505 INFO     [evaluator.py:657] Evaluation took 15.00s seconds
2021-09-16:18:24:12,646 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00, 11.12s/epoch, loss=0.000205, prev_loss=0.000205]
Training epochs on cuda: 100%|██████████| 10/10 [01:18<00:00,  7.82s/epoch, loss=0.000205, prev_loss=0.000205]
2021-09-16:18:24:12,647 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpg7hm02c7'
2021-09-16:18:24:12,717 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpg7hm02c7' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.62ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.43ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:24:27,639 INFO     [evaluator.py:657] Evaluation took 14.91s seconds
[I 2021-09-16 18:24:27,644] Trial 44 finished with value: -0.0011653854673367992 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.142558995510647, 'optimizer.lr': 0.050815716786102666, 'negative_sampler.num_negs_per_pos': 30, 'training.batch_size': 3432}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:24:27,664 WARNING  [api.py:823] No random seed is specified. Setting to 3869061515.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 15.78batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:07, 17.45batch/s]

Training batches on cuda:   6%|▋         | 8/128 [00:00<00:06, 18.95batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:05, 20.07batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:05, 20.99batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:05, 21.73batch/s]

Training batches on cuda:  16%|█▌        | 20/128 [00:00<00:04, 22.34batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:01<00:04, 22.76batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:01<00:04, 22.78batch/s]

Training batches on cuda:  23%|██▎       | 29/128 [00:01<00:04, 22.75batch/s]

Training batches on cuda:  25%|██▌       | 32/128 [00:01<00:04, 22.75batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:04, 22.75batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:03, 22.74batch/s]

Training batches on cuda:  32%|███▏      | 41/128 [00:01<00:03, 22.78batch/s]

Training batches on cuda:  34%|███▍      | 44/128 [00:01<00:03, 22.72batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:02<00:03, 22.73batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:02<00:03, 22.75batch/s]

Training batches on cuda:  41%|████▏     | 53/128 [00:02<00:03, 22.80batch/s]

Training batches on cuda:  44%|████▍     | 56/128 [00:02<00:03, 22.71batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:02<00:03, 22.66batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:02<00:02, 22.72batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:02<00:02, 22.74batch/s]

Training batches on cuda:  53%|█████▎    | 68/128 [00:02<00:02, 22.71batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:03<00:02, 22.75batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:03<00:02, 22.66batch/s]

Training batches on cuda:  60%|██████    | 77/128 [00:03<00:02, 22.60batch/s]

Training batches on cuda:  62%|██████▎   | 80/128 [00:03<00:02, 22.57batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:03<00:01, 22.65batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:03<00:01, 22.64batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:03<00:01, 22.52batch/s]

Training batches on cuda:  72%|███████▏  | 92/128 [00:04<00:01, 22.47batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:04<00:01, 20.70batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:04<00:01, 21.21batch/s]

Training batches on cuda:  79%|███████▉  | 101/128 [00:04<00:01, 21.38batch/s]

Training batches on cuda:  81%|████████▏ | 104/128 [00:04<00:01, 21.64batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:04<00:00, 21.72batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:04<00:00, 21.44batch/s]

Training batches on cuda:  88%|████████▊ | 113/128 [00:05<00:00, 21.48batch/s]

Training batches on cuda:  91%|█████████ | 116/128 [00:05<00:00, 21.40batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:05<00:00, 21.14batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:05<00:00, 21.06batch/s]

Training batches on cuda:  98%|█████████▊| 125/128 [00:05<00:00, 21.13batch/s]

Training batches on cuda: 100%|██████████| 128/128 [00:05<00:00, 20.97batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.000591, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:53,  5.99s/epoch, loss=0.000591, prev_loss=nan]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/128 [00:00<00:16,  7.75batch/s]

Training batches on cuda:   3%|▎         | 4/128 [00:00<00:12,  9.62batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:10, 11.57batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:08, 13.27batch/s]

Training batches on cuda:  10%|█         | 13/128 [00:00<00:07, 14.99batch/s]

Training batches on cuda:  12%|█▎        | 16/128 [00:00<00:06, 16.80batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:05, 18.34batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:01<00:05, 19.29batch/s]

Training batches on cuda:  20%|█▉        | 25/128 [00:01<00:05, 20.00batch/s]

Training batches on cuda:  22%|██▏       | 28/128 [00:01<00:04, 20.52batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:01<00:04, 20.82batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:04, 21.28batch/s]

Training batches on cuda:  29%|██▉       | 37/128 [00:01<00:04, 21.30batch/s]

Training batches on cuda:  31%|███▏      | 40/128 [00:01<00:04, 21.24batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:02<00:03, 21.26batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:02<00:03, 21.38batch/s]

Training batches on cuda:  38%|███▊      | 49/128 [00:02<00:03, 21.20batch/s]

Training batches on cuda:  41%|████      | 52/128 [00:02<00:03, 21.58batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:02<00:03, 21.78batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:02<00:03, 22.04batch/s]

Training batches on cuda:  48%|████▊     | 61/128 [00:02<00:03, 22.07batch/s]

Training batches on cuda:  50%|█████     | 64/128 [00:03<00:02, 22.00batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:03<00:02, 22.08batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:03<00:02, 20.42batch/s]

Training batches on cuda:  57%|█████▋    | 73/128 [00:03<00:02, 20.76batch/s]

Training batches on cuda:  59%|█████▉    | 76/128 [00:03<00:02, 21.04batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:03<00:02, 21.43batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:03<00:02, 21.76batch/s]

Training batches on cuda:  66%|██████▋   | 85/128 [00:03<00:01, 21.85batch/s]

Training batches on cuda:  69%|██████▉   | 88/128 [00:04<00:01, 21.89batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:04<00:01, 22.01batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:04<00:01, 22.09batch/s]

Training batches on cuda:  76%|███████▌  | 97/128 [00:04<00:01, 21.88batch/s]

Training batches on cuda:  78%|███████▊  | 100/128 [00:04<00:01, 21.78batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:04<00:01, 21.92batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:04<00:01, 21.89batch/s]

Training batches on cuda:  85%|████████▌ | 109/128 [00:05<00:00, 22.01batch/s]

Training batches on cuda:  88%|████████▊ | 112/128 [00:05<00:00, 21.90batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:05<00:00, 22.11batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:05<00:00, 22.28batch/s]

Training batches on cuda:  95%|█████████▍| 121/128 [00:05<00:00, 22.11batch/s]

Training batches on cuda:  97%|█████████▋| 124/128 [00:05<00:00, 22.21batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:05<00:00, 21.95batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:12<00:53,  5.99s/epoch, loss=0.000557, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:48,  6.04s/epoch, loss=0.000557, prev_loss=nan]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:08, 14.82batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:07, 16.42batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:07, 17.27batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:06, 18.30batch/s]

Training batches on cuda:  10%|█         | 13/128 [00:00<00:05, 19.33batch/s]

Training batches on cuda:  12%|█▎        | 16/128 [00:00<00:05, 20.20batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:05, 20.57batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:01<00:05, 21.03batch/s]

Training batches on cuda:  20%|█▉        | 25/128 [00:01<00:04, 21.33batch/s]

Training batches on cuda:  22%|██▏       | 28/128 [00:01<00:04, 21.51batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:01<00:04, 21.71batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:04, 21.93batch/s]

Training batches on cuda:  29%|██▉       | 37/128 [00:01<00:04, 21.52batch/s]

Training batches on cuda:  31%|███▏      | 40/128 [00:01<00:04, 21.43batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:02<00:03, 21.27batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:02<00:04, 18.92batch/s]

Training batches on cuda:  38%|███▊      | 49/128 [00:02<00:04, 19.49batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:02<00:04, 19.09batch/s]

Training batches on cuda:  41%|████▏     | 53/128 [00:02<00:03, 19.08batch/s]

Training batches on cuda:  44%|████▍     | 56/128 [00:02<00:03, 19.63batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:02<00:03, 20.15batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:03<00:03, 20.30batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:03<00:03, 20.25batch/s]

Training batches on cuda:  53%|█████▎    | 68/128 [00:03<00:02, 20.71batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:03<00:02, 21.27batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:03<00:02, 21.65batch/s]

Training batches on cuda:  60%|██████    | 77/128 [00:03<00:02, 22.00batch/s]

Training batches on cuda:  62%|██████▎   | 80/128 [00:03<00:02, 21.96batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:03<00:02, 22.07batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:04<00:01, 22.24batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:04<00:01, 22.37batch/s]

Training batches on cuda:  72%|███████▏  | 92/128 [00:04<00:01, 22.28batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:04<00:01, 22.40batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:04<00:01, 22.36batch/s]

Training batches on cuda:  79%|███████▉  | 101/128 [00:04<00:01, 22.42batch/s]

Training batches on cuda:  81%|████████▏ | 104/128 [00:04<00:01, 22.50batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:05<00:00, 22.55batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:05<00:00, 22.66batch/s]

Training batches on cuda:  88%|████████▊ | 113/128 [00:05<00:00, 22.60batch/s]

Training batches on cuda:  91%|█████████ | 116/128 [00:05<00:00, 22.57batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:05<00:00, 22.46batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:05<00:00, 22.64batch/s]

Training batches on cuda:  98%|█████████▊| 125/128 [00:05<00:00, 22.70batch/s]

Training batches on cuda: 100%|██████████| 128/128 [00:05<00:00, 23.26batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:18<00:48,  6.04s/epoch, loss=0.000559, prev_loss=0.000557]
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:42,  6.08s/epoch, loss=0.000559, prev_loss=0.000557]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:09, 13.71batch/s]

Training batches on cuda:   3%|▎         | 4/128 [00:00<00:08, 15.13batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:07, 16.79batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:06, 18.19batch/s]

Training batches on cuda:  10%|█         | 13/128 [00:00<00:05, 19.31batch/s]

Training batches on cuda:  12%|█▎        | 16/128 [00:00<00:05, 20.26batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:05, 20.91batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:01<00:05, 20.02batch/s]

Training batches on cuda:  20%|█▉        | 25/128 [00:01<00:04, 20.81batch/s]

Training batches on cuda:  22%|██▏       | 28/128 [00:01<00:04, 21.29batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:01<00:04, 21.22batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:04, 21.54batch/s]

Training batches on cuda:  29%|██▉       | 37/128 [00:01<00:04, 21.74batch/s]

Training batches on cuda:  31%|███▏      | 40/128 [00:01<00:04, 21.90batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:02<00:03, 22.16batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:02<00:03, 22.27batch/s]

Training batches on cuda:  38%|███▊      | 49/128 [00:02<00:03, 22.36batch/s]

Training batches on cuda:  41%|████      | 52/128 [00:02<00:03, 22.50batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:02<00:03, 22.56batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:02<00:03, 22.67batch/s]

Training batches on cuda:  48%|████▊     | 61/128 [00:02<00:02, 22.66batch/s]

Training batches on cuda:  50%|█████     | 64/128 [00:02<00:02, 22.51batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:03<00:02, 22.54batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:03<00:02, 22.18batch/s]

Training batches on cuda:  57%|█████▋    | 73/128 [00:03<00:02, 22.14batch/s]

Training batches on cuda:  59%|█████▉    | 76/128 [00:03<00:02, 21.98batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:03<00:02, 21.99batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:03<00:02, 22.00batch/s]

Training batches on cuda:  66%|██████▋   | 85/128 [00:03<00:01, 22.01batch/s]

Training batches on cuda:  69%|██████▉   | 88/128 [00:04<00:01, 22.23batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:04<00:01, 22.36batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:04<00:01, 22.48batch/s]

Training batches on cuda:  76%|███████▌  | 97/128 [00:04<00:01, 22.50batch/s]

Training batches on cuda:  78%|███████▊  | 100/128 [00:04<00:01, 22.41batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:04<00:01, 22.49batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:04<00:00, 22.55batch/s]

Training batches on cuda:  85%|████████▌ | 109/128 [00:04<00:00, 22.53batch/s]

Training batches on cuda:  88%|████████▊ | 112/128 [00:05<00:00, 22.48batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:05<00:00, 22.56batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:05<00:00, 22.54batch/s]

Training batches on cuda:  95%|█████████▍| 121/128 [00:05<00:00, 22.66batch/s]

Training batches on cuda:  97%|█████████▋| 124/128 [00:05<00:00, 22.67batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:05<00:00, 22.63batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:24<00:42,  6.08s/epoch, loss=0.000559, prev_loss=0.000559]
Training epochs on cuda:  40%|████      | 4/10 [00:24<00:36,  6.08s/epoch, loss=0.000559, prev_loss=0.000559]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:08, 14.19batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:07, 15.95batch/s]

Training batches on cuda:   6%|▋         | 8/128 [00:00<00:06, 17.48batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:06, 18.75batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:05, 19.28batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:05, 19.93batch/s]

Training batches on cuda:  16%|█▌        | 20/128 [00:00<00:05, 20.86batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:01<00:04, 21.02batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:01<00:04, 20.99batch/s]

Training batches on cuda:  23%|██▎       | 29/128 [00:01<00:04, 21.65batch/s]

Training batches on cuda:  25%|██▌       | 32/128 [00:01<00:04, 22.11batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:04, 21.58batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:04, 21.32batch/s]

Training batches on cuda:  32%|███▏      | 41/128 [00:01<00:04, 21.10batch/s]

Training batches on cuda:  34%|███▍      | 44/128 [00:02<00:04, 20.84batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:02<00:03, 21.23batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:02<00:03, 21.90batch/s]

Training batches on cuda:  41%|████▏     | 53/128 [00:02<00:03, 22.45batch/s]

Training batches on cuda:  44%|████▍     | 56/128 [00:02<00:03, 22.85batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:02<00:02, 23.02batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:02<00:02, 23.23batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:02<00:02, 23.42batch/s]

Training batches on cuda:  53%|█████▎    | 68/128 [00:03<00:02, 23.66batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:03<00:02, 23.57batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:03<00:02, 23.07batch/s]

Training batches on cuda:  60%|██████    | 77/128 [00:03<00:02, 23.11batch/s]

Training batches on cuda:  62%|██████▎   | 80/128 [00:03<00:02, 22.97batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:03<00:01, 22.89batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:03<00:01, 22.84batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:04<00:01, 22.79batch/s]

Training batches on cuda:  72%|███████▏  | 92/128 [00:04<00:01, 22.64batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:04<00:01, 22.63batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:04<00:01, 22.64batch/s]

Training batches on cuda:  79%|███████▉  | 101/128 [00:04<00:01, 22.64batch/s]

Training batches on cuda:  81%|████████▏ | 104/128 [00:04<00:01, 22.68batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:04<00:00, 22.31batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:04<00:00, 21.90batch/s]

Training batches on cuda:  88%|████████▊ | 113/128 [00:05<00:00, 22.10batch/s]

Training batches on cuda:  91%|█████████ | 116/128 [00:05<00:00, 22.21batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:05<00:00, 22.08batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:05<00:00, 22.18batch/s]

Training batches on cuda:  98%|█████████▊| 125/128 [00:05<00:00, 22.31batch/s]

Training batches on cuda: 100%|██████████| 128/128 [00:05<00:00, 23.04batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:30<00:36,  6.08s/epoch, loss=0.000559, prev_loss=0.000559]
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:30,  6.05s/epoch, loss=0.000559, prev_loss=0.000559]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/128 [00:00<00:13,  9.12batch/s]

Training batches on cuda:   3%|▎         | 4/128 [00:00<00:11, 11.07batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:09, 13.10batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:07, 14.93batch/s]

Training batches on cuda:  10%|█         | 13/128 [00:00<00:06, 16.57batch/s]

Training batches on cuda:  12%|█▎        | 16/128 [00:00<00:06, 17.87batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:05, 19.08batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:01<00:05, 19.74batch/s]

Training batches on cuda:  20%|█▉        | 25/128 [00:01<00:05, 20.29batch/s]

Training batches on cuda:  22%|██▏       | 28/128 [00:01<00:04, 20.82batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:01<00:04, 21.05batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:04, 21.04batch/s]

Training batches on cuda:  29%|██▉       | 37/128 [00:01<00:04, 21.35batch/s]

Training batches on cuda:  31%|███▏      | 40/128 [00:01<00:04, 21.77batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:02<00:03, 21.86batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:02<00:03, 21.88batch/s]

Training batches on cuda:  38%|███▊      | 49/128 [00:02<00:03, 21.97batch/s]

Training batches on cuda:  41%|████      | 52/128 [00:02<00:03, 21.97batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:02<00:03, 21.44batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:02<00:03, 21.81batch/s]

Training batches on cuda:  48%|████▊     | 61/128 [00:02<00:03, 21.70batch/s]

Training batches on cuda:  50%|█████     | 64/128 [00:02<00:02, 21.93batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:03<00:02, 22.12batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:03<00:02, 22.17batch/s]

Training batches on cuda:  57%|█████▋    | 73/128 [00:03<00:02, 22.20batch/s]

Training batches on cuda:  59%|█████▉    | 76/128 [00:03<00:02, 22.33batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:03<00:02, 22.31batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:03<00:02, 21.96batch/s]

Training batches on cuda:  66%|██████▋   | 85/128 [00:03<00:01, 22.12batch/s]

Training batches on cuda:  69%|██████▉   | 88/128 [00:04<00:01, 21.76batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:04<00:01, 21.81batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:04<00:01, 22.00batch/s]

Training batches on cuda:  76%|███████▌  | 97/128 [00:04<00:01, 22.03batch/s]

Training batches on cuda:  78%|███████▊  | 100/128 [00:04<00:01, 21.91batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:04<00:01, 22.12batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:04<00:00, 22.14batch/s]

Training batches on cuda:  85%|████████▌ | 109/128 [00:05<00:00, 21.77batch/s]

Training batches on cuda:  88%|████████▊ | 112/128 [00:05<00:00, 20.44batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:05<00:00, 20.92batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:05<00:00, 21.40batch/s]

Training batches on cuda:  95%|█████████▍| 121/128 [00:05<00:00, 21.64batch/s]

Training batches on cuda:  97%|█████████▋| 124/128 [00:05<00:00, 21.82batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:05<00:00, 21.97batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:36<00:30,  6.05s/epoch, loss=0.000558, prev_loss=0.000559]
Training epochs on cuda:  60%|██████    | 6/10 [00:36<00:24,  6.07s/epoch, loss=0.000558, prev_loss=0.000559]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:08, 14.90batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:07, 16.71batch/s]

Training batches on cuda:   6%|▋         | 8/128 [00:00<00:06, 18.36batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:05, 19.72batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:05, 20.84batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:05, 21.71batch/s]

Training batches on cuda:  16%|█▌        | 20/128 [00:00<00:04, 22.35batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:01<00:04, 22.86batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:01<00:04, 23.14batch/s]

Training batches on cuda:  23%|██▎       | 29/128 [00:01<00:04, 23.44batch/s]

Training batches on cuda:  25%|██▌       | 32/128 [00:01<00:04, 23.58batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:03, 23.74batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:03, 23.77batch/s]

Training batches on cuda:  32%|███▏      | 41/128 [00:01<00:03, 23.77batch/s]

Training batches on cuda:  34%|███▍      | 44/128 [00:01<00:03, 23.88batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:02<00:03, 23.98batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:02<00:03, 23.96batch/s]

Training batches on cuda:  41%|████▏     | 53/128 [00:02<00:03, 24.02batch/s]

Training batches on cuda:  44%|████▍     | 56/128 [00:02<00:02, 24.05batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:02<00:02, 24.13batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:02<00:02, 24.16batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:02<00:02, 24.09batch/s]

Training batches on cuda:  53%|█████▎    | 68/128 [00:02<00:02, 24.11batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:03<00:02, 24.15batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:03<00:02, 24.11batch/s]

Training batches on cuda:  60%|██████    | 77/128 [00:03<00:02, 23.63batch/s]

Training batches on cuda:  62%|██████▎   | 80/128 [00:03<00:02, 23.80batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:03<00:01, 23.94batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:03<00:01, 24.03batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:03<00:01, 23.98batch/s]

Training batches on cuda:  72%|███████▏  | 92/128 [00:03<00:01, 23.94batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:04<00:01, 24.00batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:04<00:01, 24.08batch/s]

Training batches on cuda:  79%|███████▉  | 101/128 [00:04<00:01, 24.14batch/s]

Training batches on cuda:  81%|████████▏ | 104/128 [00:04<00:00, 24.20batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:04<00:00, 24.24batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:04<00:00, 23.94batch/s]

Training batches on cuda:  88%|████████▊ | 113/128 [00:04<00:00, 23.95batch/s]

Training batches on cuda:  91%|█████████ | 116/128 [00:04<00:00, 23.52batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:05<00:00, 23.22batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:05<00:00, 23.15batch/s]

Training batches on cuda:  98%|█████████▊| 125/128 [00:05<00:00, 22.94batch/s]

Training batches on cuda: 100%|██████████| 128/128 [00:05<00:00, 23.10batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:42<00:24,  6.07s/epoch, loss=0.000559, prev_loss=0.000558]
Training epochs on cuda:  70%|███████   | 7/10 [00:42<00:17,  5.96s/epoch, loss=0.000559, prev_loss=0.000558]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:08, 14.19batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:07, 15.74batch/s]

Training batches on cuda:   6%|▋         | 8/128 [00:00<00:06, 17.26batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:06, 18.70batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:05, 19.78batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:05, 20.62batch/s]

Training batches on cuda:  16%|█▌        | 20/128 [00:00<00:05, 20.89batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:01<00:04, 21.58batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:01<00:04, 21.94batch/s]

Training batches on cuda:  23%|██▎       | 29/128 [00:01<00:04, 22.18batch/s]

Training batches on cuda:  25%|██▌       | 32/128 [00:01<00:04, 21.98batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:04, 21.81batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:04, 21.66batch/s]

Training batches on cuda:  32%|███▏      | 41/128 [00:01<00:03, 22.05batch/s]

Training batches on cuda:  34%|███▍      | 44/128 [00:02<00:03, 22.15batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:02<00:03, 22.26batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:02<00:03, 22.31batch/s]

Training batches on cuda:  41%|████▏     | 53/128 [00:02<00:03, 22.48batch/s]

Training batches on cuda:  44%|████▍     | 56/128 [00:02<00:03, 22.28batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:02<00:03, 21.50batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:02<00:03, 21.54batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:02<00:02, 21.58batch/s]

Training batches on cuda:  53%|█████▎    | 68/128 [00:03<00:02, 21.64batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:03<00:02, 21.65batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:03<00:02, 21.67batch/s]

Training batches on cuda:  60%|██████    | 77/128 [00:03<00:02, 21.62batch/s]

Training batches on cuda:  62%|██████▎   | 80/128 [00:03<00:02, 21.58batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:03<00:02, 21.60batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:03<00:01, 21.52batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:04<00:01, 21.63batch/s]

Training batches on cuda:  72%|███████▏  | 92/128 [00:04<00:01, 21.62batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:04<00:01, 21.65batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:04<00:01, 21.65batch/s]

Training batches on cuda:  79%|███████▉  | 101/128 [00:04<00:01, 21.64batch/s]

Training batches on cuda:  81%|████████▏ | 104/128 [00:04<00:01, 21.67batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:04<00:00, 21.39batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:05<00:00, 21.40batch/s]

Training batches on cuda:  88%|████████▊ | 113/128 [00:05<00:00, 21.33batch/s]

Training batches on cuda:  91%|█████████ | 116/128 [00:05<00:00, 21.44batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:05<00:00, 21.24batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:05<00:00, 21.36batch/s]

Training batches on cuda:  98%|█████████▊| 125/128 [00:05<00:00, 21.41batch/s]

Training batches on cuda: 100%|██████████| 128/128 [00:05<00:00, 22.05batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:48<00:17,  5.96s/epoch, loss=0.00056, prev_loss=0.000559] 
Training epochs on cuda:  80%|████████  | 8/10 [00:48<00:12,  6.01s/epoch, loss=0.00056, prev_loss=0.000559]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:09, 13.97batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:07, 15.46batch/s]

Training batches on cuda:   6%|▋         | 8/128 [00:00<00:07, 16.87batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:06, 18.13batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:05, 19.30batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:05, 20.12batch/s]

Training batches on cuda:  16%|█▌        | 20/128 [00:00<00:05, 20.62batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:01<00:05, 20.79batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:01<00:04, 21.10batch/s]

Training batches on cuda:  23%|██▎       | 29/128 [00:01<00:04, 21.19batch/s]

Training batches on cuda:  25%|██▌       | 32/128 [00:01<00:04, 21.58batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:04, 21.91batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:04, 21.93batch/s]

Training batches on cuda:  32%|███▏      | 41/128 [00:01<00:03, 22.14batch/s]

Training batches on cuda:  34%|███▍      | 44/128 [00:02<00:03, 22.32batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:02<00:03, 22.34batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:02<00:03, 22.42batch/s]

Training batches on cuda:  41%|████▏     | 53/128 [00:02<00:03, 22.35batch/s]

Training batches on cuda:  44%|████▍     | 56/128 [00:02<00:03, 22.49batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:02<00:03, 22.47batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:02<00:02, 22.49batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:02<00:02, 22.29batch/s]

Training batches on cuda:  53%|█████▎    | 68/128 [00:03<00:02, 21.67batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:03<00:02, 21.85batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:03<00:02, 22.09batch/s]

Training batches on cuda:  60%|██████    | 77/128 [00:03<00:02, 22.08batch/s]

Training batches on cuda:  62%|██████▎   | 80/128 [00:03<00:02, 22.13batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:03<00:02, 22.14batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:03<00:01, 21.96batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:04<00:01, 22.19batch/s]

Training batches on cuda:  72%|███████▏  | 92/128 [00:04<00:01, 21.97batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:04<00:01, 21.84batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:04<00:01, 21.80batch/s]

Training batches on cuda:  79%|███████▉  | 101/128 [00:04<00:01, 21.94batch/s]

Training batches on cuda:  81%|████████▏ | 104/128 [00:04<00:01, 22.10batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:04<00:00, 21.59batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:05<00:00, 21.48batch/s]

Training batches on cuda:  88%|████████▊ | 113/128 [00:05<00:00, 21.43batch/s]

Training batches on cuda:  91%|█████████ | 116/128 [00:05<00:00, 21.37batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:05<00:00, 21.31batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:05<00:00, 21.01batch/s]

Training batches on cuda:  98%|█████████▊| 125/128 [00:05<00:00, 21.06batch/s]

Training batches on cuda: 100%|██████████| 128/128 [00:05<00:00, 21.70batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:54<00:12,  6.01s/epoch, loss=0.00056, prev_loss=0.00056] 
Training epochs on cuda:  90%|█████████ | 9/10 [00:54<00:06,  6.04s/epoch, loss=0.00056, prev_loss=0.00056]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:09, 13.93batch/s]

Training batches on cuda:   3%|▎         | 4/128 [00:00<00:08, 15.17batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:07, 16.12batch/s]

Training batches on cuda:   7%|▋         | 9/128 [00:00<00:07, 16.99batch/s]

Training batches on cuda:   9%|▉         | 12/128 [00:00<00:06, 17.73batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:06, 18.63batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:05, 18.61batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:01<00:05, 18.33batch/s]

Training batches on cuda:  16%|█▋        | 21/128 [00:01<00:05, 18.79batch/s]

Training batches on cuda:  19%|█▉        | 24/128 [00:01<00:05, 19.48batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:01<00:05, 19.28batch/s]

Training batches on cuda:  22%|██▏       | 28/128 [00:01<00:05, 19.21batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:01<00:04, 19.78batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:04, 20.17batch/s]

Training batches on cuda:  29%|██▉       | 37/128 [00:01<00:04, 19.87batch/s]

Training batches on cuda:  31%|███▏      | 40/128 [00:02<00:04, 20.58batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:02<00:04, 21.16batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:02<00:03, 21.52batch/s]

Training batches on cuda:  38%|███▊      | 49/128 [00:02<00:03, 21.55batch/s]

Training batches on cuda:  41%|████      | 52/128 [00:02<00:03, 21.51batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:02<00:03, 21.98batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:02<00:03, 21.56batch/s]

Training batches on cuda:  48%|████▊     | 61/128 [00:03<00:03, 21.87batch/s]

Training batches on cuda:  50%|█████     | 64/128 [00:03<00:02, 21.77batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:03<00:02, 21.46batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:03<00:02, 21.49batch/s]

Training batches on cuda:  57%|█████▋    | 73/128 [00:03<00:02, 21.83batch/s]

Training batches on cuda:  59%|█████▉    | 76/128 [00:03<00:02, 21.42batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:03<00:02, 21.75batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:03<00:02, 21.89batch/s]

Training batches on cuda:  66%|██████▋   | 85/128 [00:04<00:01, 21.91batch/s]

Training batches on cuda:  69%|██████▉   | 88/128 [00:04<00:01, 21.91batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:04<00:01, 22.08batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:04<00:01, 22.07batch/s]

Training batches on cuda:  76%|███████▌  | 97/128 [00:04<00:01, 22.09batch/s]

Training batches on cuda:  78%|███████▊  | 100/128 [00:04<00:01, 22.17batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:04<00:01, 22.01batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:05<00:00, 22.03batch/s]

Training batches on cuda:  85%|████████▌ | 109/128 [00:05<00:00, 22.15batch/s]

Training batches on cuda:  88%|████████▊ | 112/128 [00:05<00:00, 22.31batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:05<00:00, 22.44batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:05<00:00, 22.53batch/s]

Training batches on cuda:  95%|█████████▍| 121/128 [00:05<00:00, 22.58batch/s]

Training batches on cuda:  97%|█████████▋| 124/128 [00:05<00:00, 22.20batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:05<00:00, 22.30batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:00<00:06,  6.04s/epoch, loss=0.00056, prev_loss=0.00056]2021-09-16:18:25:28,822 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:25:29,836 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:25:44,809 INFO     [evaluator.py:657] Evaluation took 14.97s seconds
2021-09-16:18:25:44,923 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 10.92s/epoch, loss=0.00056, prev_loss=0.00056]
Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00,  7.68s/epoch, loss=0.00056, prev_loss=0.00056]
2021-09-16:18:25:44,924 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpctdnlnfl'
2021-09-16:18:25:44,986 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpctdnlnfl' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   1%|▏         | 512/38.8k [00:00<00:17, 2.13ktriple/s]
Evaluating on cuda:   3%|▎         | 1.02k/38.8k [00:00<00:16, 2.27ktriple/s]
Evaluating on cuda:   4%|▍         | 1.54k/38.8k [00:00<00:15, 2.37ktriple/s]
Evaluating on cuda:   5%|▌         | 2.05k/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   7%|▋         | 2.56k/38.8k [00:01<00:14, 2.50ktriple/s]
Evaluating on cuda:   8%|▊         | 3.07k/38.8k [00:01<00:14, 2.55ktriple/s]
Evaluating on cuda:   9%|▉         | 3.58k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:25:59,873 INFO     [evaluator.py:657] Evaluation took 14.87s seconds
[I 2021-09-16 18:25:59,878] Trial 45 finished with value: -0.0031223377927835916 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.8954069165635095, 'optimizer.lr': 0.048920723512002044, 'negative_sampler.num_negs_per_pos': 39, 'training.batch_size': 2432}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:25:59,901 WARNING  [api.py:823] No random seed is specified. Setting to 3882893036.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/234 [00:00<00:08, 26.67batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:07, 29.20batch/s]

Training batches on cuda:   5%|▍         | 11/234 [00:00<00:07, 31.33batch/s]

Training batches on cuda:   6%|▋         | 15/234 [00:00<00:06, 32.61batch/s]

Training batches on cuda:   8%|▊         | 19/234 [00:00<00:06, 33.69batch/s]

Training batches on cuda:  10%|▉         | 23/234 [00:00<00:06, 34.44batch/s]

Training batches on cuda:  12%|█▏        | 27/234 [00:00<00:05, 34.97batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:05, 35.82batch/s]

Training batches on cuda:  15%|█▍        | 35/234 [00:00<00:05, 36.15batch/s]

Training batches on cuda:  17%|█▋        | 39/234 [00:01<00:05, 36.70batch/s]

Training batches on cuda:  18%|█▊        | 43/234 [00:01<00:05, 36.77batch/s]

Training batches on cuda:  20%|██        | 47/234 [00:01<00:05, 37.01batch/s]

Training batches on cuda:  22%|██▏       | 51/234 [00:01<00:04, 37.38batch/s]

Training batches on cuda:  24%|██▎       | 55/234 [00:01<00:04, 37.27batch/s]

Training batches on cuda:  25%|██▌       | 59/234 [00:01<00:04, 36.74batch/s]

Training batches on cuda:  27%|██▋       | 63/234 [00:01<00:04, 36.44batch/s]

Training batches on cuda:  29%|██▊       | 67/234 [00:01<00:04, 35.94batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:01<00:04, 35.66batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:02<00:04, 35.35batch/s]

Training batches on cuda:  34%|███▍      | 79/234 [00:02<00:04, 35.26batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:02<00:04, 35.01batch/s]

Training batches on cuda:  37%|███▋      | 87/234 [00:02<00:04, 35.09batch/s]

Training batches on cuda:  39%|███▉      | 91/234 [00:02<00:04, 35.18batch/s]

Training batches on cuda:  41%|████      | 95/234 [00:02<00:03, 35.18batch/s]

Training batches on cuda:  42%|████▏     | 99/234 [00:02<00:03, 35.12batch/s]

Training batches on cuda:  44%|████▍     | 103/234 [00:02<00:03, 35.03batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:02<00:03, 35.14batch/s]

Training batches on cuda:  47%|████▋     | 111/234 [00:03<00:03, 35.01batch/s]

Training batches on cuda:  49%|████▉     | 115/234 [00:03<00:03, 35.65batch/s]

Training batches on cuda:  51%|█████     | 119/234 [00:03<00:03, 36.02batch/s]

Training batches on cuda:  53%|█████▎    | 123/234 [00:03<00:03, 36.23batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:03<00:02, 35.83batch/s]

Training batches on cuda:  56%|█████▌    | 131/234 [00:03<00:02, 36.27batch/s]

Training batches on cuda:  58%|█████▊    | 135/234 [00:03<00:02, 36.79batch/s]

Training batches on cuda:  59%|█████▉    | 139/234 [00:03<00:02, 36.54batch/s]

Training batches on cuda:  61%|██████    | 143/234 [00:03<00:02, 36.80batch/s]

Training batches on cuda:  63%|██████▎   | 147/234 [00:04<00:02, 36.66batch/s]

Training batches on cuda:  65%|██████▍   | 151/234 [00:04<00:02, 36.45batch/s]

Training batches on cuda:  66%|██████▌   | 155/234 [00:04<00:02, 35.72batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:04<00:02, 35.65batch/s]

Training batches on cuda:  70%|██████▉   | 163/234 [00:04<00:01, 35.73batch/s]

Training batches on cuda:  71%|███████▏  | 167/234 [00:04<00:01, 35.74batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:04<00:01, 36.08batch/s]

Training batches on cuda:  75%|███████▍  | 175/234 [00:04<00:01, 35.92batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:04<00:01, 35.61batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:05<00:01, 35.59batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:05<00:01, 36.29batch/s]

Training batches on cuda:  82%|████████▏ | 191/234 [00:05<00:01, 36.09batch/s]

Training batches on cuda:  83%|████████▎ | 195/234 [00:05<00:01, 35.76batch/s]

Training batches on cuda:  85%|████████▌ | 199/234 [00:05<00:00, 35.68batch/s]

Training batches on cuda:  87%|████████▋ | 203/234 [00:05<00:00, 35.88batch/s]

Training batches on cuda:  88%|████████▊ | 207/234 [00:05<00:00, 35.48batch/s]

Training batches on cuda:  90%|█████████ | 211/234 [00:05<00:00, 34.12batch/s]

Training batches on cuda:  92%|█████████▏| 215/234 [00:06<00:00, 34.96batch/s]

Training batches on cuda:  94%|█████████▎| 219/234 [00:06<00:00, 35.79batch/s]

Training batches on cuda:  95%|█████████▌| 223/234 [00:06<00:00, 36.36batch/s]

Training batches on cuda:  97%|█████████▋| 227/234 [00:06<00:00, 36.42batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:06<00:00, 36.56batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:06<?, ?epoch/s, loss=0.000579, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:06<01:00,  6.71s/epoch, loss=0.000579, prev_loss=nan]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/234 [00:00<00:08, 27.37batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:07, 29.74batch/s]

Training batches on cuda:   5%|▍         | 11/234 [00:00<00:07, 31.71batch/s]

Training batches on cuda:   6%|▋         | 15/234 [00:00<00:06, 32.95batch/s]

Training batches on cuda:   8%|▊         | 19/234 [00:00<00:06, 34.17batch/s]

Training batches on cuda:  10%|▉         | 23/234 [00:00<00:06, 35.15batch/s]

Training batches on cuda:  12%|█▏        | 27/234 [00:00<00:05, 35.85batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:05, 36.36batch/s]

Training batches on cuda:  15%|█▍        | 35/234 [00:00<00:05, 36.39batch/s]

Training batches on cuda:  17%|█▋        | 39/234 [00:01<00:05, 34.90batch/s]

Training batches on cuda:  18%|█▊        | 43/234 [00:01<00:05, 34.94batch/s]

Training batches on cuda:  20%|██        | 47/234 [00:01<00:05, 36.09batch/s]

Training batches on cuda:  22%|██▏       | 51/234 [00:01<00:05, 36.07batch/s]

Training batches on cuda:  24%|██▎       | 55/234 [00:01<00:04, 36.69batch/s]

Training batches on cuda:  25%|██▌       | 59/234 [00:01<00:04, 36.79batch/s]

Training batches on cuda:  27%|██▋       | 63/234 [00:01<00:04, 37.00batch/s]

Training batches on cuda:  29%|██▊       | 67/234 [00:01<00:04, 35.91batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:01<00:04, 36.16batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:02<00:04, 35.64batch/s]

Training batches on cuda:  34%|███▍      | 79/234 [00:02<00:04, 36.25batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:02<00:04, 36.66batch/s]

Training batches on cuda:  37%|███▋      | 87/234 [00:02<00:04, 36.26batch/s]

Training batches on cuda:  39%|███▉      | 91/234 [00:02<00:04, 35.72batch/s]

Training batches on cuda:  41%|████      | 95/234 [00:02<00:03, 35.84batch/s]

Training batches on cuda:  42%|████▏     | 99/234 [00:02<00:03, 35.65batch/s]

Training batches on cuda:  44%|████▍     | 103/234 [00:02<00:03, 35.97batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:02<00:03, 35.11batch/s]

Training batches on cuda:  47%|████▋     | 111/234 [00:03<00:03, 35.54batch/s]

Training batches on cuda:  49%|████▉     | 115/234 [00:03<00:03, 35.75batch/s]

Training batches on cuda:  51%|█████     | 119/234 [00:03<00:03, 35.77batch/s]

Training batches on cuda:  53%|█████▎    | 123/234 [00:03<00:03, 35.74batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:03<00:03, 35.43batch/s]

Training batches on cuda:  56%|█████▌    | 131/234 [00:03<00:02, 35.37batch/s]

Training batches on cuda:  58%|█████▊    | 135/234 [00:03<00:02, 35.18batch/s]

Training batches on cuda:  59%|█████▉    | 139/234 [00:03<00:02, 34.89batch/s]

Training batches on cuda:  61%|██████    | 143/234 [00:04<00:02, 34.36batch/s]

Training batches on cuda:  63%|██████▎   | 147/234 [00:04<00:02, 34.73batch/s]

Training batches on cuda:  65%|██████▍   | 151/234 [00:04<00:02, 35.38batch/s]

Training batches on cuda:  66%|██████▌   | 155/234 [00:04<00:02, 35.43batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:04<00:02, 36.04batch/s]

Training batches on cuda:  70%|██████▉   | 163/234 [00:04<00:01, 36.60batch/s]

Training batches on cuda:  71%|███████▏  | 167/234 [00:04<00:01, 36.52batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:04<00:01, 36.67batch/s]

Training batches on cuda:  75%|███████▍  | 175/234 [00:04<00:01, 36.09batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:04<00:01, 36.59batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:05<00:01, 37.31batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:05<00:01, 31.58batch/s]

Training batches on cuda:  82%|████████▏ | 191/234 [00:05<00:01, 33.30batch/s]

Training batches on cuda:  83%|████████▎ | 195/234 [00:05<00:01, 34.03batch/s]

Training batches on cuda:  85%|████████▌ | 199/234 [00:05<00:01, 34.93batch/s]

Training batches on cuda:  87%|████████▋ | 203/234 [00:05<00:00, 35.13batch/s]

Training batches on cuda:  88%|████████▊ | 207/234 [00:05<00:00, 35.17batch/s]

Training batches on cuda:  90%|█████████ | 211/234 [00:05<00:00, 34.94batch/s]

Training batches on cuda:  92%|█████████▏| 215/234 [00:06<00:00, 35.63batch/s]

Training batches on cuda:  94%|█████████▎| 219/234 [00:06<00:00, 35.57batch/s]

Training batches on cuda:  95%|█████████▌| 223/234 [00:06<00:00, 35.43batch/s]

Training batches on cuda:  97%|█████████▋| 227/234 [00:06<00:00, 35.17batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:06<00:00, 35.34batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:13<01:00,  6.71s/epoch, loss=0.000555, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:53,  6.73s/epoch, loss=0.000555, prev_loss=nan]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/234 [00:00<00:12, 18.29batch/s]

Training batches on cuda:   3%|▎         | 6/234 [00:00<00:10, 21.53batch/s]

Training batches on cuda:   4%|▍         | 10/234 [00:00<00:09, 24.30batch/s]

Training batches on cuda:   6%|▌         | 14/234 [00:00<00:08, 26.96batch/s]

Training batches on cuda:   8%|▊         | 18/234 [00:00<00:07, 29.22batch/s]

Training batches on cuda:   9%|▉         | 22/234 [00:00<00:06, 31.07batch/s]

Training batches on cuda:  11%|█         | 26/234 [00:00<00:06, 32.02batch/s]

Training batches on cuda:  13%|█▎        | 30/234 [00:00<00:06, 33.34batch/s]

Training batches on cuda:  15%|█▍        | 34/234 [00:00<00:05, 34.38batch/s]

Training batches on cuda:  16%|█▌        | 38/234 [00:01<00:05, 34.84batch/s]

Training batches on cuda:  18%|█▊        | 42/234 [00:01<00:05, 35.55batch/s]

Training batches on cuda:  20%|█▉        | 46/234 [00:01<00:05, 35.26batch/s]

Training batches on cuda:  21%|██▏       | 50/234 [00:01<00:05, 35.77batch/s]

Training batches on cuda:  23%|██▎       | 54/234 [00:01<00:05, 35.45batch/s]

Training batches on cuda:  25%|██▍       | 58/234 [00:01<00:04, 35.23batch/s]

Training batches on cuda:  26%|██▋       | 62/234 [00:01<00:04, 35.52batch/s]

Training batches on cuda:  28%|██▊       | 66/234 [00:01<00:04, 36.04batch/s]

Training batches on cuda:  30%|██▉       | 70/234 [00:01<00:04, 36.27batch/s]

Training batches on cuda:  32%|███▏      | 74/234 [00:02<00:04, 36.61batch/s]

Training batches on cuda:  33%|███▎      | 78/234 [00:02<00:04, 36.83batch/s]

Training batches on cuda:  35%|███▌      | 82/234 [00:02<00:04, 37.15batch/s]

Training batches on cuda:  37%|███▋      | 86/234 [00:02<00:04, 36.80batch/s]

Training batches on cuda:  38%|███▊      | 90/234 [00:02<00:03, 36.38batch/s]

Training batches on cuda:  40%|████      | 94/234 [00:02<00:03, 36.68batch/s]

Training batches on cuda:  42%|████▏     | 98/234 [00:02<00:03, 36.15batch/s]

Training batches on cuda:  44%|████▎     | 102/234 [00:02<00:03, 36.22batch/s]

Training batches on cuda:  45%|████▌     | 106/234 [00:02<00:03, 36.38batch/s]

Training batches on cuda:  47%|████▋     | 110/234 [00:03<00:03, 36.64batch/s]

Training batches on cuda:  49%|████▊     | 114/234 [00:03<00:03, 36.89batch/s]

Training batches on cuda:  50%|█████     | 118/234 [00:03<00:03, 37.10batch/s]

Training batches on cuda:  52%|█████▏    | 122/234 [00:03<00:03, 36.84batch/s]

Training batches on cuda:  54%|█████▍    | 126/234 [00:03<00:03, 31.33batch/s]

Training batches on cuda:  56%|█████▌    | 130/234 [00:03<00:03, 32.52batch/s]

Training batches on cuda:  57%|█████▋    | 134/234 [00:03<00:02, 33.80batch/s]

Training batches on cuda:  59%|█████▉    | 138/234 [00:03<00:02, 34.48batch/s]

Training batches on cuda:  61%|██████    | 142/234 [00:04<00:02, 35.57batch/s]

Training batches on cuda:  62%|██████▏   | 146/234 [00:04<00:02, 36.31batch/s]

Training batches on cuda:  64%|██████▍   | 150/234 [00:04<00:02, 36.88batch/s]

Training batches on cuda:  66%|██████▌   | 154/234 [00:04<00:02, 37.00batch/s]

Training batches on cuda:  68%|██████▊   | 158/234 [00:04<00:02, 36.04batch/s]

Training batches on cuda:  69%|██████▉   | 162/234 [00:04<00:02, 35.16batch/s]

Training batches on cuda:  71%|███████   | 166/234 [00:04<00:01, 36.05batch/s]

Training batches on cuda:  73%|███████▎  | 170/234 [00:04<00:01, 36.13batch/s]

Training batches on cuda:  74%|███████▍  | 174/234 [00:04<00:01, 36.16batch/s]

Training batches on cuda:  76%|███████▌  | 178/234 [00:04<00:01, 36.79batch/s]

Training batches on cuda:  78%|███████▊  | 182/234 [00:05<00:01, 37.23batch/s]

Training batches on cuda:  79%|███████▉  | 186/234 [00:05<00:01, 37.27batch/s]

Training batches on cuda:  81%|████████  | 190/234 [00:05<00:01, 36.89batch/s]

Training batches on cuda:  83%|████████▎ | 194/234 [00:05<00:01, 36.36batch/s]

Training batches on cuda:  85%|████████▍ | 198/234 [00:05<00:01, 35.67batch/s]

Training batches on cuda:  86%|████████▋ | 202/234 [00:05<00:00, 36.06batch/s]

Training batches on cuda:  88%|████████▊ | 206/234 [00:05<00:00, 35.60batch/s]

Training batches on cuda:  90%|████████▉ | 210/234 [00:05<00:00, 35.50batch/s]

Training batches on cuda:  91%|█████████▏| 214/234 [00:06<00:00, 35.58batch/s]

Training batches on cuda:  93%|█████████▎| 218/234 [00:06<00:00, 36.19batch/s]

Training batches on cuda:  95%|█████████▍| 222/234 [00:06<00:00, 36.37batch/s]

Training batches on cuda:  97%|█████████▋| 226/234 [00:06<00:00, 37.24batch/s]

Training batches on cuda:  98%|█████████▊| 230/234 [00:06<00:00, 37.61batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:20<00:53,  6.73s/epoch, loss=0.000557, prev_loss=0.000555]
Training epochs on cuda:  30%|███       | 3/10 [00:20<00:47,  6.76s/epoch, loss=0.000557, prev_loss=0.000555]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/234 [00:00<00:09, 23.75batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:08, 26.64batch/s]

Training batches on cuda:   5%|▍         | 11/234 [00:00<00:07, 29.10batch/s]

Training batches on cuda:   6%|▋         | 15/234 [00:00<00:07, 31.02batch/s]

Training batches on cuda:   8%|▊         | 19/234 [00:00<00:06, 32.69batch/s]

Training batches on cuda:  10%|▉         | 23/234 [00:00<00:06, 33.98batch/s]

Training batches on cuda:  12%|█▏        | 27/234 [00:00<00:05, 34.97batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:05, 35.07batch/s]

Training batches on cuda:  15%|█▍        | 35/234 [00:00<00:05, 35.59batch/s]

Training batches on cuda:  17%|█▋        | 39/234 [00:01<00:05, 36.04batch/s]

Training batches on cuda:  18%|█▊        | 43/234 [00:01<00:05, 36.44batch/s]

Training batches on cuda:  20%|██        | 47/234 [00:01<00:05, 36.40batch/s]

Training batches on cuda:  22%|██▏       | 51/234 [00:01<00:05, 36.46batch/s]

Training batches on cuda:  24%|██▎       | 55/234 [00:01<00:04, 36.46batch/s]

Training batches on cuda:  25%|██▌       | 59/234 [00:01<00:04, 36.33batch/s]

Training batches on cuda:  27%|██▋       | 63/234 [00:01<00:05, 29.58batch/s]

Training batches on cuda:  29%|██▊       | 67/234 [00:01<00:05, 31.22batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:02<00:05, 32.29batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:02<00:04, 33.46batch/s]

Training batches on cuda:  34%|███▍      | 79/234 [00:02<00:04, 34.61batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:02<00:04, 35.34batch/s]

Training batches on cuda:  37%|███▋      | 87/234 [00:02<00:04, 35.85batch/s]

Training batches on cuda:  39%|███▉      | 91/234 [00:02<00:03, 36.16batch/s]

Training batches on cuda:  41%|████      | 95/234 [00:02<00:03, 36.56batch/s]

Training batches on cuda:  42%|████▏     | 99/234 [00:02<00:03, 36.66batch/s]

Training batches on cuda:  44%|████▍     | 103/234 [00:02<00:03, 36.54batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:03<00:03, 36.47batch/s]

Training batches on cuda:  47%|████▋     | 111/234 [00:03<00:03, 36.62batch/s]

Training batches on cuda:  49%|████▉     | 115/234 [00:03<00:03, 36.89batch/s]

Training batches on cuda:  51%|█████     | 119/234 [00:03<00:03, 37.01batch/s]

Training batches on cuda:  53%|█████▎    | 123/234 [00:03<00:02, 37.17batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:03<00:02, 37.10batch/s]

Training batches on cuda:  56%|█████▌    | 131/234 [00:03<00:02, 37.37batch/s]

Training batches on cuda:  58%|█████▊    | 135/234 [00:03<00:02, 36.97batch/s]

Training batches on cuda:  59%|█████▉    | 139/234 [00:03<00:02, 37.23batch/s]

Training batches on cuda:  61%|██████    | 143/234 [00:04<00:02, 37.24batch/s]

Training batches on cuda:  63%|██████▎   | 147/234 [00:04<00:02, 36.91batch/s]

Training batches on cuda:  65%|██████▍   | 151/234 [00:04<00:02, 37.33batch/s]

Training batches on cuda:  66%|██████▌   | 155/234 [00:04<00:02, 37.88batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:04<00:02, 37.39batch/s]

Training batches on cuda:  70%|██████▉   | 163/234 [00:04<00:01, 37.72batch/s]

Training batches on cuda:  71%|███████▏  | 167/234 [00:04<00:01, 38.00batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:04<00:01, 38.04batch/s]

Training batches on cuda:  75%|███████▍  | 175/234 [00:04<00:01, 38.17batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:04<00:01, 36.88batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:05<00:01, 35.80batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:05<00:01, 35.93batch/s]

Training batches on cuda:  82%|████████▏ | 191/234 [00:05<00:01, 36.44batch/s]

Training batches on cuda:  83%|████████▎ | 195/234 [00:05<00:01, 36.74batch/s]

Training batches on cuda:  85%|████████▌ | 199/234 [00:05<00:00, 35.94batch/s]

Training batches on cuda:  87%|████████▋ | 203/234 [00:05<00:00, 35.96batch/s]

Training batches on cuda:  88%|████████▊ | 207/234 [00:05<00:00, 36.37batch/s]

Training batches on cuda:  90%|█████████ | 211/234 [00:05<00:00, 36.85batch/s]

Training batches on cuda:  92%|█████████▏| 215/234 [00:05<00:00, 35.92batch/s]

Training batches on cuda:  94%|█████████▎| 219/234 [00:06<00:00, 35.79batch/s]

Training batches on cuda:  95%|█████████▌| 223/234 [00:06<00:00, 35.92batch/s]

Training batches on cuda:  97%|█████████▋| 227/234 [00:06<00:00, 35.50batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:06<00:00, 35.74batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:27<00:47,  6.76s/epoch, loss=0.000557, prev_loss=0.000557]
Training epochs on cuda:  40%|████      | 4/10 [00:27<00:40,  6.77s/epoch, loss=0.000557, prev_loss=0.000557]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/234 [00:00<00:09, 25.46batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:08, 27.93batch/s]

Training batches on cuda:   5%|▍         | 11/234 [00:00<00:07, 30.23batch/s]

Training batches on cuda:   6%|▋         | 15/234 [00:00<00:07, 30.56batch/s]

Training batches on cuda:   8%|▊         | 19/234 [00:00<00:06, 31.83batch/s]

Training batches on cuda:  10%|▉         | 23/234 [00:00<00:06, 33.38batch/s]

Training batches on cuda:  12%|█▏        | 27/234 [00:00<00:06, 34.40batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:05, 35.13batch/s]

Training batches on cuda:  15%|█▍        | 35/234 [00:01<00:05, 35.97batch/s]

Training batches on cuda:  17%|█▋        | 39/234 [00:01<00:05, 35.43batch/s]

Training batches on cuda:  18%|█▊        | 43/234 [00:01<00:05, 35.88batch/s]

Training batches on cuda:  20%|██        | 47/234 [00:01<00:05, 36.25batch/s]

Training batches on cuda:  22%|██▏       | 51/234 [00:01<00:05, 35.50batch/s]

Training batches on cuda:  24%|██▎       | 55/234 [00:01<00:05, 35.07batch/s]

Training batches on cuda:  25%|██▌       | 59/234 [00:01<00:04, 36.12batch/s]

Training batches on cuda:  27%|██▋       | 63/234 [00:01<00:04, 36.77batch/s]

Training batches on cuda:  29%|██▊       | 67/234 [00:01<00:04, 35.78batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:02<00:04, 35.22batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:02<00:04, 36.27batch/s]

Training batches on cuda:  34%|███▍      | 79/234 [00:02<00:04, 36.72batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:02<00:04, 36.33batch/s]

Training batches on cuda:  37%|███▋      | 87/234 [00:02<00:03, 36.90batch/s]

Training batches on cuda:  39%|███▉      | 91/234 [00:02<00:03, 36.37batch/s]

Training batches on cuda:  41%|████      | 95/234 [00:02<00:03, 36.12batch/s]

Training batches on cuda:  42%|████▏     | 99/234 [00:02<00:03, 36.45batch/s]

Training batches on cuda:  44%|████▍     | 103/234 [00:02<00:03, 36.92batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:02<00:03, 37.27batch/s]

Training batches on cuda:  47%|████▋     | 111/234 [00:03<00:03, 37.34batch/s]

Training batches on cuda:  49%|████▉     | 115/234 [00:03<00:03, 37.42batch/s]

Training batches on cuda:  51%|█████     | 119/234 [00:03<00:03, 37.50batch/s]

Training batches on cuda:  53%|█████▎    | 123/234 [00:03<00:02, 37.53batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:03<00:02, 37.54batch/s]

Training batches on cuda:  56%|█████▌    | 131/234 [00:03<00:02, 37.60batch/s]

Training batches on cuda:  58%|█████▊    | 135/234 [00:03<00:02, 37.61batch/s]

Training batches on cuda:  59%|█████▉    | 139/234 [00:03<00:02, 36.59batch/s]

Training batches on cuda:  61%|██████    | 143/234 [00:03<00:02, 36.78batch/s]

Training batches on cuda:  63%|██████▎   | 147/234 [00:04<00:02, 36.45batch/s]

Training batches on cuda:  65%|██████▍   | 151/234 [00:04<00:02, 36.08batch/s]

Training batches on cuda:  66%|██████▌   | 155/234 [00:04<00:02, 35.74batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:04<00:02, 35.55batch/s]

Training batches on cuda:  70%|██████▉   | 163/234 [00:04<00:01, 35.54batch/s]

Training batches on cuda:  71%|███████▏  | 167/234 [00:04<00:01, 36.25batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:04<00:01, 35.89batch/s]

Training batches on cuda:  75%|███████▍  | 175/234 [00:04<00:01, 35.35batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:04<00:01, 35.85batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:05<00:01, 32.25batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:05<00:01, 33.65batch/s]

Training batches on cuda:  82%|████████▏ | 191/234 [00:05<00:01, 34.38batch/s]

Training batches on cuda:  83%|████████▎ | 195/234 [00:05<00:01, 34.17batch/s]

Training batches on cuda:  85%|████████▌ | 199/234 [00:05<00:01, 34.79batch/s]

Training batches on cuda:  87%|████████▋ | 203/234 [00:05<00:00, 35.62batch/s]

Training batches on cuda:  88%|████████▊ | 207/234 [00:05<00:00, 36.16batch/s]

Training batches on cuda:  90%|█████████ | 211/234 [00:05<00:00, 36.45batch/s]

Training batches on cuda:  92%|█████████▏| 215/234 [00:05<00:00, 36.66batch/s]

Training batches on cuda:  94%|█████████▎| 219/234 [00:06<00:00, 36.85batch/s]

Training batches on cuda:  95%|█████████▌| 223/234 [00:06<00:00, 37.12batch/s]

Training batches on cuda:  97%|█████████▋| 227/234 [00:06<00:00, 37.20batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:06<00:00, 37.02batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:33<00:40,  6.77s/epoch, loss=0.000558, prev_loss=0.000557]
Training epochs on cuda:  50%|█████     | 5/10 [00:33<00:33,  6.75s/epoch, loss=0.000558, prev_loss=0.000557]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/234 [00:00<00:12, 19.01batch/s]

Training batches on cuda:   3%|▎         | 6/234 [00:00<00:10, 22.25batch/s]

Training batches on cuda:   4%|▍         | 10/234 [00:00<00:08, 25.24batch/s]

Training batches on cuda:   6%|▌         | 14/234 [00:00<00:07, 27.51batch/s]

Training batches on cuda:   8%|▊         | 18/234 [00:00<00:07, 29.37batch/s]

Training batches on cuda:   9%|▉         | 22/234 [00:00<00:06, 31.43batch/s]

Training batches on cuda:  11%|█         | 26/234 [00:00<00:06, 32.89batch/s]

Training batches on cuda:  13%|█▎        | 30/234 [00:00<00:06, 33.57batch/s]

Training batches on cuda:  15%|█▍        | 34/234 [00:00<00:05, 34.94batch/s]

Training batches on cuda:  16%|█▌        | 38/234 [00:01<00:05, 35.81batch/s]

Training batches on cuda:  18%|█▊        | 42/234 [00:01<00:05, 35.90batch/s]

Training batches on cuda:  20%|█▉        | 46/234 [00:01<00:05, 36.36batch/s]

Training batches on cuda:  21%|██▏       | 50/234 [00:01<00:05, 35.94batch/s]

Training batches on cuda:  23%|██▎       | 54/234 [00:01<00:05, 35.57batch/s]

Training batches on cuda:  25%|██▍       | 58/234 [00:01<00:04, 35.62batch/s]

Training batches on cuda:  26%|██▋       | 62/234 [00:01<00:04, 35.11batch/s]

Training batches on cuda:  28%|██▊       | 66/234 [00:01<00:04, 34.94batch/s]

Training batches on cuda:  30%|██▉       | 70/234 [00:01<00:04, 35.14batch/s]

Training batches on cuda:  32%|███▏      | 74/234 [00:02<00:04, 34.92batch/s]

Training batches on cuda:  33%|███▎      | 78/234 [00:02<00:04, 34.88batch/s]

Training batches on cuda:  35%|███▌      | 82/234 [00:02<00:04, 35.08batch/s]

Training batches on cuda:  37%|███▋      | 86/234 [00:02<00:04, 35.94batch/s]

Training batches on cuda:  38%|███▊      | 90/234 [00:02<00:03, 36.15batch/s]

Training batches on cuda:  40%|████      | 94/234 [00:02<00:03, 36.52batch/s]

Training batches on cuda:  42%|████▏     | 98/234 [00:02<00:03, 37.41batch/s]

Training batches on cuda:  44%|████▎     | 102/234 [00:02<00:03, 38.10batch/s]

Training batches on cuda:  45%|████▌     | 106/234 [00:02<00:03, 38.33batch/s]

Training batches on cuda:  47%|████▋     | 110/234 [00:03<00:03, 38.50batch/s]

Training batches on cuda:  49%|████▊     | 114/234 [00:03<00:03, 38.39batch/s]

Training batches on cuda:  50%|█████     | 118/234 [00:03<00:03, 38.01batch/s]

Training batches on cuda:  52%|█████▏    | 122/234 [00:03<00:03, 33.37batch/s]

Training batches on cuda:  54%|█████▍    | 126/234 [00:03<00:03, 34.07batch/s]

Training batches on cuda:  56%|█████▌    | 130/234 [00:03<00:03, 33.81batch/s]

Training batches on cuda:  57%|█████▋    | 134/234 [00:03<00:02, 34.15batch/s]

Training batches on cuda:  59%|█████▉    | 138/234 [00:03<00:02, 34.48batch/s]

Training batches on cuda:  61%|██████    | 142/234 [00:03<00:02, 35.46batch/s]

Training batches on cuda:  62%|██████▏   | 146/234 [00:04<00:02, 35.55batch/s]

Training batches on cuda:  64%|██████▍   | 150/234 [00:04<00:02, 35.59batch/s]

Training batches on cuda:  66%|██████▌   | 154/234 [00:04<00:02, 36.07batch/s]

Training batches on cuda:  68%|██████▊   | 158/234 [00:04<00:02, 35.95batch/s]

Training batches on cuda:  69%|██████▉   | 162/234 [00:04<00:02, 34.31batch/s]

Training batches on cuda:  71%|███████   | 166/234 [00:04<00:02, 33.84batch/s]

Training batches on cuda:  73%|███████▎  | 170/234 [00:04<00:01, 33.76batch/s]

Training batches on cuda:  74%|███████▍  | 174/234 [00:04<00:01, 34.70batch/s]

Training batches on cuda:  76%|███████▌  | 178/234 [00:05<00:01, 35.39batch/s]

Training batches on cuda:  78%|███████▊  | 182/234 [00:05<00:01, 35.90batch/s]

Training batches on cuda:  79%|███████▉  | 186/234 [00:05<00:01, 36.08batch/s]

Training batches on cuda:  81%|████████  | 190/234 [00:05<00:01, 36.54batch/s]

Training batches on cuda:  83%|████████▎ | 194/234 [00:05<00:01, 36.21batch/s]

Training batches on cuda:  85%|████████▍ | 198/234 [00:05<00:01, 35.89batch/s]

Training batches on cuda:  86%|████████▋ | 202/234 [00:05<00:00, 35.61batch/s]

Training batches on cuda:  88%|████████▊ | 206/234 [00:05<00:00, 35.51batch/s]

Training batches on cuda:  90%|████████▉ | 210/234 [00:05<00:00, 36.35batch/s]

Training batches on cuda:  91%|█████████▏| 214/234 [00:06<00:00, 36.52batch/s]

Training batches on cuda:  93%|█████████▎| 218/234 [00:06<00:00, 36.24batch/s]

Training batches on cuda:  95%|█████████▍| 222/234 [00:06<00:00, 36.50batch/s]

Training batches on cuda:  97%|█████████▋| 226/234 [00:06<00:00, 36.39batch/s]

Training batches on cuda:  98%|█████████▊| 230/234 [00:06<00:00, 36.29batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:40<00:33,  6.75s/epoch, loss=0.000558, prev_loss=0.000558]
Training epochs on cuda:  60%|██████    | 6/10 [00:40<00:27,  6.78s/epoch, loss=0.000558, prev_loss=0.000558]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/234 [00:00<00:08, 26.51batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:07, 28.95batch/s]

Training batches on cuda:   5%|▍         | 11/234 [00:00<00:07, 30.92batch/s]

Training batches on cuda:   6%|▋         | 15/234 [00:00<00:06, 32.24batch/s]

Training batches on cuda:   8%|▊         | 19/234 [00:00<00:06, 33.33batch/s]

Training batches on cuda:  10%|▉         | 23/234 [00:00<00:06, 34.55batch/s]

Training batches on cuda:  12%|█▏        | 27/234 [00:00<00:06, 34.25batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:05, 34.82batch/s]

Training batches on cuda:  15%|█▍        | 35/234 [00:00<00:05, 35.24batch/s]

Training batches on cuda:  17%|█▋        | 39/234 [00:01<00:05, 35.50batch/s]

Training batches on cuda:  18%|█▊        | 43/234 [00:01<00:05, 36.27batch/s]

Training batches on cuda:  20%|██        | 47/234 [00:01<00:05, 36.36batch/s]

Training batches on cuda:  22%|██▏       | 51/234 [00:01<00:04, 36.84batch/s]

Training batches on cuda:  24%|██▎       | 55/234 [00:01<00:05, 30.92batch/s]

Training batches on cuda:  25%|██▌       | 59/234 [00:01<00:05, 32.62batch/s]

Training batches on cuda:  27%|██▋       | 63/234 [00:01<00:05, 33.63batch/s]

Training batches on cuda:  29%|██▊       | 67/234 [00:01<00:04, 34.65batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:02<00:04, 34.73batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:02<00:04, 35.45batch/s]

Training batches on cuda:  34%|███▍      | 79/234 [00:02<00:04, 35.33batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:02<00:04, 35.41batch/s]

Training batches on cuda:  37%|███▋      | 87/234 [00:02<00:04, 35.66batch/s]

Training batches on cuda:  39%|███▉      | 91/234 [00:02<00:04, 35.75batch/s]

Training batches on cuda:  41%|████      | 95/234 [00:02<00:03, 36.11batch/s]

Training batches on cuda:  42%|████▏     | 99/234 [00:02<00:03, 36.31batch/s]

Training batches on cuda:  44%|████▍     | 103/234 [00:02<00:03, 36.75batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:03<00:03, 37.02batch/s]

Training batches on cuda:  47%|████▋     | 111/234 [00:03<00:03, 36.84batch/s]

Training batches on cuda:  49%|████▉     | 115/234 [00:03<00:03, 36.43batch/s]

Training batches on cuda:  51%|█████     | 119/234 [00:03<00:03, 35.62batch/s]

Training batches on cuda:  53%|█████▎    | 123/234 [00:03<00:03, 35.84batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:03<00:03, 35.43batch/s]

Training batches on cuda:  56%|█████▌    | 131/234 [00:03<00:02, 35.99batch/s]

Training batches on cuda:  58%|█████▊    | 135/234 [00:03<00:02, 36.01batch/s]

Training batches on cuda:  59%|█████▉    | 139/234 [00:03<00:02, 36.55batch/s]

Training batches on cuda:  61%|██████    | 143/234 [00:04<00:02, 36.67batch/s]

Training batches on cuda:  63%|██████▎   | 147/234 [00:04<00:02, 37.11batch/s]

Training batches on cuda:  65%|██████▍   | 151/234 [00:04<00:02, 36.94batch/s]

Training batches on cuda:  66%|██████▌   | 155/234 [00:04<00:02, 36.60batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:04<00:02, 36.12batch/s]

Training batches on cuda:  70%|██████▉   | 163/234 [00:04<00:01, 36.09batch/s]

Training batches on cuda:  71%|███████▏  | 167/234 [00:04<00:01, 36.19batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:04<00:01, 36.03batch/s]

Training batches on cuda:  75%|███████▍  | 175/234 [00:04<00:01, 36.29batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:05<00:01, 36.74batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:05<00:01, 37.05batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:05<00:01, 37.26batch/s]

Training batches on cuda:  82%|████████▏ | 191/234 [00:05<00:01, 37.21batch/s]

Training batches on cuda:  83%|████████▎ | 195/234 [00:05<00:01, 36.77batch/s]

Training batches on cuda:  85%|████████▌ | 199/234 [00:05<00:00, 36.41batch/s]

Training batches on cuda:  87%|████████▋ | 203/234 [00:05<00:00, 35.53batch/s]

Training batches on cuda:  88%|████████▊ | 207/234 [00:05<00:00, 34.39batch/s]

Training batches on cuda:  90%|█████████ | 211/234 [00:05<00:00, 34.05batch/s]

Training batches on cuda:  92%|█████████▏| 215/234 [00:06<00:00, 34.87batch/s]

Training batches on cuda:  94%|█████████▎| 219/234 [00:06<00:00, 34.94batch/s]

Training batches on cuda:  95%|█████████▌| 223/234 [00:06<00:00, 34.27batch/s]

Training batches on cuda:  97%|█████████▋| 227/234 [00:06<00:00, 35.20batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:06<00:00, 35.30batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:47<00:27,  6.78s/epoch, loss=0.000557, prev_loss=0.000558]
Training epochs on cuda:  70%|███████   | 7/10 [00:47<00:20,  6.80s/epoch, loss=0.000557, prev_loss=0.000558]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/234 [00:00<00:12, 17.90batch/s]

Training batches on cuda:   3%|▎         | 6/234 [00:00<00:10, 21.15batch/s]

Training batches on cuda:   4%|▍         | 10/234 [00:00<00:09, 24.29batch/s]

Training batches on cuda:   6%|▌         | 14/234 [00:00<00:08, 26.95batch/s]

Training batches on cuda:   8%|▊         | 18/234 [00:00<00:07, 29.29batch/s]

Training batches on cuda:   9%|▉         | 22/234 [00:00<00:06, 31.42batch/s]

Training batches on cuda:  11%|█         | 26/234 [00:00<00:06, 33.05batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:05, 35.15batch/s]

Training batches on cuda:  15%|█▍        | 35/234 [00:00<00:05, 35.67batch/s]

Training batches on cuda:  17%|█▋        | 39/234 [00:01<00:05, 36.03batch/s]

Training batches on cuda:  18%|█▊        | 43/234 [00:01<00:05, 36.59batch/s]

Training batches on cuda:  20%|██        | 47/234 [00:01<00:05, 37.03batch/s]

Training batches on cuda:  22%|██▏       | 51/234 [00:01<00:05, 36.16batch/s]

Training batches on cuda:  24%|██▎       | 55/234 [00:01<00:04, 36.16batch/s]

Training batches on cuda:  25%|██▌       | 59/234 [00:01<00:04, 36.44batch/s]

Training batches on cuda:  27%|██▋       | 63/234 [00:01<00:04, 36.78batch/s]

Training batches on cuda:  29%|██▊       | 67/234 [00:01<00:04, 36.27batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:01<00:04, 36.18batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:02<00:04, 36.54batch/s]

Training batches on cuda:  34%|███▍      | 79/234 [00:02<00:04, 36.80batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:02<00:04, 36.74batch/s]

Training batches on cuda:  37%|███▋      | 87/234 [00:02<00:04, 36.37batch/s]

Training batches on cuda:  39%|███▉      | 91/234 [00:02<00:03, 36.24batch/s]

Training batches on cuda:  41%|████      | 95/234 [00:02<00:03, 36.72batch/s]

Training batches on cuda:  42%|████▏     | 99/234 [00:02<00:03, 37.47batch/s]

Training batches on cuda:  44%|████▍     | 103/234 [00:02<00:03, 37.47batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:02<00:03, 36.97batch/s]

Training batches on cuda:  47%|████▋     | 111/234 [00:03<00:03, 36.76batch/s]

Training batches on cuda:  49%|████▉     | 115/234 [00:03<00:03, 36.73batch/s]

Training batches on cuda:  51%|█████     | 119/234 [00:03<00:03, 36.69batch/s]

Training batches on cuda:  53%|█████▎    | 123/234 [00:03<00:03, 36.97batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:03<00:02, 36.91batch/s]

Training batches on cuda:  56%|█████▌    | 131/234 [00:03<00:02, 37.05batch/s]

Training batches on cuda:  58%|█████▊    | 135/234 [00:03<00:02, 36.99batch/s]

Training batches on cuda:  59%|█████▉    | 139/234 [00:03<00:02, 36.69batch/s]

Training batches on cuda:  61%|██████    | 143/234 [00:03<00:02, 36.72batch/s]

Training batches on cuda:  63%|██████▎   | 147/234 [00:04<00:02, 36.76batch/s]

Training batches on cuda:  65%|██████▍   | 151/234 [00:04<00:02, 36.19batch/s]

Training batches on cuda:  66%|██████▌   | 155/234 [00:04<00:02, 36.29batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:04<00:02, 36.29batch/s]

Training batches on cuda:  70%|██████▉   | 163/234 [00:04<00:01, 36.32batch/s]

Training batches on cuda:  71%|███████▏  | 167/234 [00:04<00:01, 36.34batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:04<00:01, 36.42batch/s]

Training batches on cuda:  75%|███████▍  | 175/234 [00:04<00:01, 36.52batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:04<00:01, 36.73batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:05<00:01, 35.67batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:05<00:01, 36.05batch/s]

Training batches on cuda:  82%|████████▏ | 191/234 [00:05<00:01, 36.35batch/s]

Training batches on cuda:  83%|████████▎ | 195/234 [00:05<00:01, 36.50batch/s]

Training batches on cuda:  85%|████████▌ | 199/234 [00:05<00:00, 36.23batch/s]

Training batches on cuda:  87%|████████▋ | 203/234 [00:05<00:00, 36.06batch/s]

Training batches on cuda:  88%|████████▊ | 207/234 [00:05<00:00, 36.13batch/s]

Training batches on cuda:  90%|█████████ | 211/234 [00:05<00:00, 36.18batch/s]

Training batches on cuda:  92%|█████████▏| 215/234 [00:05<00:00, 35.99batch/s]

Training batches on cuda:  94%|█████████▎| 219/234 [00:06<00:00, 36.56batch/s]

Training batches on cuda:  95%|█████████▌| 223/234 [00:06<00:00, 36.48batch/s]

Training batches on cuda:  97%|█████████▋| 227/234 [00:06<00:00, 36.65batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:06<00:00, 36.36batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:54<00:20,  6.80s/epoch, loss=0.000558, prev_loss=0.000557]
Training epochs on cuda:  80%|████████  | 8/10 [00:54<00:13,  6.75s/epoch, loss=0.000558, prev_loss=0.000557]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/234 [00:00<00:08, 27.51batch/s]

Training batches on cuda:   3%|▎         | 7/234 [00:00<00:07, 30.07batch/s]

Training batches on cuda:   5%|▍         | 11/234 [00:00<00:07, 31.86batch/s]

Training batches on cuda:   6%|▋         | 15/234 [00:00<00:06, 33.29batch/s]

Training batches on cuda:   8%|▊         | 19/234 [00:00<00:06, 34.40batch/s]

Training batches on cuda:  10%|▉         | 23/234 [00:00<00:05, 35.42batch/s]

Training batches on cuda:  12%|█▏        | 27/234 [00:00<00:05, 35.39batch/s]

Training batches on cuda:  13%|█▎        | 31/234 [00:00<00:05, 36.13batch/s]

Training batches on cuda:  15%|█▍        | 35/234 [00:00<00:05, 36.37batch/s]

Training batches on cuda:  17%|█▋        | 39/234 [00:01<00:05, 36.86batch/s]

Training batches on cuda:  18%|█▊        | 43/234 [00:01<00:05, 37.10batch/s]

Training batches on cuda:  20%|██        | 47/234 [00:01<00:04, 37.50batch/s]

Training batches on cuda:  22%|██▏       | 51/234 [00:01<00:04, 37.33batch/s]

Training batches on cuda:  24%|██▎       | 55/234 [00:01<00:04, 37.30batch/s]

Training batches on cuda:  25%|██▌       | 59/234 [00:01<00:04, 37.68batch/s]

Training batches on cuda:  27%|██▋       | 63/234 [00:01<00:04, 37.91batch/s]

Training batches on cuda:  29%|██▊       | 67/234 [00:01<00:04, 37.56batch/s]

Training batches on cuda:  30%|███       | 71/234 [00:01<00:04, 37.52batch/s]

Training batches on cuda:  32%|███▏      | 75/234 [00:02<00:04, 37.78batch/s]

Training batches on cuda:  34%|███▍      | 79/234 [00:02<00:04, 37.94batch/s]

Training batches on cuda:  35%|███▌      | 83/234 [00:02<00:03, 37.80batch/s]

Training batches on cuda:  37%|███▋      | 87/234 [00:02<00:03, 37.98batch/s]

Training batches on cuda:  39%|███▉      | 91/234 [00:02<00:03, 38.12batch/s]

Training batches on cuda:  41%|████      | 95/234 [00:02<00:03, 38.09batch/s]

Training batches on cuda:  42%|████▏     | 99/234 [00:02<00:03, 38.00batch/s]

Training batches on cuda:  44%|████▍     | 103/234 [00:02<00:03, 37.56batch/s]

Training batches on cuda:  46%|████▌     | 107/234 [00:02<00:03, 37.47batch/s]

Training batches on cuda:  47%|████▋     | 111/234 [00:02<00:03, 37.40batch/s]

Training batches on cuda:  49%|████▉     | 115/234 [00:03<00:03, 37.58batch/s]

Training batches on cuda:  51%|█████     | 119/234 [00:03<00:03, 35.79batch/s]

Training batches on cuda:  53%|█████▎    | 123/234 [00:03<00:03, 36.45batch/s]

Training batches on cuda:  54%|█████▍    | 127/234 [00:03<00:02, 36.99batch/s]

Training batches on cuda:  56%|█████▌    | 131/234 [00:03<00:02, 37.32batch/s]

Training batches on cuda:  58%|█████▊    | 135/234 [00:03<00:02, 37.64batch/s]

Training batches on cuda:  59%|█████▉    | 139/234 [00:03<00:02, 37.36batch/s]

Training batches on cuda:  61%|██████    | 143/234 [00:03<00:02, 37.61batch/s]

Training batches on cuda:  63%|██████▎   | 147/234 [00:03<00:02, 37.52batch/s]

Training batches on cuda:  65%|██████▍   | 151/234 [00:04<00:02, 37.84batch/s]

Training batches on cuda:  66%|██████▌   | 155/234 [00:04<00:02, 38.06batch/s]

Training batches on cuda:  68%|██████▊   | 159/234 [00:04<00:01, 37.91batch/s]

Training batches on cuda:  70%|██████▉   | 163/234 [00:04<00:01, 37.96batch/s]

Training batches on cuda:  71%|███████▏  | 167/234 [00:04<00:01, 37.92batch/s]

Training batches on cuda:  73%|███████▎  | 171/234 [00:04<00:01, 37.67batch/s]

Training batches on cuda:  75%|███████▍  | 175/234 [00:04<00:01, 37.66batch/s]

Training batches on cuda:  76%|███████▋  | 179/234 [00:04<00:01, 37.89batch/s]

Training batches on cuda:  78%|███████▊  | 183/234 [00:04<00:01, 37.99batch/s]

Training batches on cuda:  80%|███████▉  | 187/234 [00:05<00:01, 37.85batch/s]

Training batches on cuda:  82%|████████▏ | 191/234 [00:05<00:01, 37.75batch/s]

Training batches on cuda:  83%|████████▎ | 195/234 [00:05<00:01, 37.64batch/s]

Training batches on cuda:  85%|████████▌ | 199/234 [00:05<00:00, 38.11batch/s]

Training batches on cuda:  87%|████████▋ | 203/234 [00:05<00:00, 38.12batch/s]

Training batches on cuda:  88%|████████▊ | 207/234 [00:05<00:00, 37.94batch/s]

Training batches on cuda:  90%|█████████ | 211/234 [00:05<00:00, 37.95batch/s]

Training batches on cuda:  92%|█████████▏| 215/234 [00:05<00:00, 38.10batch/s]

Training batches on cuda:  94%|█████████▎| 219/234 [00:05<00:00, 37.82batch/s]

Training batches on cuda:  95%|█████████▌| 223/234 [00:05<00:00, 37.87batch/s]

Training batches on cuda:  97%|█████████▋| 227/234 [00:06<00:00, 37.82batch/s]

Training batches on cuda:  99%|█████████▊| 231/234 [00:06<00:00, 37.32batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:00<00:13,  6.75s/epoch, loss=0.000556, prev_loss=0.000558]
Training epochs on cuda:  90%|█████████ | 9/10 [01:00<00:06,  6.66s/epoch, loss=0.000556, prev_loss=0.000558]

Training batches on cuda:   0%|          | 0/234 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/234 [00:00<00:11, 19.91batch/s]

Training batches on cuda:   3%|▎         | 6/234 [00:00<00:09, 23.09batch/s]

Training batches on cuda:   4%|▍         | 10/234 [00:00<00:08, 25.97batch/s]

Training batches on cuda:   6%|▌         | 14/234 [00:00<00:07, 28.02batch/s]

Training batches on cuda:   8%|▊         | 18/234 [00:00<00:07, 30.29batch/s]

Training batches on cuda:   9%|▉         | 22/234 [00:00<00:06, 32.34batch/s]

Training batches on cuda:  11%|█         | 26/234 [00:00<00:06, 33.89batch/s]

Training batches on cuda:  13%|█▎        | 30/234 [00:00<00:05, 35.06batch/s]

Training batches on cuda:  15%|█▍        | 34/234 [00:00<00:05, 35.84batch/s]

Training batches on cuda:  16%|█▌        | 38/234 [00:01<00:05, 36.40batch/s]

Training batches on cuda:  18%|█▊        | 42/234 [00:01<00:05, 36.81batch/s]

Training batches on cuda:  20%|█▉        | 46/234 [00:01<00:05, 37.08batch/s]

Training batches on cuda:  21%|██▏       | 50/234 [00:01<00:04, 36.90batch/s]

Training batches on cuda:  23%|██▎       | 54/234 [00:01<00:04, 36.95batch/s]

Training batches on cuda:  25%|██▍       | 58/234 [00:01<00:04, 36.51batch/s]

Training batches on cuda:  26%|██▋       | 62/234 [00:01<00:05, 33.51batch/s]

Training batches on cuda:  28%|██▊       | 66/234 [00:01<00:04, 34.47batch/s]

Training batches on cuda:  30%|██▉       | 70/234 [00:01<00:04, 35.04batch/s]

Training batches on cuda:  32%|███▏      | 74/234 [00:02<00:04, 34.57batch/s]

Training batches on cuda:  33%|███▎      | 78/234 [00:02<00:04, 35.10batch/s]

Training batches on cuda:  35%|███▌      | 82/234 [00:02<00:04, 35.28batch/s]

Training batches on cuda:  37%|███▋      | 86/234 [00:02<00:04, 35.83batch/s]

Training batches on cuda:  38%|███▊      | 90/234 [00:02<00:04, 35.63batch/s]

Training batches on cuda:  40%|████      | 94/234 [00:02<00:03, 36.21batch/s]

Training batches on cuda:  42%|████▏     | 98/234 [00:02<00:03, 36.48batch/s]

Training batches on cuda:  44%|████▎     | 102/234 [00:02<00:03, 36.63batch/s]

Training batches on cuda:  45%|████▌     | 106/234 [00:02<00:03, 36.51batch/s]

Training batches on cuda:  47%|████▋     | 110/234 [00:03<00:03, 36.37batch/s]

Training batches on cuda:  49%|████▊     | 114/234 [00:03<00:03, 37.15batch/s]

Training batches on cuda:  50%|█████     | 118/234 [00:03<00:03, 37.15batch/s]

Training batches on cuda:  52%|█████▏    | 122/234 [00:03<00:03, 36.64batch/s]

Training batches on cuda:  54%|█████▍    | 126/234 [00:03<00:02, 36.37batch/s]

Training batches on cuda:  56%|█████▌    | 130/234 [00:03<00:02, 36.71batch/s]

Training batches on cuda:  57%|█████▋    | 134/234 [00:03<00:02, 36.89batch/s]

Training batches on cuda:  59%|█████▉    | 138/234 [00:03<00:02, 37.01batch/s]

Training batches on cuda:  61%|██████    | 142/234 [00:03<00:02, 36.82batch/s]

Training batches on cuda:  62%|██████▏   | 146/234 [00:04<00:02, 36.84batch/s]

Training batches on cuda:  64%|██████▍   | 150/234 [00:04<00:02, 36.84batch/s]

Training batches on cuda:  66%|██████▌   | 154/234 [00:04<00:02, 36.24batch/s]

Training batches on cuda:  68%|██████▊   | 158/234 [00:04<00:02, 35.98batch/s]

Training batches on cuda:  69%|██████▉   | 162/234 [00:04<00:02, 35.98batch/s]

Training batches on cuda:  71%|███████   | 166/234 [00:04<00:01, 36.30batch/s]

Training batches on cuda:  73%|███████▎  | 170/234 [00:04<00:01, 36.50batch/s]

Training batches on cuda:  74%|███████▍  | 174/234 [00:04<00:01, 36.79batch/s]

Training batches on cuda:  76%|███████▌  | 178/234 [00:04<00:01, 37.16batch/s]

Training batches on cuda:  78%|███████▊  | 182/234 [00:05<00:01, 37.33batch/s]

Training batches on cuda:  79%|███████▉  | 186/234 [00:05<00:01, 37.34batch/s]

Training batches on cuda:  81%|████████  | 190/234 [00:05<00:01, 37.33batch/s]

Training batches on cuda:  83%|████████▎ | 194/234 [00:05<00:01, 36.85batch/s]

Training batches on cuda:  85%|████████▍ | 198/234 [00:05<00:00, 36.65batch/s]

Training batches on cuda:  86%|████████▋ | 202/234 [00:05<00:00, 36.54batch/s]

Training batches on cuda:  88%|████████▊ | 206/234 [00:05<00:00, 36.65batch/s]

Training batches on cuda:  90%|████████▉ | 210/234 [00:05<00:00, 36.89batch/s]

Training batches on cuda:  91%|█████████▏| 214/234 [00:05<00:00, 36.17batch/s]

Training batches on cuda:  93%|█████████▎| 218/234 [00:06<00:00, 36.32batch/s]

Training batches on cuda:  95%|█████████▍| 222/234 [00:06<00:00, 36.35batch/s]

Training batches on cuda:  97%|█████████▋| 226/234 [00:06<00:00, 36.71batch/s]

Training batches on cuda:  98%|█████████▊| 230/234 [00:06<00:00, 36.15batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:07<00:06,  6.66s/epoch, loss=0.000557, prev_loss=0.000556]2021-09-16:18:27:07,818 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:27:08,814 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:27:23,843 INFO     [evaluator.py:657] Evaluation took 15.03s seconds
2021-09-16:18:27:23,967 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 11.54s/epoch, loss=0.000557, prev_loss=0.000556]
Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00,  8.35s/epoch, loss=0.000557, prev_loss=0.000556]
2021-09-16:18:27:23,968 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpzdeooob6'
2021-09-16:18:27:24,41 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzdeooob6' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.38ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.50ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.60ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.62ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.57ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.59ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.60ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.62ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:27:38,914 INFO     [evaluator.py:657] Evaluation took 14.85s seconds
[I 2021-09-16 18:27:38,919] Trial 46 finished with value: 0.000551155189810526 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.6892056390827564, 'optimizer.lr': 0.06342417985840915, 'negative_sampler.num_negs_per_pos': 67, 'training.batch_size': 1332}. Best is trial 21 with value: 0.9617823966700032.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:27:38,940 WARNING  [api.py:823] No random seed is specified. Setting to 2623136497.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:18,  5.59batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:15,  6.78batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  7.97batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.03batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09,  9.99batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.78batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.35batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.83batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.29batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.54batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.82batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.95batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 13.17batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 13.14batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.24batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.38batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.38batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.55batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.54batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:04, 13.43batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.45batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.43batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:05, 11.69batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 12.22batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 12.55batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 12.85batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 12.99batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.04batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.76batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.85batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.93batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.08batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.13batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.17batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 13.10batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.98batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.88batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 12.62batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 12.71batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 12.68batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.71batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.77batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.49batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.77batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.93batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 12.95batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.18batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.30batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.45batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.61batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.68batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 13.73batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 13.76batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:08<?, ?epoch/s, loss=0.000725, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:08<01:16,  8.46s/epoch, loss=0.000725, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:13,  8.05batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:11,  9.10batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:10,  9.96batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:09, 10.61batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:08, 11.26batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 11.54batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:07, 11.89batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.21batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.60batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.64batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.80batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.92batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 12.84batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 13.00batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.22batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.29batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.33batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.43batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.46batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:04, 13.53batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.38batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.30batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.39batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.45batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.47batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 13.41batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:03, 13.33batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.28batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.31batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.35batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.15batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 12.79batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 12.75batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.74batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 12.76batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.89batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.89batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 12.99batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 13.10batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 13.24batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 13.07batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.96batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.91batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 13.12batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 13.20batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:06<00:01, 13.16batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.20batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.19batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 12.97batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.75batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 12.78batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 12.94batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.89batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:16<01:16,  8.46s/epoch, loss=0.000703, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:16<01:07,  8.43s/epoch, loss=0.000703, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:17,  6.04batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.27batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:11,  8.53batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.70batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.73batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 11.57batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:07, 12.25batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.79batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:06, 13.19batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 13.47batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 13.66batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.72batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 13.11batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:05, 13.43batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.65batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.82batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.94batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 14.02batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:04, 14.08batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:02<00:04, 14.10batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 14.10batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 14.13batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.82batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.94batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 14.04batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:03, 14.07batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:03<00:03, 14.11batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 14.11batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.99batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 14.04batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 14.06batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 14.10batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:04<00:02, 14.14batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:04<00:02, 14.16batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 14.18batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 14.13batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 14.14batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 14.14batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 14.18batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:05<00:01, 14.22batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:05<00:01, 14.21batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 14.22batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 14.20batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 14.19batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 14.18batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:06<00:01, 14.17batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:06<00:00, 13.07batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:06<00:00, 13.38batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.61batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.77batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.88batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 13.97batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:07<00:00, 14.04batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:24<01:07,  8.43s/epoch, loss=0.000587, prev_loss=0.000703]
Training epochs on cuda:  30%|███       | 3/10 [00:24<00:57,  8.28s/epoch, loss=0.000587, prev_loss=0.000703]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:17,  5.97batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.15batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.23batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.30batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.13batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.63batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.11batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.51batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 11.89batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 11.99batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.24batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.30batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.53batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.45batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:06, 12.51batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:06, 12.11batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 12.27batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 12.32batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:03<00:05, 12.19batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 12.35batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:05, 12.34batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:05, 12.45batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 12.49batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:05, 11.44batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:04<00:04, 11.71batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 12.09batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 12.35batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:04, 12.54batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.74batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.85batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.96batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:05<00:03, 12.97batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.10batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.82batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 12.96batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.74batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.81batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:06<00:02, 12.84batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:06<00:02, 12.91batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 12.73batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.71batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.66batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.73batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.75batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:07<00:01, 12.78batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 12.87batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:01, 12.87batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.00batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.02batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.09batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:08<00:00, 13.01batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:08<00:00, 12.90batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.68batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:33<00:57,  8.28s/epoch, loss=0.000463, prev_loss=0.000587]
Training epochs on cuda:  40%|████      | 4/10 [00:33<00:50,  8.45s/epoch, loss=0.000463, prev_loss=0.000587]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:17,  6.17batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.30batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.39batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.39batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.13batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.87batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.47batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.89batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.13batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 12.38batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.42batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.59batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.74batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.83batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 12.91batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 12.99batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.00batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 12.99batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 12.91batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 12.95batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:05, 12.97batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.06batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.07batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.10batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.07batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 13.12batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 13.20batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.18batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.17batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.93batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.01batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.11batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.20batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.13batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 13.11batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 13.18batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 13.22batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 13.16batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 13.19batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 13.23batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 13.28batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 13.26batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 13.28batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 13.07batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.93batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 13.02batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.01batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.15batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.11batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.85batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 12.89batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 12.77batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.76batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:42<00:50,  8.45s/epoch, loss=0.000386, prev_loss=0.000463]
Training epochs on cuda:  50%|█████     | 5/10 [00:42<00:42,  8.47s/epoch, loss=0.000386, prev_loss=0.000463]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:16,  6.32batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:13,  7.47batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:11,  8.59batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.61batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.46batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 11.16batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:07, 11.67batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.09batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.41batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 11.72batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:07, 12.06batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.32batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.68batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.85batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:06, 12.77batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 12.98batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.13batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.08batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.23batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.26batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.29batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.33batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.25batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.31batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.34batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 13.34batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:03, 13.30batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.35batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.39batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.37batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.39batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.31batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.29batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.33batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 13.32batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 13.34batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 13.41batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 13.09batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 13.17batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 13.23batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 13.30batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 13.27batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.48batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.70batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.74batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 12.88batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.05batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.15batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.19batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.25batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.27batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 13.14batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 13.12batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:50<00:42,  8.47s/epoch, loss=0.000308, prev_loss=0.000386]
Training epochs on cuda:  60%|██████    | 6/10 [00:50<00:33,  8.47s/epoch, loss=0.000308, prev_loss=0.000386]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:17,  6.10batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.25batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.39batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.42batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.32batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 11.05batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:07, 11.64batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.14batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.50batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.78batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.93batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.96batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 12.98batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.97batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 12.93batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.03batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 12.97batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.04batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.12batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.15batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.20batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.28batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.27batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.34batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.27batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 13.09batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 13.19batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 12.93batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.95batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.78batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.83batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 12.74batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 12.70batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.58batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 12.82batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 13.01batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 13.21batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 13.14batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 13.29batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 13.39batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 13.36batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 13.20batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 13.14batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.85batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 13.19batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 13.39batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.52batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.40batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.49batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.47batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.52batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 13.45batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 13.37batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:59<00:33,  8.47s/epoch, loss=0.000271, prev_loss=0.000308]
Training epochs on cuda:  70%|███████   | 7/10 [00:59<00:25,  8.45s/epoch, loss=0.000271, prev_loss=0.000308]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:17,  6.16batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.28batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:11,  8.46batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.45batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.36batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 11.05batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:07, 11.66batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.13batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.56batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.73batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.75batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.85batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 13.00batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.99batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.03batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.04batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.09batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 12.95batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 12.84batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 12.96batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.17batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.03batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.05batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.08batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.09batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 12.80batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 12.76batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:04, 12.71batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.66batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.79batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.12batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 12.37batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 12.55batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.69batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 12.88batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.91batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.81batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 12.98batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:06<00:02, 12.98batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 13.09batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.95batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.93batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.92batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.87batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.80batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 13.00batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:01, 12.99batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 12.99batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.05batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.84batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 12.78batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:08<00:00, 12.76batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.83batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [01:07<00:25,  8.45s/epoch, loss=0.000256, prev_loss=0.000271]
Training epochs on cuda:  80%|████████  | 8/10 [01:07<00:16,  8.49s/epoch, loss=0.000256, prev_loss=0.000271]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:16,  6.18batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.24batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.23batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.07batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.04batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.73batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.42batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.87batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.10batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 12.28batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.42batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.61batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.70batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.90batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.02batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.07batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.17batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.20batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.20batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.24batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.34batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.22batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.07batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.21batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.08batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 12.93batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 12.83batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 12.76batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.70batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.68batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.70batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 12.53batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 12.69batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.71batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 12.90batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 13.01batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.91batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 13.15batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:06<00:02, 13.37batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 13.50batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 13.59batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 13.68batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 13.72batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 13.78batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 13.59batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 13.49batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.32batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.07batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 12.92batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.96batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.05batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 13.13batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 13.25batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:16<00:16,  8.49s/epoch, loss=0.000247, prev_loss=0.000256]
Training epochs on cuda:  90%|█████████ | 9/10 [01:16<00:08,  8.49s/epoch, loss=0.000247, prev_loss=0.000256]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:18,  5.56batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:15,  6.74batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  7.94batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.07batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.06batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.94batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.55batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.08batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.38batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.60batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.77batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.98batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 13.09batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.97batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 12.95batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 12.84batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 12.74batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 12.76batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 12.91batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 12.79batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:05, 12.85batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 12.96batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 12.98batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.00batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 12.96batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 13.12batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 13.22batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.23batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.27batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.31batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.23batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 12.97batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 12.97batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.94batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 12.76batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.69batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.77batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 12.58batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 12.79batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 12.94batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.99batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.94batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.57batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.50batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.72batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 12.78batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:01, 12.78batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 12.88batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 12.83batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.81batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 12.95batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:08<00:00, 13.01batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 13.07batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:24<00:08,  8.49s/epoch, loss=0.00024, prev_loss=0.000247] 2021-09-16:18:29:04,158 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:29:05,252 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:29:20,308 INFO     [evaluator.py:657] Evaluation took 15.06s seconds
2021-09-16:18:29:20,449 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 13.38s/epoch, loss=0.00024, prev_loss=0.000247]
Training epochs on cuda: 100%|██████████| 10/10 [01:40<00:00, 10.09s/epoch, loss=0.00024, prev_loss=0.000247]
2021-09-16:18:29:20,450 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpdrj2aaak'
2021-09-16:18:29:20,522 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdrj2aaak' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.51ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.54ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.57ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.62ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:12, 2.48ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.52ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.56ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.58ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:11, 2.60ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.62ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.62ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.55ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.58ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.59ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.60ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:29:35,469 INFO     [evaluator.py:657] Evaluation took 14.93s seconds
[I 2021-09-16 18:29:35,475] Trial 47 finished with value: 0.9731190935614269 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.010778785241093441, 'optimizer.lr': 0.0413695750744088, 'negative_sampler.num_negs_per_pos': 53, 'training.batch_size': 2932}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:29:35,496 WARNING  [api.py:823] No random seed is specified. Setting to 2838663091.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:20,  4.90batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:16,  5.89batch/s]

Training batches on cuda:   5%|▌         | 5/100 [00:00<00:13,  6.88batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:11,  7.82batch/s]

Training batches on cuda:   9%|▉         | 9/100 [00:00<00:10,  8.67batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:01<00:09,  9.37batch/s]

Training batches on cuda:  13%|█▎        | 13/100 [00:01<00:08, 10.04batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:01<00:08, 10.46batch/s]

Training batches on cuda:  17%|█▋        | 17/100 [00:01<00:07, 10.85batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:01<00:07, 11.13batch/s]

Training batches on cuda:  21%|██        | 21/100 [00:01<00:06, 11.33batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:02<00:06, 11.39batch/s]

Training batches on cuda:  25%|██▌       | 25/100 [00:02<00:06, 11.30batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:02<00:06, 11.07batch/s]

Training batches on cuda:  29%|██▉       | 29/100 [00:02<00:06, 10.90batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:02<00:06, 11.01batch/s]

Training batches on cuda:  33%|███▎      | 33/100 [00:03<00:06, 10.84batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:03<00:05, 10.87batch/s]

Training batches on cuda:  37%|███▋      | 37/100 [00:03<00:05, 11.05batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:03<00:05, 11.13batch/s]

Training batches on cuda:  41%|████      | 41/100 [00:03<00:05, 11.03batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:03<00:05, 10.66batch/s]

Training batches on cuda:  45%|████▌     | 45/100 [00:04<00:05, 10.73batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:04<00:04, 10.82batch/s]

Training batches on cuda:  49%|████▉     | 49/100 [00:04<00:04, 10.78batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:04<00:04, 10.72batch/s]

Training batches on cuda:  53%|█████▎    | 53/100 [00:04<00:04, 10.68batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:05<00:04, 10.70batch/s]

Training batches on cuda:  57%|█████▋    | 57/100 [00:05<00:04, 10.63batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:05<00:03, 10.68batch/s]

Training batches on cuda:  61%|██████    | 61/100 [00:05<00:03, 10.72batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:05<00:03, 10.86batch/s]

Training batches on cuda:  65%|██████▌   | 65/100 [00:05<00:03, 10.93batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:06<00:02, 11.07batch/s]

Training batches on cuda:  69%|██████▉   | 69/100 [00:06<00:02, 11.12batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:06<00:02, 11.19batch/s]

Training batches on cuda:  73%|███████▎  | 73/100 [00:06<00:02, 11.17batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:06<00:02, 11.18batch/s]

Training batches on cuda:  77%|███████▋  | 77/100 [00:07<00:02, 11.16batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:07<00:01, 11.17batch/s]

Training batches on cuda:  81%|████████  | 81/100 [00:07<00:01, 11.09batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:07<00:01, 11.13batch/s]

Training batches on cuda:  85%|████████▌ | 85/100 [00:07<00:01, 11.13batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:07<00:01, 11.12batch/s]

Training batches on cuda:  89%|████████▉ | 89/100 [00:08<00:01, 10.86batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:08<00:00, 10.63batch/s]

Training batches on cuda:  93%|█████████▎| 93/100 [00:08<00:00, 10.60batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:08<00:00, 10.67batch/s]

Training batches on cuda:  97%|█████████▋| 97/100 [00:08<00:00, 10.52batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:09<00:00, 10.56batch/s]

                                                                             
Training epochs on cuda:   0%|          | 0/10 [00:09<?, ?epoch/s, loss=0.000712, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:09<01:24,  9.40s/epoch, loss=0.000712, prev_loss=nan]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:24,  4.01batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:19,  4.94batch/s]

Training batches on cuda:   5%|▌         | 5/100 [00:00<00:16,  5.92batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:13,  6.92batch/s]

Training batches on cuda:   9%|▉         | 9/100 [00:00<00:11,  7.80batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:01<00:10,  8.59batch/s]

Training batches on cuda:  13%|█▎        | 13/100 [00:01<00:09,  9.27batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:01<00:08,  9.69batch/s]

Training batches on cuda:  17%|█▋        | 17/100 [00:01<00:08, 10.07batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:01<00:07, 10.31batch/s]

Training batches on cuda:  21%|██        | 21/100 [00:02<00:07, 10.37batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:02<00:07, 10.59batch/s]

Training batches on cuda:  25%|██▌       | 25/100 [00:02<00:06, 10.80batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:02<00:06, 10.92batch/s]

Training batches on cuda:  29%|██▉       | 29/100 [00:02<00:06, 11.01batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:02<00:06, 10.97batch/s]

Training batches on cuda:  33%|███▎      | 33/100 [00:03<00:06, 10.88batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:03<00:05, 10.99batch/s]

Training batches on cuda:  37%|███▋      | 37/100 [00:03<00:05, 11.10batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:03<00:05, 11.09batch/s]

Training batches on cuda:  41%|████      | 41/100 [00:03<00:05, 11.01batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:04<00:05, 11.11batch/s]

Training batches on cuda:  45%|████▌     | 45/100 [00:04<00:05, 10.98batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:04<00:04, 11.05batch/s]

Training batches on cuda:  49%|████▉     | 49/100 [00:04<00:04, 11.00batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:04<00:04, 11.06batch/s]

Training batches on cuda:  53%|█████▎    | 53/100 [00:04<00:04, 11.13batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:05<00:04, 11.01batch/s]

Training batches on cuda:  57%|█████▋    | 57/100 [00:05<00:03, 11.06batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:05<00:03, 11.04batch/s]

Training batches on cuda:  61%|██████    | 61/100 [00:05<00:03, 11.10batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:05<00:03, 11.22batch/s]

Training batches on cuda:  65%|██████▌   | 65/100 [00:06<00:03, 11.29batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:06<00:02, 11.43batch/s]

Training batches on cuda:  69%|██████▉   | 69/100 [00:06<00:02, 11.37batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:06<00:02, 11.32batch/s]

Training batches on cuda:  73%|███████▎  | 73/100 [00:06<00:02, 11.25batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:06<00:02, 11.24batch/s]

Training batches on cuda:  77%|███████▋  | 77/100 [00:07<00:02, 11.28batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:07<00:01, 11.33batch/s]

Training batches on cuda:  81%|████████  | 81/100 [00:07<00:01, 11.38batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:07<00:01, 11.36batch/s]

Training batches on cuda:  85%|████████▌ | 85/100 [00:07<00:01, 11.22batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:07<00:01, 11.13batch/s]

Training batches on cuda:  89%|████████▉ | 89/100 [00:08<00:00, 11.17batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:08<00:00, 11.02batch/s]

Training batches on cuda:  93%|█████████▎| 93/100 [00:08<00:00, 11.07batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:08<00:00, 11.07batch/s]

Training batches on cuda:  97%|█████████▋| 97/100 [00:08<00:00, 11.10batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:09<00:00, 10.97batch/s]

                                                                             
Training epochs on cuda:  10%|█         | 1/10 [00:18<01:24,  9.40s/epoch, loss=0.00069, prev_loss=nan] 
Training epochs on cuda:  20%|██        | 2/10 [00:18<01:15,  9.40s/epoch, loss=0.00069, prev_loss=nan]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:23,  4.24batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:18,  5.16batch/s]

Training batches on cuda:   4%|▍         | 4/100 [00:00<00:16,  5.99batch/s]

Training batches on cuda:   6%|▌         | 6/100 [00:00<00:13,  6.92batch/s]

Training batches on cuda:   8%|▊         | 8/100 [00:00<00:11,  7.78batch/s]

Training batches on cuda:  10%|█         | 10/100 [00:01<00:10,  8.54batch/s]

Training batches on cuda:  12%|█▏        | 12/100 [00:01<00:09,  9.17batch/s]

Training batches on cuda:  14%|█▍        | 14/100 [00:01<00:08,  9.57batch/s]

Training batches on cuda:  16%|█▌        | 16/100 [00:01<00:08,  9.99batch/s]

Training batches on cuda:  18%|█▊        | 18/100 [00:01<00:07, 10.31batch/s]

Training batches on cuda:  20%|██        | 20/100 [00:01<00:07, 10.41batch/s]

Training batches on cuda:  22%|██▏       | 22/100 [00:02<00:07, 10.65batch/s]

Training batches on cuda:  24%|██▍       | 24/100 [00:02<00:06, 10.87batch/s]

Training batches on cuda:  26%|██▌       | 26/100 [00:02<00:06, 11.04batch/s]

Training batches on cuda:  28%|██▊       | 28/100 [00:02<00:06, 11.03batch/s]

Training batches on cuda:  30%|███       | 30/100 [00:02<00:06, 11.19batch/s]

Training batches on cuda:  32%|███▏      | 32/100 [00:03<00:06, 11.28batch/s]

Training batches on cuda:  34%|███▍      | 34/100 [00:03<00:05, 11.26batch/s]

Training batches on cuda:  36%|███▌      | 36/100 [00:03<00:05, 11.25batch/s]

Training batches on cuda:  38%|███▊      | 38/100 [00:03<00:05, 11.40batch/s]

Training batches on cuda:  40%|████      | 40/100 [00:03<00:05, 11.49batch/s]

Training batches on cuda:  42%|████▏     | 42/100 [00:03<00:05, 11.47batch/s]

Training batches on cuda:  44%|████▍     | 44/100 [00:04<00:04, 11.52batch/s]

Training batches on cuda:  46%|████▌     | 46/100 [00:04<00:04, 11.55batch/s]

Training batches on cuda:  48%|████▊     | 48/100 [00:04<00:04, 11.47batch/s]

Training batches on cuda:  50%|█████     | 50/100 [00:04<00:04, 11.42batch/s]

Training batches on cuda:  52%|█████▏    | 52/100 [00:04<00:04, 11.29batch/s]

Training batches on cuda:  54%|█████▍    | 54/100 [00:04<00:04, 11.32batch/s]

Training batches on cuda:  56%|█████▌    | 56/100 [00:05<00:03, 11.25batch/s]

Training batches on cuda:  58%|█████▊    | 58/100 [00:05<00:03, 11.30batch/s]

Training batches on cuda:  60%|██████    | 60/100 [00:05<00:03, 11.23batch/s]

Training batches on cuda:  62%|██████▏   | 62/100 [00:05<00:03, 11.26batch/s]

Training batches on cuda:  64%|██████▍   | 64/100 [00:05<00:03, 11.16batch/s]

Training batches on cuda:  66%|██████▌   | 66/100 [00:06<00:03, 11.17batch/s]

Training batches on cuda:  68%|██████▊   | 68/100 [00:06<00:02, 11.22batch/s]

Training batches on cuda:  70%|███████   | 70/100 [00:06<00:02, 11.26batch/s]

Training batches on cuda:  72%|███████▏  | 72/100 [00:06<00:02, 11.29batch/s]

Training batches on cuda:  74%|███████▍  | 74/100 [00:06<00:02, 11.28batch/s]

Training batches on cuda:  76%|███████▌  | 76/100 [00:06<00:02, 11.33batch/s]

Training batches on cuda:  78%|███████▊  | 78/100 [00:07<00:01, 11.35batch/s]

Training batches on cuda:  80%|████████  | 80/100 [00:07<00:01, 11.41batch/s]

Training batches on cuda:  82%|████████▏ | 82/100 [00:07<00:01, 11.44batch/s]

Training batches on cuda:  84%|████████▍ | 84/100 [00:07<00:01, 11.35batch/s]

Training batches on cuda:  86%|████████▌ | 86/100 [00:07<00:01, 11.12batch/s]

Training batches on cuda:  88%|████████▊ | 88/100 [00:08<00:01, 11.13batch/s]

Training batches on cuda:  90%|█████████ | 90/100 [00:08<00:00, 11.02batch/s]

Training batches on cuda:  92%|█████████▏| 92/100 [00:08<00:00, 11.18batch/s]

Training batches on cuda:  94%|█████████▍| 94/100 [00:08<00:00, 11.14batch/s]

Training batches on cuda:  96%|█████████▌| 96/100 [00:08<00:00, 11.21batch/s]

Training batches on cuda:  98%|█████████▊| 98/100 [00:08<00:00, 10.77batch/s]

Training batches on cuda: 100%|██████████| 100/100 [00:09<00:00, 12.32batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:28<01:15,  9.40s/epoch, loss=0.00069, prev_loss=0.00069]
Training epochs on cuda:  30%|███       | 3/10 [00:28<01:05,  9.39s/epoch, loss=0.00069, prev_loss=0.00069]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:19,  4.97batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:16,  5.92batch/s]

Training batches on cuda:   5%|▌         | 5/100 [00:00<00:13,  6.89batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:11,  7.76batch/s]

Training batches on cuda:   9%|▉         | 9/100 [00:00<00:10,  8.51batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:01<00:09,  9.11batch/s]

Training batches on cuda:  13%|█▎        | 13/100 [00:01<00:09,  9.48batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:01<00:08,  9.82batch/s]

Training batches on cuda:  17%|█▋        | 17/100 [00:01<00:08,  9.66batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:01<00:08,  9.92batch/s]

Training batches on cuda:  21%|██        | 21/100 [00:02<00:07, 10.01batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:02<00:07, 10.10batch/s]

Training batches on cuda:  25%|██▌       | 25/100 [00:02<00:07, 10.38batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:02<00:06, 10.56batch/s]

Training batches on cuda:  29%|██▉       | 29/100 [00:02<00:06, 10.67batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:03<00:06, 10.62batch/s]

Training batches on cuda:  33%|███▎      | 33/100 [00:03<00:06, 10.37batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:03<00:06, 10.49batch/s]

Training batches on cuda:  37%|███▋      | 37/100 [00:03<00:05, 10.53batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:03<00:05, 10.79batch/s]

Training batches on cuda:  41%|████      | 41/100 [00:03<00:05, 10.89batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:04<00:05, 11.12batch/s]

Training batches on cuda:  45%|████▌     | 45/100 [00:04<00:04, 11.14batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:04<00:04, 11.21batch/s]

Training batches on cuda:  49%|████▉     | 49/100 [00:04<00:04, 11.28batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:04<00:04, 11.35batch/s]

Training batches on cuda:  53%|█████▎    | 53/100 [00:05<00:04, 11.23batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:05<00:04, 11.24batch/s]

Training batches on cuda:  57%|█████▋    | 57/100 [00:05<00:03, 11.18batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:05<00:03, 10.90batch/s]

Training batches on cuda:  61%|██████    | 61/100 [00:05<00:03, 10.92batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:05<00:03, 10.93batch/s]

Training batches on cuda:  65%|██████▌   | 65/100 [00:06<00:03, 11.02batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:06<00:02, 11.11batch/s]

Training batches on cuda:  69%|██████▉   | 69/100 [00:06<00:02, 11.19batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:06<00:02, 11.25batch/s]

Training batches on cuda:  73%|███████▎  | 73/100 [00:06<00:02, 11.18batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:06<00:02, 11.20batch/s]

Training batches on cuda:  77%|███████▋  | 77/100 [00:07<00:02, 11.28batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:07<00:01, 11.27batch/s]

Training batches on cuda:  81%|████████  | 81/100 [00:07<00:01, 11.20batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:07<00:01, 11.20batch/s]

Training batches on cuda:  85%|████████▌ | 85/100 [00:07<00:01, 11.13batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:08<00:01, 11.22batch/s]

Training batches on cuda:  89%|████████▉ | 89/100 [00:08<00:00, 11.31batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:08<00:00, 11.32batch/s]

Training batches on cuda:  93%|█████████▎| 93/100 [00:08<00:00, 11.33batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:08<00:00, 11.20batch/s]

Training batches on cuda:  97%|█████████▋| 97/100 [00:08<00:00, 11.20batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:09<00:00, 11.02batch/s]

                                                                             
Training epochs on cuda:  30%|███       | 3/10 [00:37<01:05,  9.39s/epoch, loss=0.00069, prev_loss=0.00069]
Training epochs on cuda:  40%|████      | 4/10 [00:37<00:56,  9.41s/epoch, loss=0.00069, prev_loss=0.00069]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:21,  4.61batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:17,  5.60batch/s]

Training batches on cuda:   5%|▌         | 5/100 [00:00<00:14,  6.61batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:12,  7.55batch/s]

Training batches on cuda:   9%|▉         | 9/100 [00:00<00:10,  8.31batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:01<00:09,  9.08batch/s]

Training batches on cuda:  13%|█▎        | 13/100 [00:01<00:08,  9.67batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:01<00:08, 10.14batch/s]

Training batches on cuda:  17%|█▋        | 17/100 [00:01<00:07, 10.52batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:01<00:07, 10.74batch/s]

Training batches on cuda:  21%|██        | 21/100 [00:01<00:07, 10.86batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:02<00:07, 10.71batch/s]

Training batches on cuda:  25%|██▌       | 25/100 [00:02<00:07, 10.71batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:02<00:06, 10.79batch/s]

Training batches on cuda:  29%|██▉       | 29/100 [00:02<00:06, 10.97batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:02<00:06, 11.00batch/s]

Training batches on cuda:  33%|███▎      | 33/100 [00:03<00:06, 10.97batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:03<00:05, 10.95batch/s]

Training batches on cuda:  37%|███▋      | 37/100 [00:03<00:05, 10.93batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:03<00:05, 10.97batch/s]

Training batches on cuda:  41%|████      | 41/100 [00:03<00:05, 11.09batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:03<00:05, 11.17batch/s]

Training batches on cuda:  45%|████▌     | 45/100 [00:04<00:04, 11.24batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:04<00:04, 11.27batch/s]

Training batches on cuda:  49%|████▉     | 49/100 [00:04<00:04, 11.29batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:04<00:04, 11.15batch/s]

Training batches on cuda:  53%|█████▎    | 53/100 [00:04<00:04, 11.23batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:05<00:04, 11.25batch/s]

Training batches on cuda:  57%|█████▋    | 57/100 [00:05<00:03, 11.34batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:05<00:03, 11.35batch/s]

Training batches on cuda:  61%|██████    | 61/100 [00:05<00:03, 11.35batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:05<00:03, 11.34batch/s]

Training batches on cuda:  65%|██████▌   | 65/100 [00:05<00:03, 11.16batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:06<00:02, 11.09batch/s]

Training batches on cuda:  69%|██████▉   | 69/100 [00:06<00:02, 11.13batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:06<00:02, 11.20batch/s]

Training batches on cuda:  73%|███████▎  | 73/100 [00:06<00:02, 11.18batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:06<00:02, 11.17batch/s]

Training batches on cuda:  77%|███████▋  | 77/100 [00:07<00:02, 11.06batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:07<00:01, 11.08batch/s]

Training batches on cuda:  81%|████████  | 81/100 [00:07<00:01, 11.03batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:07<00:01, 11.09batch/s]

Training batches on cuda:  85%|████████▌ | 85/100 [00:07<00:01, 11.03batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:07<00:01, 11.08batch/s]

Training batches on cuda:  89%|████████▉ | 89/100 [00:08<00:00, 11.10batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:08<00:00, 11.10batch/s]

Training batches on cuda:  93%|█████████▎| 93/100 [00:08<00:00, 11.15batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:08<00:00, 11.22batch/s]

Training batches on cuda:  97%|█████████▋| 97/100 [00:08<00:00, 11.19batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:08<00:00, 11.17batch/s]

                                                                             
Training epochs on cuda:  40%|████      | 4/10 [00:46<00:56,  9.41s/epoch, loss=0.000689, prev_loss=0.00069]
Training epochs on cuda:  50%|█████     | 5/10 [00:46<00:46,  9.39s/epoch, loss=0.000689, prev_loss=0.00069]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:22,  4.46batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:17,  5.42batch/s]

Training batches on cuda:   5%|▌         | 5/100 [00:00<00:14,  6.46batch/s]

Training batches on cuda:   6%|▌         | 6/100 [00:00<00:13,  6.82batch/s]

Training batches on cuda:   8%|▊         | 8/100 [00:00<00:11,  7.78batch/s]

Training batches on cuda:  10%|█         | 10/100 [00:01<00:10,  8.58batch/s]

Training batches on cuda:  12%|█▏        | 12/100 [00:01<00:09,  9.25batch/s]

Training batches on cuda:  14%|█▍        | 14/100 [00:01<00:08,  9.80batch/s]

Training batches on cuda:  16%|█▌        | 16/100 [00:01<00:08, 10.22batch/s]

Training batches on cuda:  18%|█▊        | 18/100 [00:01<00:07, 10.47batch/s]

Training batches on cuda:  20%|██        | 20/100 [00:01<00:07, 10.68batch/s]

Training batches on cuda:  22%|██▏       | 22/100 [00:02<00:07, 10.89batch/s]

Training batches on cuda:  24%|██▍       | 24/100 [00:02<00:07, 10.82batch/s]

Training batches on cuda:  26%|██▌       | 26/100 [00:02<00:06, 10.90batch/s]

Training batches on cuda:  28%|██▊       | 28/100 [00:02<00:06, 10.94batch/s]

Training batches on cuda:  30%|███       | 30/100 [00:02<00:06, 10.96batch/s]

Training batches on cuda:  32%|███▏      | 32/100 [00:03<00:06, 11.05batch/s]

Training batches on cuda:  34%|███▍      | 34/100 [00:03<00:05, 11.18batch/s]

Training batches on cuda:  36%|███▌      | 36/100 [00:03<00:05, 11.08batch/s]

Training batches on cuda:  38%|███▊      | 38/100 [00:03<00:05, 11.18batch/s]

Training batches on cuda:  40%|████      | 40/100 [00:03<00:05, 11.30batch/s]

Training batches on cuda:  42%|████▏     | 42/100 [00:03<00:05, 11.38batch/s]

Training batches on cuda:  44%|████▍     | 44/100 [00:04<00:04, 11.29batch/s]

Training batches on cuda:  46%|████▌     | 46/100 [00:04<00:04, 11.17batch/s]

Training batches on cuda:  48%|████▊     | 48/100 [00:04<00:04, 11.03batch/s]

Training batches on cuda:  50%|█████     | 50/100 [00:04<00:04, 11.08batch/s]

Training batches on cuda:  52%|█████▏    | 52/100 [00:04<00:04, 11.16batch/s]

Training batches on cuda:  54%|█████▍    | 54/100 [00:04<00:04, 11.20batch/s]

Training batches on cuda:  56%|█████▌    | 56/100 [00:05<00:03, 11.25batch/s]

Training batches on cuda:  58%|█████▊    | 58/100 [00:05<00:03, 11.31batch/s]

Training batches on cuda:  60%|██████    | 60/100 [00:05<00:03, 11.20batch/s]

Training batches on cuda:  62%|██████▏   | 62/100 [00:05<00:03, 11.17batch/s]

Training batches on cuda:  64%|██████▍   | 64/100 [00:05<00:03, 11.23batch/s]

Training batches on cuda:  66%|██████▌   | 66/100 [00:06<00:03, 11.25batch/s]

Training batches on cuda:  68%|██████▊   | 68/100 [00:06<00:02, 11.14batch/s]

Training batches on cuda:  70%|███████   | 70/100 [00:06<00:02, 11.10batch/s]

Training batches on cuda:  72%|███████▏  | 72/100 [00:06<00:02, 10.96batch/s]

Training batches on cuda:  74%|███████▍  | 74/100 [00:06<00:02, 11.12batch/s]

Training batches on cuda:  76%|███████▌  | 76/100 [00:06<00:02, 11.08batch/s]

Training batches on cuda:  78%|███████▊  | 78/100 [00:07<00:01, 11.09batch/s]

Training batches on cuda:  80%|████████  | 80/100 [00:07<00:01, 11.20batch/s]

Training batches on cuda:  82%|████████▏ | 82/100 [00:07<00:01, 11.25batch/s]

Training batches on cuda:  84%|████████▍ | 84/100 [00:07<00:01, 11.24batch/s]

Training batches on cuda:  86%|████████▌ | 86/100 [00:07<00:01, 11.21batch/s]

Training batches on cuda:  88%|████████▊ | 88/100 [00:08<00:01, 11.16batch/s]

Training batches on cuda:  90%|█████████ | 90/100 [00:08<00:00, 11.14batch/s]

Training batches on cuda:  92%|█████████▏| 92/100 [00:08<00:00, 11.11batch/s]

Training batches on cuda:  94%|█████████▍| 94/100 [00:08<00:00, 11.18batch/s]

Training batches on cuda:  96%|█████████▌| 96/100 [00:08<00:00, 11.25batch/s]

Training batches on cuda:  98%|█████████▊| 98/100 [00:08<00:00, 11.22batch/s]

Training batches on cuda: 100%|██████████| 100/100 [00:09<00:00, 12.83batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:56<00:46,  9.39s/epoch, loss=0.000689, prev_loss=0.000689]
Training epochs on cuda:  60%|██████    | 6/10 [00:56<00:37,  9.37s/epoch, loss=0.000689, prev_loss=0.000689]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:22,  4.36batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:18,  5.30batch/s]

Training batches on cuda:   5%|▌         | 5/100 [00:00<00:15,  6.25batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:12,  7.24batch/s]

Training batches on cuda:   9%|▉         | 9/100 [00:00<00:11,  8.00batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:01<00:10,  8.74batch/s]

Training batches on cuda:  13%|█▎        | 13/100 [00:01<00:09,  9.34batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:01<00:08,  9.81batch/s]

Training batches on cuda:  17%|█▋        | 17/100 [00:01<00:08, 10.22batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:01<00:07, 10.54batch/s]

Training batches on cuda:  21%|██        | 21/100 [00:02<00:07, 10.80batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:02<00:07, 10.92batch/s]

Training batches on cuda:  25%|██▌       | 25/100 [00:02<00:06, 11.07batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:02<00:06, 11.09batch/s]

Training batches on cuda:  29%|██▉       | 29/100 [00:02<00:06, 11.06batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:02<00:06, 11.14batch/s]

Training batches on cuda:  33%|███▎      | 33/100 [00:03<00:05, 11.25batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:03<00:05, 11.23batch/s]

Training batches on cuda:  37%|███▋      | 37/100 [00:03<00:05, 11.22batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:03<00:05, 11.11batch/s]

Training batches on cuda:  41%|████      | 41/100 [00:03<00:05, 11.21batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:03<00:05, 11.26batch/s]

Training batches on cuda:  45%|████▌     | 45/100 [00:04<00:04, 11.15batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:04<00:04, 11.22batch/s]

Training batches on cuda:  49%|████▉     | 49/100 [00:04<00:04, 11.15batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:04<00:04, 11.15batch/s]

Training batches on cuda:  53%|█████▎    | 53/100 [00:04<00:04, 11.20batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:05<00:04, 11.16batch/s]

Training batches on cuda:  57%|█████▋    | 57/100 [00:05<00:03, 11.17batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:05<00:03, 11.22batch/s]

Training batches on cuda:  61%|██████    | 61/100 [00:05<00:03, 11.22batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:05<00:03, 11.11batch/s]

Training batches on cuda:  65%|██████▌   | 65/100 [00:05<00:03, 10.83batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:06<00:02, 11.10batch/s]

Training batches on cuda:  69%|██████▉   | 69/100 [00:06<00:02, 11.23batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:06<00:02, 11.28batch/s]

Training batches on cuda:  73%|███████▎  | 73/100 [00:06<00:02, 11.29batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:06<00:02, 11.34batch/s]

Training batches on cuda:  77%|███████▋  | 77/100 [00:07<00:02, 11.30batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:07<00:01, 11.27batch/s]

Training batches on cuda:  81%|████████  | 81/100 [00:07<00:01, 11.32batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:07<00:01, 11.29batch/s]

Training batches on cuda:  85%|████████▌ | 85/100 [00:07<00:01, 11.07batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:07<00:01, 11.13batch/s]

Training batches on cuda:  89%|████████▉ | 89/100 [00:08<00:00, 11.08batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:08<00:00, 11.14batch/s]

Training batches on cuda:  93%|█████████▎| 93/100 [00:08<00:00, 11.19batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:08<00:00, 11.06batch/s]

Training batches on cuda:  97%|█████████▋| 97/100 [00:08<00:00, 11.01batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:09<00:00, 10.89batch/s]

                                                                             
Training epochs on cuda:  60%|██████    | 6/10 [01:05<00:37,  9.37s/epoch, loss=0.00069, prev_loss=0.000689] 
Training epochs on cuda:  70%|███████   | 7/10 [01:05<00:28,  9.37s/epoch, loss=0.00069, prev_loss=0.000689]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:23,  4.28batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:18,  5.24batch/s]

Training batches on cuda:   4%|▍         | 4/100 [00:00<00:15,  6.06batch/s]

Training batches on cuda:   6%|▌         | 6/100 [00:00<00:13,  6.86batch/s]

Training batches on cuda:   8%|▊         | 8/100 [00:00<00:11,  7.78batch/s]

Training batches on cuda:  10%|█         | 10/100 [00:01<00:10,  8.55batch/s]

Training batches on cuda:  12%|█▏        | 12/100 [00:01<00:09,  9.09batch/s]

Training batches on cuda:  14%|█▍        | 14/100 [00:01<00:08,  9.62batch/s]

Training batches on cuda:  16%|█▌        | 16/100 [00:01<00:08, 10.06batch/s]

Training batches on cuda:  18%|█▊        | 18/100 [00:01<00:07, 10.31batch/s]

Training batches on cuda:  20%|██        | 20/100 [00:01<00:07, 10.62batch/s]

Training batches on cuda:  22%|██▏       | 22/100 [00:02<00:07, 10.82batch/s]

Training batches on cuda:  24%|██▍       | 24/100 [00:02<00:07, 10.85batch/s]

Training batches on cuda:  26%|██▌       | 26/100 [00:02<00:06, 10.91batch/s]

Training batches on cuda:  28%|██▊       | 28/100 [00:02<00:06, 10.92batch/s]

Training batches on cuda:  30%|███       | 30/100 [00:02<00:06, 11.04batch/s]

Training batches on cuda:  32%|███▏      | 32/100 [00:03<00:06, 11.27batch/s]

Training batches on cuda:  34%|███▍      | 34/100 [00:03<00:05, 11.39batch/s]

Training batches on cuda:  36%|███▌      | 36/100 [00:03<00:05, 11.51batch/s]

Training batches on cuda:  38%|███▊      | 38/100 [00:03<00:05, 11.31batch/s]

Training batches on cuda:  40%|████      | 40/100 [00:03<00:05, 11.23batch/s]

Training batches on cuda:  42%|████▏     | 42/100 [00:03<00:05, 11.24batch/s]

Training batches on cuda:  44%|████▍     | 44/100 [00:04<00:05, 11.18batch/s]

Training batches on cuda:  46%|████▌     | 46/100 [00:04<00:04, 11.17batch/s]

Training batches on cuda:  48%|████▊     | 48/100 [00:04<00:04, 11.20batch/s]

Training batches on cuda:  50%|█████     | 50/100 [00:04<00:04, 11.21batch/s]

Training batches on cuda:  52%|█████▏    | 52/100 [00:04<00:04, 10.98batch/s]

Training batches on cuda:  54%|█████▍    | 54/100 [00:05<00:04, 11.00batch/s]

Training batches on cuda:  56%|█████▌    | 56/100 [00:05<00:03, 11.07batch/s]

Training batches on cuda:  58%|█████▊    | 58/100 [00:05<00:03, 10.98batch/s]

Training batches on cuda:  60%|██████    | 60/100 [00:05<00:03, 11.02batch/s]

Training batches on cuda:  62%|██████▏   | 62/100 [00:05<00:03, 11.19batch/s]

Training batches on cuda:  64%|██████▍   | 64/100 [00:05<00:03, 11.21batch/s]

Training batches on cuda:  66%|██████▌   | 66/100 [00:06<00:03, 11.32batch/s]

Training batches on cuda:  68%|██████▊   | 68/100 [00:06<00:02, 11.11batch/s]

Training batches on cuda:  70%|███████   | 70/100 [00:06<00:02, 11.09batch/s]

Training batches on cuda:  72%|███████▏  | 72/100 [00:06<00:02, 11.06batch/s]

Training batches on cuda:  74%|███████▍  | 74/100 [00:06<00:02, 11.03batch/s]

Training batches on cuda:  76%|███████▌  | 76/100 [00:07<00:02, 10.99batch/s]

Training batches on cuda:  78%|███████▊  | 78/100 [00:07<00:02, 11.00batch/s]

Training batches on cuda:  80%|████████  | 80/100 [00:07<00:01, 10.98batch/s]

Training batches on cuda:  82%|████████▏ | 82/100 [00:07<00:01, 10.95batch/s]

Training batches on cuda:  84%|████████▍ | 84/100 [00:07<00:01, 10.93batch/s]

Training batches on cuda:  86%|████████▌ | 86/100 [00:07<00:01, 10.97batch/s]

Training batches on cuda:  88%|████████▊ | 88/100 [00:08<00:01, 10.93batch/s]

Training batches on cuda:  90%|█████████ | 90/100 [00:08<00:00, 10.90batch/s]

Training batches on cuda:  92%|█████████▏| 92/100 [00:08<00:00, 10.81batch/s]

Training batches on cuda:  94%|█████████▍| 94/100 [00:08<00:00, 10.64batch/s]

Training batches on cuda:  96%|█████████▌| 96/100 [00:08<00:00, 10.74batch/s]

Training batches on cuda:  98%|█████████▊| 98/100 [00:09<00:00, 10.88batch/s]

                                                                             
Training epochs on cuda:  70%|███████   | 7/10 [01:15<00:28,  9.37s/epoch, loss=0.00069, prev_loss=0.00069] 
Training epochs on cuda:  80%|████████  | 8/10 [01:15<00:18,  9.37s/epoch, loss=0.00069, prev_loss=0.00069]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:20,  4.81batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:16,  5.76batch/s]

Training batches on cuda:   5%|▌         | 5/100 [00:00<00:14,  6.77batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:12,  7.72batch/s]

Training batches on cuda:   9%|▉         | 9/100 [00:00<00:10,  8.56batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:01<00:09,  9.26batch/s]

Training batches on cuda:  13%|█▎        | 13/100 [00:01<00:08,  9.71batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:01<00:08, 10.18batch/s]

Training batches on cuda:  17%|█▋        | 17/100 [00:01<00:07, 10.55batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:01<00:07, 10.80batch/s]

Training batches on cuda:  21%|██        | 21/100 [00:01<00:07, 11.00batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:02<00:06, 11.06batch/s]

Training batches on cuda:  25%|██▌       | 25/100 [00:02<00:06, 11.06batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:02<00:06, 11.11batch/s]

Training batches on cuda:  29%|██▉       | 29/100 [00:02<00:06, 11.17batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:02<00:06, 11.27batch/s]

Training batches on cuda:  33%|███▎      | 33/100 [00:03<00:05, 11.35batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:03<00:05, 11.40batch/s]

Training batches on cuda:  37%|███▋      | 37/100 [00:03<00:05, 11.37batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:03<00:05, 11.31batch/s]

Training batches on cuda:  41%|████      | 41/100 [00:03<00:05, 11.32batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:03<00:05, 11.25batch/s]

Training batches on cuda:  45%|████▌     | 45/100 [00:04<00:04, 11.20batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:04<00:04, 11.29batch/s]

Training batches on cuda:  49%|████▉     | 49/100 [00:04<00:04, 11.38batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:04<00:04, 11.35batch/s]

Training batches on cuda:  53%|█████▎    | 53/100 [00:04<00:04, 11.33batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:04<00:03, 11.31batch/s]

Training batches on cuda:  57%|█████▋    | 57/100 [00:05<00:03, 11.15batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:05<00:03, 11.09batch/s]

Training batches on cuda:  61%|██████    | 61/100 [00:05<00:03, 11.14batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:05<00:03, 11.11batch/s]

Training batches on cuda:  65%|██████▌   | 65/100 [00:05<00:03, 11.05batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:06<00:02, 11.10batch/s]

Training batches on cuda:  69%|██████▉   | 69/100 [00:06<00:02, 11.16batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:06<00:02, 11.19batch/s]

Training batches on cuda:  73%|███████▎  | 73/100 [00:06<00:02, 11.15batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:06<00:02, 11.18batch/s]

Training batches on cuda:  77%|███████▋  | 77/100 [00:06<00:02, 11.14batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:07<00:01, 11.14batch/s]

Training batches on cuda:  81%|████████  | 81/100 [00:07<00:01, 11.18batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:07<00:01, 11.27batch/s]

Training batches on cuda:  85%|████████▌ | 85/100 [00:07<00:01, 11.34batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:07<00:01, 11.41batch/s]

Training batches on cuda:  89%|████████▉ | 89/100 [00:08<00:00, 11.47batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:08<00:00, 11.56batch/s]

Training batches on cuda:  93%|█████████▎| 93/100 [00:08<00:00, 11.67batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:08<00:00, 11.75batch/s]

Training batches on cuda:  97%|█████████▋| 97/100 [00:08<00:00, 11.76batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:08<00:00, 11.71batch/s]

                                                                             
Training epochs on cuda:  80%|████████  | 8/10 [01:24<00:18,  9.37s/epoch, loss=0.00069, prev_loss=0.00069]
Training epochs on cuda:  90%|█████████ | 9/10 [01:24<00:09,  9.29s/epoch, loss=0.00069, prev_loss=0.00069]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/100 [00:00<00:19,  5.03batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:16,  6.01batch/s]

Training batches on cuda:   5%|▌         | 5/100 [00:00<00:13,  7.00batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:11,  7.87batch/s]

Training batches on cuda:   9%|▉         | 9/100 [00:00<00:10,  8.58batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:01<00:09,  9.29batch/s]

Training batches on cuda:  13%|█▎        | 13/100 [00:01<00:08,  9.81batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:01<00:08, 10.20batch/s]

Training batches on cuda:  17%|█▋        | 17/100 [00:01<00:07, 10.58batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:01<00:07, 10.81batch/s]

Training batches on cuda:  21%|██        | 21/100 [00:01<00:07, 11.00batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:02<00:06, 11.04batch/s]

Training batches on cuda:  25%|██▌       | 25/100 [00:02<00:06, 11.08batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:02<00:06, 11.14batch/s]

Training batches on cuda:  29%|██▉       | 29/100 [00:02<00:06, 11.20batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:02<00:06, 11.18batch/s]

Training batches on cuda:  33%|███▎      | 33/100 [00:03<00:05, 11.28batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:03<00:05, 11.35batch/s]

Training batches on cuda:  37%|███▋      | 37/100 [00:03<00:05, 11.42batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:03<00:05, 11.35batch/s]

Training batches on cuda:  41%|████      | 41/100 [00:03<00:05, 11.30batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:03<00:05, 10.73batch/s]

Training batches on cuda:  45%|████▌     | 45/100 [00:04<00:05, 10.84batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:04<00:04, 11.04batch/s]

Training batches on cuda:  49%|████▉     | 49/100 [00:04<00:04, 11.16batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:04<00:04, 11.18batch/s]

Training batches on cuda:  53%|█████▎    | 53/100 [00:04<00:04, 11.20batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:05<00:04, 11.21batch/s]

Training batches on cuda:  57%|█████▋    | 57/100 [00:05<00:03, 11.21batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:05<00:03, 11.22batch/s]

Training batches on cuda:  61%|██████    | 61/100 [00:05<00:03, 11.29batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:05<00:03, 11.32batch/s]

Training batches on cuda:  65%|██████▌   | 65/100 [00:05<00:03, 11.38batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:06<00:02, 11.38batch/s]

Training batches on cuda:  69%|██████▉   | 69/100 [00:06<00:02, 11.39batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:06<00:02, 11.27batch/s]

Training batches on cuda:  73%|███████▎  | 73/100 [00:06<00:02, 11.29batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:06<00:02, 11.28batch/s]

Training batches on cuda:  77%|███████▋  | 77/100 [00:06<00:02, 11.22batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:07<00:01, 11.22batch/s]

Training batches on cuda:  81%|████████  | 81/100 [00:07<00:01, 11.30batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:07<00:01, 11.36batch/s]

Training batches on cuda:  85%|████████▌ | 85/100 [00:07<00:01, 11.41batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:07<00:01, 11.49batch/s]

Training batches on cuda:  89%|████████▉ | 89/100 [00:08<00:00, 11.43batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:08<00:00, 11.40batch/s]

Training batches on cuda:  93%|█████████▎| 93/100 [00:08<00:00, 11.34batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:08<00:00, 11.21batch/s]

Training batches on cuda:  97%|█████████▋| 97/100 [00:08<00:00, 11.07batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:08<00:00, 11.15batch/s]

                                                                             
Training epochs on cuda:  90%|█████████ | 9/10 [01:33<00:09,  9.29s/epoch, loss=0.00069, prev_loss=0.00069]2021-09-16:18:31:09,468 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:31:10,654 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:31:25,679 INFO     [evaluator.py:657] Evaluation took 15.02s seconds
2021-09-16:18:31:25,827 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 14.16s/epoch, loss=0.00069, prev_loss=0.00069]
Training epochs on cuda: 100%|██████████| 10/10 [01:49<00:00, 10.97s/epoch, loss=0.00069, prev_loss=0.00069]
2021-09-16:18:31:25,828 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpbcyu4cdy'
2021-09-16:18:31:25,889 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbcyu4cdy' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.53ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.52ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.55ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.57ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.62ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.59ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.60ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.61ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.62ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.61ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.62ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:31:40,799 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
[I 2021-09-16 18:31:40,804] Trial 48 finished with value: -0.0034689972793873203 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.5184789021544637, 'optimizer.lr': 0.03078013783955686, 'negative_sampler.num_negs_per_pos': 56, 'training.batch_size': 3132}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:31:40,827 WARNING  [api.py:823] No random seed is specified. Setting to 4287632714.
2021-09-16:18:31:41,182 INFO     [training_loop.py:972] Starting sub_batch_size search for training now...
2021-09-16:18:31:41,774 INFO     [training_loop.py:1004] Concluded search with sub_batch_size 1616.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:11,  8.26batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:10,  8.81batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:09,  9.36batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:09,  9.70batch/s]

Training batches on cuda:   9%|▉         | 9/97 [00:00<00:08,  9.97batch/s]

Training batches on cuda:  11%|█▏        | 11/97 [00:01<00:08, 10.19batch/s]

Training batches on cuda:  13%|█▎        | 13/97 [00:01<00:08, 10.30batch/s]

Training batches on cuda:  15%|█▌        | 15/97 [00:01<00:07, 10.38batch/s]

Training batches on cuda:  18%|█▊        | 17/97 [00:01<00:07, 10.38batch/s]

Training batches on cuda:  20%|█▉        | 19/97 [00:01<00:07, 10.44batch/s]

Training batches on cuda:  22%|██▏       | 21/97 [00:02<00:07, 10.52batch/s]

Training batches on cuda:  24%|██▎       | 23/97 [00:02<00:06, 10.63batch/s]

Training batches on cuda:  26%|██▌       | 25/97 [00:02<00:06, 10.71batch/s]

Training batches on cuda:  28%|██▊       | 27/97 [00:02<00:06, 10.75batch/s]

Training batches on cuda:  30%|██▉       | 29/97 [00:02<00:06, 10.30batch/s]

Training batches on cuda:  32%|███▏      | 31/97 [00:02<00:06, 10.30batch/s]

Training batches on cuda:  34%|███▍      | 33/97 [00:03<00:06, 10.33batch/s]

Training batches on cuda:  36%|███▌      | 35/97 [00:03<00:05, 10.46batch/s]

Training batches on cuda:  38%|███▊      | 37/97 [00:03<00:05, 10.53batch/s]

Training batches on cuda:  40%|████      | 39/97 [00:03<00:05, 10.52batch/s]

Training batches on cuda:  42%|████▏     | 41/97 [00:03<00:05, 10.54batch/s]

Training batches on cuda:  44%|████▍     | 43/97 [00:04<00:05, 10.63batch/s]

Training batches on cuda:  46%|████▋     | 45/97 [00:04<00:04, 10.68batch/s]

Training batches on cuda:  48%|████▊     | 47/97 [00:04<00:04, 10.72batch/s]

Training batches on cuda:  51%|█████     | 49/97 [00:04<00:04, 10.64batch/s]

Training batches on cuda:  53%|█████▎    | 51/97 [00:04<00:04, 10.65batch/s]

Training batches on cuda:  55%|█████▍    | 53/97 [00:05<00:04, 10.71batch/s]

Training batches on cuda:  57%|█████▋    | 55/97 [00:05<00:03, 10.66batch/s]

Training batches on cuda:  59%|█████▉    | 57/97 [00:05<00:03, 10.58batch/s]

Training batches on cuda:  61%|██████    | 59/97 [00:05<00:03, 10.68batch/s]

Training batches on cuda:  63%|██████▎   | 61/97 [00:05<00:03, 10.44batch/s]

Training batches on cuda:  65%|██████▍   | 63/97 [00:06<00:03, 10.16batch/s]

Training batches on cuda:  67%|██████▋   | 65/97 [00:06<00:03, 10.00batch/s]

Training batches on cuda:  69%|██████▉   | 67/97 [00:06<00:02, 10.19batch/s]

Training batches on cuda:  71%|███████   | 69/97 [00:06<00:02, 10.35batch/s]

Training batches on cuda:  73%|███████▎  | 71/97 [00:06<00:02, 10.45batch/s]

Training batches on cuda:  75%|███████▌  | 73/97 [00:06<00:02, 10.46batch/s]

Training batches on cuda:  77%|███████▋  | 75/97 [00:07<00:02, 10.50batch/s]

Training batches on cuda:  79%|███████▉  | 77/97 [00:07<00:01, 10.49batch/s]

Training batches on cuda:  81%|████████▏ | 79/97 [00:07<00:01, 10.63batch/s]

Training batches on cuda:  84%|████████▎ | 81/97 [00:07<00:01, 10.76batch/s]

Training batches on cuda:  86%|████████▌ | 83/97 [00:07<00:01, 10.82batch/s]

Training batches on cuda:  88%|████████▊ | 85/97 [00:08<00:01, 10.63batch/s]

Training batches on cuda:  90%|████████▉ | 87/97 [00:08<00:00, 10.69batch/s]

Training batches on cuda:  92%|█████████▏| 89/97 [00:08<00:00, 10.68batch/s]

Training batches on cuda:  94%|█████████▍| 91/97 [00:08<00:00, 10.73batch/s]

Training batches on cuda:  96%|█████████▌| 93/97 [00:08<00:00, 10.74batch/s]

Training batches on cuda:  98%|█████████▊| 95/97 [00:09<00:00, 10.77batch/s]

Training batches on cuda: 100%|██████████| 97/97 [00:09<00:00, 12.33batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:09<?, ?epoch/s, loss=0.000631, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:09<01:24,  9.34s/epoch, loss=0.000631, prev_loss=nan]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:14,  6.74batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:12,  7.53batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:11,  8.28batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:10,  8.95batch/s]

Training batches on cuda:   9%|▉         | 9/97 [00:00<00:09,  9.48batch/s]

Training batches on cuda:  11%|█▏        | 11/97 [00:01<00:08,  9.97batch/s]

Training batches on cuda:  13%|█▎        | 13/97 [00:01<00:08, 10.26batch/s]

Training batches on cuda:  15%|█▌        | 15/97 [00:01<00:07, 10.39batch/s]

Training batches on cuda:  18%|█▊        | 17/97 [00:01<00:07, 10.40batch/s]

Training batches on cuda:  20%|█▉        | 19/97 [00:01<00:07, 10.49batch/s]

Training batches on cuda:  22%|██▏       | 21/97 [00:01<00:07, 10.59batch/s]

Training batches on cuda:  24%|██▎       | 23/97 [00:02<00:06, 10.65batch/s]

Training batches on cuda:  26%|██▌       | 25/97 [00:02<00:06, 10.40batch/s]

Training batches on cuda:  28%|██▊       | 27/97 [00:02<00:06, 10.54batch/s]

Training batches on cuda:  30%|██▉       | 29/97 [00:02<00:06, 10.52batch/s]

Training batches on cuda:  32%|███▏      | 31/97 [00:02<00:06, 10.47batch/s]

Training batches on cuda:  34%|███▍      | 33/97 [00:03<00:06, 10.52batch/s]

Training batches on cuda:  36%|███▌      | 35/97 [00:03<00:05, 10.62batch/s]

Training batches on cuda:  38%|███▊      | 37/97 [00:03<00:05, 10.53batch/s]

Training batches on cuda:  40%|████      | 39/97 [00:03<00:05, 10.62batch/s]

Training batches on cuda:  42%|████▏     | 41/97 [00:03<00:05, 10.59batch/s]

Training batches on cuda:  44%|████▍     | 43/97 [00:04<00:05, 10.51batch/s]

Training batches on cuda:  46%|████▋     | 45/97 [00:04<00:04, 10.58batch/s]

Training batches on cuda:  48%|████▊     | 47/97 [00:04<00:04, 10.70batch/s]

Training batches on cuda:  51%|█████     | 49/97 [00:04<00:04, 10.66batch/s]

Training batches on cuda:  53%|█████▎    | 51/97 [00:04<00:04, 10.65batch/s]

Training batches on cuda:  55%|█████▍    | 53/97 [00:05<00:04, 10.68batch/s]

Training batches on cuda:  57%|█████▋    | 55/97 [00:05<00:03, 10.69batch/s]

Training batches on cuda:  59%|█████▉    | 57/97 [00:05<00:03, 10.56batch/s]

Training batches on cuda:  61%|██████    | 59/97 [00:05<00:03, 10.47batch/s]

Training batches on cuda:  63%|██████▎   | 61/97 [00:05<00:03, 10.40batch/s]

Training batches on cuda:  65%|██████▍   | 63/97 [00:05<00:03, 10.33batch/s]

Training batches on cuda:  67%|██████▋   | 65/97 [00:06<00:03, 10.49batch/s]

Training batches on cuda:  69%|██████▉   | 67/97 [00:06<00:02, 10.55batch/s]

Training batches on cuda:  71%|███████   | 69/97 [00:06<00:02, 10.58batch/s]

Training batches on cuda:  73%|███████▎  | 71/97 [00:06<00:02, 10.66batch/s]

Training batches on cuda:  75%|███████▌  | 73/97 [00:06<00:02, 10.64batch/s]

Training batches on cuda:  77%|███████▋  | 75/97 [00:07<00:02, 10.69batch/s]

Training batches on cuda:  79%|███████▉  | 77/97 [00:07<00:01, 10.71batch/s]

Training batches on cuda:  81%|████████▏ | 79/97 [00:07<00:01, 10.68batch/s]

Training batches on cuda:  84%|████████▎ | 81/97 [00:07<00:01, 10.66batch/s]

Training batches on cuda:  86%|████████▌ | 83/97 [00:07<00:01, 10.56batch/s]

Training batches on cuda:  88%|████████▊ | 85/97 [00:08<00:01, 10.60batch/s]

Training batches on cuda:  90%|████████▉ | 87/97 [00:08<00:00, 10.65batch/s]

Training batches on cuda:  92%|█████████▏| 89/97 [00:08<00:00, 10.63batch/s]

Training batches on cuda:  94%|█████████▍| 91/97 [00:08<00:00, 10.48batch/s]

Training batches on cuda:  96%|█████████▌| 93/97 [00:08<00:00, 10.37batch/s]

Training batches on cuda:  98%|█████████▊| 95/97 [00:09<00:00, 10.08batch/s]

Training batches on cuda: 100%|██████████| 97/97 [00:09<00:00, 11.26batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:18<01:24,  9.34s/epoch, loss=0.000625, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:18<01:14,  9.35s/epoch, loss=0.000625, prev_loss=nan]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:14,  6.70batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:12,  7.47batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:11,  8.17batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:10,  8.69batch/s]

Training batches on cuda:   9%|▉         | 9/97 [00:00<00:09,  9.14batch/s]

Training batches on cuda:  11%|█▏        | 11/97 [00:01<00:09,  9.50batch/s]

Training batches on cuda:  13%|█▎        | 13/97 [00:01<00:08,  9.80batch/s]

Training batches on cuda:  14%|█▍        | 14/97 [00:01<00:08,  9.85batch/s]

Training batches on cuda:  16%|█▋        | 16/97 [00:01<00:08, 10.09batch/s]

Training batches on cuda:  19%|█▊        | 18/97 [00:01<00:07, 10.32batch/s]

Training batches on cuda:  21%|██        | 20/97 [00:01<00:07, 10.48batch/s]

Training batches on cuda:  23%|██▎       | 22/97 [00:02<00:07, 10.39batch/s]

Training batches on cuda:  25%|██▍       | 24/97 [00:02<00:06, 10.45batch/s]

Training batches on cuda:  27%|██▋       | 26/97 [00:02<00:06, 10.57batch/s]

Training batches on cuda:  29%|██▉       | 28/97 [00:02<00:06, 10.48batch/s]

Training batches on cuda:  31%|███       | 30/97 [00:02<00:06, 10.44batch/s]

Training batches on cuda:  33%|███▎      | 32/97 [00:03<00:06, 10.44batch/s]

Training batches on cuda:  35%|███▌      | 34/97 [00:03<00:06, 10.48batch/s]

Training batches on cuda:  37%|███▋      | 36/97 [00:03<00:05, 10.48batch/s]

Training batches on cuda:  39%|███▉      | 38/97 [00:03<00:05, 10.42batch/s]

Training batches on cuda:  41%|████      | 40/97 [00:03<00:05, 10.37batch/s]

Training batches on cuda:  43%|████▎     | 42/97 [00:04<00:05,  9.72batch/s]

Training batches on cuda:  45%|████▌     | 44/97 [00:04<00:05,  9.98batch/s]

Training batches on cuda:  47%|████▋     | 46/97 [00:04<00:04, 10.21batch/s]

Training batches on cuda:  49%|████▉     | 48/97 [00:04<00:04, 10.28batch/s]

Training batches on cuda:  52%|█████▏    | 50/97 [00:04<00:04, 10.35batch/s]

Training batches on cuda:  54%|█████▎    | 52/97 [00:05<00:04, 10.46batch/s]

Training batches on cuda:  56%|█████▌    | 54/97 [00:05<00:04, 10.50batch/s]

Training batches on cuda:  58%|█████▊    | 56/97 [00:05<00:03, 10.62batch/s]

Training batches on cuda:  60%|█████▉    | 58/97 [00:05<00:03, 10.65batch/s]

Training batches on cuda:  62%|██████▏   | 60/97 [00:05<00:03, 10.65batch/s]

Training batches on cuda:  64%|██████▍   | 62/97 [00:05<00:03, 10.74batch/s]

Training batches on cuda:  66%|██████▌   | 64/97 [00:06<00:03, 10.64batch/s]

Training batches on cuda:  68%|██████▊   | 66/97 [00:06<00:02, 10.63batch/s]

Training batches on cuda:  70%|███████   | 68/97 [00:06<00:02, 10.61batch/s]

Training batches on cuda:  72%|███████▏  | 70/97 [00:06<00:02, 10.32batch/s]

Training batches on cuda:  74%|███████▍  | 72/97 [00:06<00:02, 10.19batch/s]

Training batches on cuda:  76%|███████▋  | 74/97 [00:07<00:02,  9.90batch/s]

Training batches on cuda:  78%|███████▊  | 76/97 [00:07<00:02, 10.12batch/s]

Training batches on cuda:  80%|████████  | 78/97 [00:07<00:01, 10.33batch/s]

Training batches on cuda:  82%|████████▏ | 80/97 [00:07<00:01, 10.52batch/s]

Training batches on cuda:  85%|████████▍ | 82/97 [00:07<00:01, 10.58batch/s]

Training batches on cuda:  87%|████████▋ | 84/97 [00:08<00:01, 10.60batch/s]

Training batches on cuda:  89%|████████▊ | 86/97 [00:08<00:01, 10.56batch/s]

Training batches on cuda:  91%|█████████ | 88/97 [00:08<00:00, 10.55batch/s]

Training batches on cuda:  93%|█████████▎| 90/97 [00:08<00:00, 10.42batch/s]

Training batches on cuda:  95%|█████████▍| 92/97 [00:08<00:00, 10.43batch/s]

Training batches on cuda:  97%|█████████▋| 94/97 [00:09<00:00, 10.40batch/s]

Training batches on cuda:  99%|█████████▉| 96/97 [00:09<00:00, 10.28batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:28<01:14,  9.35s/epoch, loss=0.000625, prev_loss=0.000625]
Training epochs on cuda:  30%|███       | 3/10 [00:28<01:05,  9.39s/epoch, loss=0.000625, prev_loss=0.000625]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:12,  7.67batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:11,  8.24batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:10,  8.82batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:09,  9.33batch/s]

Training batches on cuda:   9%|▉         | 9/97 [00:00<00:09,  9.63batch/s]

Training batches on cuda:  11%|█▏        | 11/97 [00:01<00:08,  9.94batch/s]

Training batches on cuda:  13%|█▎        | 13/97 [00:01<00:08, 10.13batch/s]

Training batches on cuda:  15%|█▌        | 15/97 [00:01<00:08, 10.20batch/s]

Training batches on cuda:  18%|█▊        | 17/97 [00:01<00:07, 10.36batch/s]

Training batches on cuda:  20%|█▉        | 19/97 [00:01<00:07, 10.26batch/s]

Training batches on cuda:  22%|██▏       | 21/97 [00:02<00:07, 10.42batch/s]

Training batches on cuda:  24%|██▎       | 23/97 [00:02<00:07, 10.50batch/s]

Training batches on cuda:  26%|██▌       | 25/97 [00:02<00:06, 10.59batch/s]

Training batches on cuda:  28%|██▊       | 27/97 [00:02<00:06, 10.63batch/s]

Training batches on cuda:  30%|██▉       | 29/97 [00:02<00:06, 10.67batch/s]

Training batches on cuda:  32%|███▏      | 31/97 [00:02<00:06, 10.40batch/s]

Training batches on cuda:  34%|███▍      | 33/97 [00:03<00:06, 10.43batch/s]

Training batches on cuda:  36%|███▌      | 35/97 [00:03<00:05, 10.40batch/s]

Training batches on cuda:  38%|███▊      | 37/97 [00:03<00:05, 10.35batch/s]

Training batches on cuda:  40%|████      | 39/97 [00:03<00:05, 10.45batch/s]

Training batches on cuda:  42%|████▏     | 41/97 [00:03<00:05, 10.48batch/s]

Training batches on cuda:  44%|████▍     | 43/97 [00:04<00:05, 10.45batch/s]

Training batches on cuda:  46%|████▋     | 45/97 [00:04<00:04, 10.44batch/s]

Training batches on cuda:  48%|████▊     | 47/97 [00:04<00:04, 10.45batch/s]

Training batches on cuda:  51%|█████     | 49/97 [00:04<00:04, 10.39batch/s]

Training batches on cuda:  53%|█████▎    | 51/97 [00:04<00:04, 10.35batch/s]

Training batches on cuda:  55%|█████▍    | 53/97 [00:05<00:04, 10.19batch/s]

Training batches on cuda:  57%|█████▋    | 55/97 [00:05<00:04, 10.37batch/s]

Training batches on cuda:  59%|█████▉    | 57/97 [00:05<00:03, 10.41batch/s]

Training batches on cuda:  61%|██████    | 59/97 [00:05<00:03, 10.53batch/s]

Training batches on cuda:  63%|██████▎   | 61/97 [00:05<00:03, 10.60batch/s]

Training batches on cuda:  65%|██████▍   | 63/97 [00:06<00:03, 10.61batch/s]

Training batches on cuda:  67%|██████▋   | 65/97 [00:06<00:03, 10.57batch/s]

Training batches on cuda:  69%|██████▉   | 67/97 [00:06<00:02, 10.61batch/s]

Training batches on cuda:  71%|███████   | 69/97 [00:06<00:02, 10.54batch/s]

Training batches on cuda:  73%|███████▎  | 71/97 [00:06<00:02, 10.42batch/s]

Training batches on cuda:  75%|███████▌  | 73/97 [00:06<00:02, 10.55batch/s]

Training batches on cuda:  77%|███████▋  | 75/97 [00:07<00:02, 10.45batch/s]

Training batches on cuda:  79%|███████▉  | 77/97 [00:07<00:01, 10.54batch/s]

Training batches on cuda:  81%|████████▏ | 79/97 [00:07<00:01, 10.59batch/s]

Training batches on cuda:  84%|████████▎ | 81/97 [00:07<00:01, 10.50batch/s]

Training batches on cuda:  86%|████████▌ | 83/97 [00:07<00:01, 10.47batch/s]

Training batches on cuda:  88%|████████▊ | 85/97 [00:08<00:01, 10.44batch/s]

Training batches on cuda:  90%|████████▉ | 87/97 [00:08<00:00, 10.49batch/s]

Training batches on cuda:  92%|█████████▏| 89/97 [00:08<00:00, 10.51batch/s]

Training batches on cuda:  94%|█████████▍| 91/97 [00:08<00:00, 10.63batch/s]

Training batches on cuda:  96%|█████████▌| 93/97 [00:08<00:00, 10.68batch/s]

Training batches on cuda:  98%|█████████▊| 95/97 [00:09<00:00, 10.49batch/s]

Training batches on cuda: 100%|██████████| 97/97 [00:09<00:00, 12.07batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:37<01:05,  9.39s/epoch, loss=0.000625, prev_loss=0.000625]
Training epochs on cuda:  40%|████      | 4/10 [00:37<00:56,  9.40s/epoch, loss=0.000625, prev_loss=0.000625]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:12,  7.89batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:11,  8.50batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:10,  8.98batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:09,  9.40batch/s]

Training batches on cuda:   9%|▉         | 9/97 [00:00<00:09,  9.73batch/s]

Training batches on cuda:  11%|█▏        | 11/97 [00:01<00:08,  9.95batch/s]

Training batches on cuda:  13%|█▎        | 13/97 [00:01<00:08, 10.11batch/s]

Training batches on cuda:  15%|█▌        | 15/97 [00:01<00:08, 10.22batch/s]

Training batches on cuda:  18%|█▊        | 17/97 [00:01<00:07, 10.39batch/s]

Training batches on cuda:  20%|█▉        | 19/97 [00:01<00:07, 10.48batch/s]

Training batches on cuda:  22%|██▏       | 21/97 [00:02<00:07, 10.56batch/s]

Training batches on cuda:  24%|██▎       | 23/97 [00:02<00:06, 10.71batch/s]

Training batches on cuda:  26%|██▌       | 25/97 [00:02<00:06, 10.90batch/s]

Training batches on cuda:  28%|██▊       | 27/97 [00:02<00:06, 10.73batch/s]

Training batches on cuda:  30%|██▉       | 29/97 [00:02<00:06, 10.57batch/s]

Training batches on cuda:  32%|███▏      | 31/97 [00:02<00:06, 10.61batch/s]

Training batches on cuda:  34%|███▍      | 33/97 [00:03<00:06, 10.45batch/s]

Training batches on cuda:  36%|███▌      | 35/97 [00:03<00:05, 10.40batch/s]

Training batches on cuda:  38%|███▊      | 37/97 [00:03<00:05, 10.54batch/s]

Training batches on cuda:  40%|████      | 39/97 [00:03<00:05, 10.34batch/s]

Training batches on cuda:  42%|████▏     | 41/97 [00:03<00:05, 10.10batch/s]

Training batches on cuda:  44%|████▍     | 43/97 [00:04<00:05,  9.92batch/s]

Training batches on cuda:  46%|████▋     | 45/97 [00:04<00:05, 10.08batch/s]

Training batches on cuda:  48%|████▊     | 47/97 [00:04<00:05,  9.94batch/s]

Training batches on cuda:  51%|█████     | 49/97 [00:04<00:04, 10.00batch/s]

Training batches on cuda:  53%|█████▎    | 51/97 [00:04<00:04, 10.16batch/s]

Training batches on cuda:  55%|█████▍    | 53/97 [00:05<00:04, 10.40batch/s]

Training batches on cuda:  57%|█████▋    | 55/97 [00:05<00:04, 10.41batch/s]

Training batches on cuda:  59%|█████▉    | 57/97 [00:05<00:03, 10.07batch/s]

Training batches on cuda:  61%|██████    | 59/97 [00:05<00:03, 10.27batch/s]

Training batches on cuda:  63%|██████▎   | 61/97 [00:05<00:03, 10.34batch/s]

Training batches on cuda:  65%|██████▍   | 63/97 [00:06<00:03, 10.52batch/s]

Training batches on cuda:  67%|██████▋   | 65/97 [00:06<00:03, 10.54batch/s]

Training batches on cuda:  69%|██████▉   | 67/97 [00:06<00:02, 10.28batch/s]

Training batches on cuda:  71%|███████   | 69/97 [00:06<00:02, 10.29batch/s]

Training batches on cuda:  73%|███████▎  | 71/97 [00:06<00:02, 10.35batch/s]

Training batches on cuda:  75%|███████▌  | 73/97 [00:07<00:02, 10.34batch/s]

Training batches on cuda:  77%|███████▋  | 75/97 [00:07<00:02, 10.40batch/s]

Training batches on cuda:  79%|███████▉  | 77/97 [00:07<00:01, 10.53batch/s]

Training batches on cuda:  81%|████████▏ | 79/97 [00:07<00:01, 10.61batch/s]

Training batches on cuda:  84%|████████▎ | 81/97 [00:07<00:01, 10.65batch/s]

Training batches on cuda:  86%|████████▌ | 83/97 [00:07<00:01, 10.70batch/s]

Training batches on cuda:  88%|████████▊ | 85/97 [00:08<00:01, 10.74batch/s]

Training batches on cuda:  90%|████████▉ | 87/97 [00:08<00:00, 10.74batch/s]

Training batches on cuda:  92%|█████████▏| 89/97 [00:08<00:00, 10.72batch/s]

Training batches on cuda:  94%|█████████▍| 91/97 [00:08<00:00, 10.66batch/s]

Training batches on cuda:  96%|█████████▌| 93/97 [00:08<00:00, 10.60batch/s]

Training batches on cuda:  98%|█████████▊| 95/97 [00:09<00:00, 10.69batch/s]

Training batches on cuda: 100%|██████████| 97/97 [00:09<00:00, 12.27batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:47<00:56,  9.40s/epoch, loss=0.000625, prev_loss=0.000625]
Training epochs on cuda:  50%|█████     | 5/10 [00:47<00:47,  9.43s/epoch, loss=0.000625, prev_loss=0.000625]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:15,  6.02batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:13,  6.85batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:12,  7.64batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:10,  8.37batch/s]

Training batches on cuda:   9%|▉         | 9/97 [00:00<00:09,  8.93batch/s]

Training batches on cuda:  11%|█▏        | 11/97 [00:01<00:09,  9.40batch/s]

Training batches on cuda:  13%|█▎        | 13/97 [00:01<00:08,  9.57batch/s]

Training batches on cuda:  14%|█▍        | 14/97 [00:01<00:08,  9.51batch/s]

Training batches on cuda:  16%|█▋        | 16/97 [00:01<00:08,  9.72batch/s]

Training batches on cuda:  19%|█▊        | 18/97 [00:01<00:07,  9.95batch/s]

Training batches on cuda:  21%|██        | 20/97 [00:02<00:07,  9.94batch/s]

Training batches on cuda:  22%|██▏       | 21/97 [00:02<00:07,  9.85batch/s]

Training batches on cuda:  24%|██▎       | 23/97 [00:02<00:07,  9.99batch/s]

Training batches on cuda:  26%|██▌       | 25/97 [00:02<00:07, 10.02batch/s]

Training batches on cuda:  28%|██▊       | 27/97 [00:02<00:07,  9.85batch/s]

Training batches on cuda:  30%|██▉       | 29/97 [00:02<00:06,  9.98batch/s]

Training batches on cuda:  32%|███▏      | 31/97 [00:03<00:06, 10.09batch/s]

Training batches on cuda:  34%|███▍      | 33/97 [00:03<00:06, 10.08batch/s]

Training batches on cuda:  36%|███▌      | 35/97 [00:03<00:06, 10.07batch/s]

Training batches on cuda:  38%|███▊      | 37/97 [00:03<00:05, 10.30batch/s]

Training batches on cuda:  40%|████      | 39/97 [00:03<00:05, 10.42batch/s]

Training batches on cuda:  42%|████▏     | 41/97 [00:04<00:05, 10.40batch/s]

Training batches on cuda:  44%|████▍     | 43/97 [00:04<00:05, 10.02batch/s]

Training batches on cuda:  46%|████▋     | 45/97 [00:04<00:05,  9.89batch/s]

Training batches on cuda:  47%|████▋     | 46/97 [00:04<00:05,  9.70batch/s]

Training batches on cuda:  49%|████▉     | 48/97 [00:04<00:04,  9.91batch/s]

Training batches on cuda:  52%|█████▏    | 50/97 [00:04<00:04, 10.18batch/s]

Training batches on cuda:  54%|█████▎    | 52/97 [00:05<00:04, 10.17batch/s]

Training batches on cuda:  56%|█████▌    | 54/97 [00:05<00:04, 10.31batch/s]

Training batches on cuda:  58%|█████▊    | 56/97 [00:05<00:03, 10.32batch/s]

Training batches on cuda:  60%|█████▉    | 58/97 [00:05<00:03, 10.42batch/s]

Training batches on cuda:  62%|██████▏   | 60/97 [00:05<00:03, 10.35batch/s]

Training batches on cuda:  64%|██████▍   | 62/97 [00:06<00:03, 10.22batch/s]

Training batches on cuda:  66%|██████▌   | 64/97 [00:06<00:03, 10.24batch/s]

Training batches on cuda:  68%|██████▊   | 66/97 [00:06<00:03, 10.19batch/s]

Training batches on cuda:  70%|███████   | 68/97 [00:06<00:02, 10.39batch/s]

Training batches on cuda:  72%|███████▏  | 70/97 [00:06<00:02, 10.17batch/s]

Training batches on cuda:  74%|███████▍  | 72/97 [00:07<00:02, 10.20batch/s]

Training batches on cuda:  76%|███████▋  | 74/97 [00:07<00:02, 10.42batch/s]

Training batches on cuda:  78%|███████▊  | 76/97 [00:07<00:01, 10.54batch/s]

Training batches on cuda:  80%|████████  | 78/97 [00:07<00:01, 10.55batch/s]

Training batches on cuda:  82%|████████▏ | 80/97 [00:07<00:01, 10.33batch/s]

Training batches on cuda:  85%|████████▍ | 82/97 [00:08<00:01, 10.36batch/s]

Training batches on cuda:  87%|████████▋ | 84/97 [00:08<00:01, 10.41batch/s]

Training batches on cuda:  89%|████████▊ | 86/97 [00:08<00:01, 10.50batch/s]

Training batches on cuda:  91%|█████████ | 88/97 [00:08<00:00, 10.54batch/s]

Training batches on cuda:  93%|█████████▎| 90/97 [00:08<00:00, 10.52batch/s]

Training batches on cuda:  95%|█████████▍| 92/97 [00:09<00:00, 10.54batch/s]

Training batches on cuda:  97%|█████████▋| 94/97 [00:09<00:00, 10.59batch/s]

Training batches on cuda:  99%|█████████▉| 96/97 [00:09<00:00, 10.68batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:56<00:47,  9.43s/epoch, loss=0.000625, prev_loss=0.000625]
Training epochs on cuda:  60%|██████    | 6/10 [00:56<00:38,  9.52s/epoch, loss=0.000625, prev_loss=0.000625]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:16,  5.72batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:14,  6.52batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:12,  7.34batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:11,  8.02batch/s]

Training batches on cuda:   9%|▉         | 9/97 [00:00<00:10,  8.57batch/s]

Training batches on cuda:  11%|█▏        | 11/97 [00:01<00:09,  8.99batch/s]

Training batches on cuda:  12%|█▏        | 12/97 [00:01<00:10,  8.24batch/s]

Training batches on cuda:  14%|█▍        | 14/97 [00:01<00:09,  8.77batch/s]

Training batches on cuda:  16%|█▋        | 16/97 [00:01<00:08,  9.15batch/s]

Training batches on cuda:  19%|█▊        | 18/97 [00:01<00:08,  9.46batch/s]

Training batches on cuda:  21%|██        | 20/97 [00:02<00:08,  9.57batch/s]

Training batches on cuda:  23%|██▎       | 22/97 [00:02<00:07,  9.91batch/s]

Training batches on cuda:  25%|██▍       | 24/97 [00:02<00:07, 10.07batch/s]

Training batches on cuda:  27%|██▋       | 26/97 [00:02<00:06, 10.16batch/s]

Training batches on cuda:  29%|██▉       | 28/97 [00:02<00:06, 10.28batch/s]

Training batches on cuda:  31%|███       | 30/97 [00:03<00:06, 10.37batch/s]

Training batches on cuda:  33%|███▎      | 32/97 [00:03<00:06, 10.36batch/s]

Training batches on cuda:  35%|███▌      | 34/97 [00:03<00:06, 10.43batch/s]

Training batches on cuda:  37%|███▋      | 36/97 [00:03<00:06, 10.16batch/s]

Training batches on cuda:  39%|███▉      | 38/97 [00:03<00:05, 10.28batch/s]

Training batches on cuda:  41%|████      | 40/97 [00:04<00:05, 10.35batch/s]

Training batches on cuda:  43%|████▎     | 42/97 [00:04<00:05, 10.30batch/s]

Training batches on cuda:  45%|████▌     | 44/97 [00:04<00:05, 10.27batch/s]

Training batches on cuda:  47%|████▋     | 46/97 [00:04<00:04, 10.31batch/s]

Training batches on cuda:  49%|████▉     | 48/97 [00:04<00:04, 10.44batch/s]

Training batches on cuda:  52%|█████▏    | 50/97 [00:04<00:04, 10.42batch/s]

Training batches on cuda:  54%|█████▎    | 52/97 [00:05<00:04, 10.36batch/s]

Training batches on cuda:  56%|█████▌    | 54/97 [00:05<00:04, 10.32batch/s]

Training batches on cuda:  58%|█████▊    | 56/97 [00:05<00:03, 10.37batch/s]

Training batches on cuda:  60%|█████▉    | 58/97 [00:05<00:03, 10.14batch/s]

Training batches on cuda:  62%|██████▏   | 60/97 [00:05<00:03, 10.29batch/s]

Training batches on cuda:  64%|██████▍   | 62/97 [00:06<00:03, 10.16batch/s]

Training batches on cuda:  66%|██████▌   | 64/97 [00:06<00:03, 10.24batch/s]

Training batches on cuda:  68%|██████▊   | 66/97 [00:06<00:03, 10.28batch/s]

Training batches on cuda:  70%|███████   | 68/97 [00:06<00:02, 10.24batch/s]

Training batches on cuda:  72%|███████▏  | 70/97 [00:06<00:02, 10.29batch/s]

Training batches on cuda:  74%|███████▍  | 72/97 [00:07<00:02, 10.21batch/s]

Training batches on cuda:  76%|███████▋  | 74/97 [00:07<00:02, 10.28batch/s]

Training batches on cuda:  78%|███████▊  | 76/97 [00:07<00:02, 10.39batch/s]

Training batches on cuda:  80%|████████  | 78/97 [00:07<00:01, 10.48batch/s]

Training batches on cuda:  82%|████████▏ | 80/97 [00:07<00:01, 10.45batch/s]

Training batches on cuda:  85%|████████▍ | 82/97 [00:08<00:01, 10.47batch/s]

Training batches on cuda:  87%|████████▋ | 84/97 [00:08<00:01, 10.51batch/s]

Training batches on cuda:  89%|████████▊ | 86/97 [00:08<00:01, 10.54batch/s]

Training batches on cuda:  91%|█████████ | 88/97 [00:08<00:00, 10.58batch/s]

Training batches on cuda:  93%|█████████▎| 90/97 [00:08<00:00, 10.46batch/s]

Training batches on cuda:  95%|█████████▍| 92/97 [00:09<00:00, 10.06batch/s]

Training batches on cuda:  97%|█████████▋| 94/97 [00:09<00:00, 10.00batch/s]

Training batches on cuda:  99%|█████████▉| 96/97 [00:09<00:00, 10.20batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [01:06<00:38,  9.52s/epoch, loss=0.000625, prev_loss=0.000625]
Training epochs on cuda:  70%|███████   | 7/10 [01:06<00:28,  9.60s/epoch, loss=0.000625, prev_loss=0.000625]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:12,  7.77batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:11,  8.33batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:10,  9.00batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:09,  9.39batch/s]

Training batches on cuda:   9%|▉         | 9/97 [00:00<00:09,  9.64batch/s]

Training batches on cuda:  11%|█▏        | 11/97 [00:01<00:08,  9.76batch/s]

Training batches on cuda:  13%|█▎        | 13/97 [00:01<00:08,  9.97batch/s]

Training batches on cuda:  15%|█▌        | 15/97 [00:01<00:07, 10.26batch/s]

Training batches on cuda:  18%|█▊        | 17/97 [00:01<00:07, 10.42batch/s]

Training batches on cuda:  20%|█▉        | 19/97 [00:01<00:07, 10.47batch/s]

Training batches on cuda:  22%|██▏       | 21/97 [00:02<00:07, 10.34batch/s]

Training batches on cuda:  24%|██▎       | 23/97 [00:02<00:07, 10.47batch/s]

Training batches on cuda:  26%|██▌       | 25/97 [00:02<00:06, 10.58batch/s]

Training batches on cuda:  28%|██▊       | 27/97 [00:02<00:06, 10.67batch/s]

Training batches on cuda:  30%|██▉       | 29/97 [00:02<00:06, 10.72batch/s]

Training batches on cuda:  32%|███▏      | 31/97 [00:02<00:06, 10.77batch/s]

Training batches on cuda:  34%|███▍      | 33/97 [00:03<00:05, 10.80batch/s]

Training batches on cuda:  36%|███▌      | 35/97 [00:03<00:05, 10.82batch/s]

Training batches on cuda:  38%|███▊      | 37/97 [00:03<00:05, 10.83batch/s]

Training batches on cuda:  40%|████      | 39/97 [00:03<00:05, 10.82batch/s]

Training batches on cuda:  42%|████▏     | 41/97 [00:03<00:05, 10.77batch/s]

Training batches on cuda:  44%|████▍     | 43/97 [00:04<00:05, 10.71batch/s]

Training batches on cuda:  46%|████▋     | 45/97 [00:04<00:05,  9.99batch/s]

Training batches on cuda:  48%|████▊     | 47/97 [00:04<00:04, 10.19batch/s]

Training batches on cuda:  51%|█████     | 49/97 [00:04<00:04, 10.30batch/s]

Training batches on cuda:  53%|█████▎    | 51/97 [00:04<00:04, 10.46batch/s]

Training batches on cuda:  55%|█████▍    | 53/97 [00:05<00:04, 10.58batch/s]

Training batches on cuda:  57%|█████▋    | 55/97 [00:05<00:03, 10.63batch/s]

Training batches on cuda:  59%|█████▉    | 57/97 [00:05<00:03, 10.67batch/s]

Training batches on cuda:  61%|██████    | 59/97 [00:05<00:03, 10.78batch/s]

Training batches on cuda:  63%|██████▎   | 61/97 [00:05<00:03, 10.82batch/s]

Training batches on cuda:  65%|██████▍   | 63/97 [00:05<00:03, 10.84batch/s]

Training batches on cuda:  67%|██████▋   | 65/97 [00:06<00:02, 10.82batch/s]

Training batches on cuda:  69%|██████▉   | 67/97 [00:06<00:02, 10.84batch/s]

Training batches on cuda:  71%|███████   | 69/97 [00:06<00:02, 10.63batch/s]

Training batches on cuda:  73%|███████▎  | 71/97 [00:06<00:02, 10.71batch/s]

Training batches on cuda:  75%|███████▌  | 73/97 [00:06<00:02, 10.88batch/s]

Training batches on cuda:  77%|███████▋  | 75/97 [00:07<00:01, 11.05batch/s]

Training batches on cuda:  79%|███████▉  | 77/97 [00:07<00:01, 11.09batch/s]

Training batches on cuda:  81%|████████▏ | 79/97 [00:07<00:01, 11.01batch/s]

Training batches on cuda:  84%|████████▎ | 81/97 [00:07<00:01, 10.99batch/s]

Training batches on cuda:  86%|████████▌ | 83/97 [00:07<00:01, 10.69batch/s]

Training batches on cuda:  88%|████████▊ | 85/97 [00:08<00:01, 10.65batch/s]

Training batches on cuda:  90%|████████▉ | 87/97 [00:08<00:00, 10.68batch/s]

Training batches on cuda:  92%|█████████▏| 89/97 [00:08<00:00, 10.60batch/s]

Training batches on cuda:  94%|█████████▍| 91/97 [00:08<00:00, 10.66batch/s]

Training batches on cuda:  96%|█████████▌| 93/97 [00:08<00:00, 10.67batch/s]

Training batches on cuda:  98%|█████████▊| 95/97 [00:08<00:00, 10.47batch/s]

Training batches on cuda: 100%|██████████| 97/97 [00:09<00:00, 12.03batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [01:15<00:28,  9.60s/epoch, loss=0.000624, prev_loss=0.000625]
Training epochs on cuda:  80%|████████  | 8/10 [01:15<00:19,  9.52s/epoch, loss=0.000624, prev_loss=0.000625]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:12,  7.74batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:11,  8.40batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:10,  8.95batch/s]

Training batches on cuda:   6%|▌         | 6/97 [00:00<00:09,  9.23batch/s]

Training batches on cuda:   8%|▊         | 8/97 [00:00<00:09,  9.60batch/s]

Training batches on cuda:  10%|█         | 10/97 [00:00<00:08,  9.92batch/s]

Training batches on cuda:  12%|█▏        | 12/97 [00:01<00:08, 10.11batch/s]

Training batches on cuda:  14%|█▍        | 14/97 [00:01<00:08, 10.35batch/s]

Training batches on cuda:  16%|█▋        | 16/97 [00:01<00:07, 10.51batch/s]

Training batches on cuda:  19%|█▊        | 18/97 [00:01<00:07, 10.58batch/s]

Training batches on cuda:  21%|██        | 20/97 [00:01<00:07, 10.53batch/s]

Training batches on cuda:  23%|██▎       | 22/97 [00:02<00:07, 10.58batch/s]

Training batches on cuda:  25%|██▍       | 24/97 [00:02<00:07, 10.37batch/s]

Training batches on cuda:  27%|██▋       | 26/97 [00:02<00:06, 10.34batch/s]

Training batches on cuda:  29%|██▉       | 28/97 [00:02<00:06, 10.45batch/s]

Training batches on cuda:  31%|███       | 30/97 [00:02<00:06, 10.48batch/s]

Training batches on cuda:  33%|███▎      | 32/97 [00:03<00:06, 10.48batch/s]

Training batches on cuda:  35%|███▌      | 34/97 [00:03<00:05, 10.53batch/s]

Training batches on cuda:  37%|███▋      | 36/97 [00:03<00:05, 10.59batch/s]

Training batches on cuda:  39%|███▉      | 38/97 [00:03<00:05, 10.62batch/s]

Training batches on cuda:  41%|████      | 40/97 [00:03<00:05, 10.68batch/s]

Training batches on cuda:  43%|████▎     | 42/97 [00:04<00:05, 10.61batch/s]

Training batches on cuda:  45%|████▌     | 44/97 [00:04<00:04, 10.62batch/s]

Training batches on cuda:  47%|████▋     | 46/97 [00:04<00:04, 10.59batch/s]

Training batches on cuda:  49%|████▉     | 48/97 [00:04<00:04, 10.46batch/s]

Training batches on cuda:  52%|█████▏    | 50/97 [00:04<00:04, 10.59batch/s]

Training batches on cuda:  54%|█████▎    | 52/97 [00:04<00:04,  9.87batch/s]

Training batches on cuda:  56%|█████▌    | 54/97 [00:05<00:04, 10.13batch/s]

Training batches on cuda:  58%|█████▊    | 56/97 [00:05<00:03, 10.38batch/s]

Training batches on cuda:  60%|█████▉    | 58/97 [00:05<00:03, 10.53batch/s]

Training batches on cuda:  62%|██████▏   | 60/97 [00:05<00:03, 10.65batch/s]

Training batches on cuda:  64%|██████▍   | 62/97 [00:05<00:03, 10.76batch/s]

Training batches on cuda:  66%|██████▌   | 64/97 [00:06<00:03, 10.75batch/s]

Training batches on cuda:  68%|██████▊   | 66/97 [00:06<00:02, 10.76batch/s]

Training batches on cuda:  70%|███████   | 68/97 [00:06<00:02, 10.68batch/s]

Training batches on cuda:  72%|███████▏  | 70/97 [00:06<00:02, 10.56batch/s]

Training batches on cuda:  74%|███████▍  | 72/97 [00:06<00:02, 10.40batch/s]

Training batches on cuda:  76%|███████▋  | 74/97 [00:07<00:02, 10.16batch/s]

Training batches on cuda:  78%|███████▊  | 76/97 [00:07<00:02,  9.85batch/s]

Training batches on cuda:  80%|████████  | 78/97 [00:07<00:01, 10.09batch/s]

Training batches on cuda:  82%|████████▏ | 80/97 [00:07<00:01, 10.30batch/s]

Training batches on cuda:  85%|████████▍ | 82/97 [00:07<00:01, 10.32batch/s]

Training batches on cuda:  87%|████████▋ | 84/97 [00:08<00:01, 10.36batch/s]

Training batches on cuda:  89%|████████▊ | 86/97 [00:08<00:01, 10.41batch/s]

Training batches on cuda:  91%|█████████ | 88/97 [00:08<00:00, 10.31batch/s]

Training batches on cuda:  93%|█████████▎| 90/97 [00:08<00:00, 10.40batch/s]

Training batches on cuda:  95%|█████████▍| 92/97 [00:08<00:00, 10.48batch/s]

Training batches on cuda:  97%|█████████▋| 94/97 [00:09<00:00, 10.48batch/s]

Training batches on cuda:  99%|█████████▉| 96/97 [00:09<00:00, 10.38batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [01:25<00:19,  9.52s/epoch, loss=0.000621, prev_loss=0.000624]
Training epochs on cuda:  90%|█████████ | 9/10 [01:25<00:09,  9.51s/epoch, loss=0.000621, prev_loss=0.000624]

Training batches on cuda:   0%|          | 0/97 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/97 [00:00<00:15,  6.08batch/s]

Training batches on cuda:   3%|▎         | 3/97 [00:00<00:13,  6.91batch/s]

Training batches on cuda:   5%|▌         | 5/97 [00:00<00:11,  7.68batch/s]

Training batches on cuda:   7%|▋         | 7/97 [00:00<00:10,  8.33batch/s]

Training batches on cuda:   8%|▊         | 8/97 [00:00<00:10,  8.64batch/s]

Training batches on cuda:  10%|█         | 10/97 [00:01<00:09,  9.17batch/s]

Training batches on cuda:  12%|█▏        | 12/97 [00:01<00:08,  9.49batch/s]

Training batches on cuda:  14%|█▍        | 14/97 [00:01<00:08,  9.74batch/s]

Training batches on cuda:  16%|█▋        | 16/97 [00:01<00:08, 10.00batch/s]

Training batches on cuda:  19%|█▊        | 18/97 [00:01<00:07, 10.23batch/s]

Training batches on cuda:  21%|██        | 20/97 [00:01<00:07, 10.40batch/s]

Training batches on cuda:  23%|██▎       | 22/97 [00:02<00:07, 10.49batch/s]

Training batches on cuda:  25%|██▍       | 24/97 [00:02<00:06, 10.58batch/s]

Training batches on cuda:  27%|██▋       | 26/97 [00:02<00:06, 10.59batch/s]

Training batches on cuda:  29%|██▉       | 28/97 [00:02<00:06, 10.62batch/s]

Training batches on cuda:  31%|███       | 30/97 [00:02<00:06, 10.74batch/s]

Training batches on cuda:  33%|███▎      | 32/97 [00:03<00:06, 10.82batch/s]

Training batches on cuda:  35%|███▌      | 34/97 [00:03<00:05, 10.87batch/s]

Training batches on cuda:  37%|███▋      | 36/97 [00:03<00:05, 10.76batch/s]

Training batches on cuda:  39%|███▉      | 38/97 [00:03<00:05, 10.86batch/s]

Training batches on cuda:  41%|████      | 40/97 [00:03<00:05, 10.75batch/s]

Training batches on cuda:  43%|████▎     | 42/97 [00:04<00:05, 10.71batch/s]

Training batches on cuda:  45%|████▌     | 44/97 [00:04<00:04, 10.80batch/s]

Training batches on cuda:  47%|████▋     | 46/97 [00:04<00:04, 10.65batch/s]

Training batches on cuda:  49%|████▉     | 48/97 [00:04<00:04, 10.25batch/s]

Training batches on cuda:  52%|█████▏    | 50/97 [00:04<00:04, 10.35batch/s]

Training batches on cuda:  54%|█████▎    | 52/97 [00:04<00:04, 10.30batch/s]

Training batches on cuda:  56%|█████▌    | 54/97 [00:05<00:04, 10.35batch/s]

Training batches on cuda:  58%|█████▊    | 56/97 [00:05<00:03, 10.35batch/s]

Training batches on cuda:  60%|█████▉    | 58/97 [00:05<00:03, 10.26batch/s]

Training batches on cuda:  62%|██████▏   | 60/97 [00:05<00:03, 10.39batch/s]

Training batches on cuda:  64%|██████▍   | 62/97 [00:05<00:03, 10.42batch/s]

Training batches on cuda:  66%|██████▌   | 64/97 [00:06<00:03, 10.50batch/s]

Training batches on cuda:  68%|██████▊   | 66/97 [00:06<00:02, 10.40batch/s]

Training batches on cuda:  70%|███████   | 68/97 [00:06<00:02, 10.54batch/s]

Training batches on cuda:  72%|███████▏  | 70/97 [00:06<00:02, 10.62batch/s]

Training batches on cuda:  74%|███████▍  | 72/97 [00:06<00:02, 10.77batch/s]

Training batches on cuda:  76%|███████▋  | 74/97 [00:07<00:02, 10.76batch/s]

Training batches on cuda:  78%|███████▊  | 76/97 [00:07<00:01, 10.80batch/s]

Training batches on cuda:  80%|████████  | 78/97 [00:07<00:01, 10.78batch/s]

Training batches on cuda:  82%|████████▏ | 80/97 [00:07<00:01, 10.67batch/s]

Training batches on cuda:  85%|████████▍ | 82/97 [00:07<00:01, 10.55batch/s]

Training batches on cuda:  87%|████████▋ | 84/97 [00:08<00:01, 10.62batch/s]

Training batches on cuda:  89%|████████▊ | 86/97 [00:08<00:01, 10.64batch/s]

Training batches on cuda:  91%|█████████ | 88/97 [00:08<00:00, 10.64batch/s]

Training batches on cuda:  93%|█████████▎| 90/97 [00:08<00:00, 10.59batch/s]

Training batches on cuda:  95%|█████████▍| 92/97 [00:08<00:00, 10.66batch/s]

Training batches on cuda:  97%|█████████▋| 94/97 [00:08<00:00, 10.62batch/s]

Training batches on cuda:  99%|█████████▉| 96/97 [00:09<00:00, 10.61batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [01:34<00:09,  9.51s/epoch, loss=0.000619, prev_loss=0.000621]2021-09-16:18:33:16,757 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:33:17,867 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:33:32,945 INFO     [evaluator.py:657] Evaluation took 15.08s seconds
2021-09-16:18:33:33,84 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 14.41s/epoch, loss=0.000619, prev_loss=0.000621]
Training epochs on cuda: 100%|██████████| 10/10 [01:51<00:00, 11.13s/epoch, loss=0.000619, prev_loss=0.000621]
2021-09-16:18:33:33,85 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp1fcdmkuq'
2021-09-16:18:33:33,154 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1fcdmkuq' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.15ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.27ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.36ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.49ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.52ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.55ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.60ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.60ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.61ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.61ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.61ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.62ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.62ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.62ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.61ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:33:48,129 INFO     [evaluator.py:657] Evaluation took 14.96s seconds
[I 2021-09-16 18:33:48,135] Trial 49 finished with value: 0.10791390599065698 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.10375800638178034, 'optimizer.lr': 0.013238223606180492, 'negative_sampler.num_negs_per_pos': 100, 'training.batch_size': 3232}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:33:48,157 WARNING  [api.py:823] No random seed is specified. Setting to 1713519382.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:16,  5.19batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:13,  6.34batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:10,  7.55batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:08,  8.81batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:07,  9.94batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 10.90batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:01<00:06, 11.49batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 12.14batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 12.66batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:05, 13.06batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:04, 13.25batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 13.26batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 13.01batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:02<00:04, 13.22batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.18batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:04, 13.45batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:03, 13.52batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 13.60batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 13.72batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:02<00:03, 13.74batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 13.79batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.77batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:02, 13.85batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.91batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 13.90batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.93batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:03<00:02, 13.76batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.71batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 12.87batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:02, 13.11batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 13.31batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 13.25batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.48batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:05<00:01, 13.23batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.41batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 12.85batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:01, 12.47batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 12.32batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 12.74batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 13.14batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:06<00:00, 13.28batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 13.26batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 13.15batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:06<?, ?epoch/s, loss=0.000607, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:06<01:00,  6.74s/epoch, loss=0.000607, prev_loss=nan]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:15,  5.35batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:12,  6.54batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:10,  7.77batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:08,  8.91batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:07,  9.99batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 10.88batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:01<00:06, 11.60batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 12.09batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 12.53batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:05, 12.83batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:04, 13.11batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 13.30batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 13.35batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:02<00:04, 13.47batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.52batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:04, 13.30batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:04, 12.44batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 12.76batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 13.01batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:03<00:03, 13.24batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 13.42batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.60batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:02, 13.76batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.68batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 13.70batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.67batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:04<00:02, 13.45batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.46batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.41batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:02, 13.35batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 13.26batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 13.28batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.38batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:05<00:01, 13.31batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.39batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 13.40batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 13.51batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 13.41batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 13.44batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 13.90batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:06<00:00, 14.29batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 14.55batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 14.77batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:13<01:00,  6.74s/epoch, loss=0.000594, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:53,  6.70s/epoch, loss=0.000594, prev_loss=nan]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:11,  7.69batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:09,  8.93batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:08,  9.84batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:07, 10.88batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:06, 11.73batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 12.20batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:00<00:05, 12.75batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 13.18batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 13.42batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:04, 13.66batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:04, 13.63batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 13.88batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 14.03batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:01<00:04, 14.10batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.81batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:03, 13.77batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:03, 13.65batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 13.72batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 13.39batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:02<00:03, 13.64batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 13.90batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 14.07batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:02, 14.09batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.92batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 13.88batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.91batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:03<00:02, 13.74batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.79batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.88batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:01, 13.91batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 14.05batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 13.57batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.36batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:04<00:01, 13.50batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.42batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 13.51batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 13.65batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 13.51batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 13.52batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 13.67batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:05<00:00, 13.03batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 13.32batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 13.10batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:20<00:53,  6.70s/epoch, loss=0.000594, prev_loss=0.000594]
Training epochs on cuda:  30%|███       | 3/10 [00:20<00:46,  6.69s/epoch, loss=0.000594, prev_loss=0.000594]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:12,  7.01batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:10,  8.19batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:08,  9.30batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:07, 10.35batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:06, 11.28batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 11.89batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:01<00:05, 12.26batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 12.53batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 12.90batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:05, 13.03batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:04, 13.12batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 13.01batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 12.92batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:02<00:04, 12.83batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 12.82batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:04, 12.95batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:04, 12.99batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 13.04batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 13.05batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:03<00:03, 12.83batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 12.97batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.05batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:03, 12.96batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:03, 12.99batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 12.96batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 12.88batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:04<00:02, 13.09batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.50batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.79batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:01, 14.08batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 14.12batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 14.04batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.99batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:05<00:01, 14.05batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 14.17batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 14.19batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 14.14batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 14.13batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 14.14batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 13.89batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:06<00:00, 13.77batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 13.85batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 13.82batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:26<00:46,  6.69s/epoch, loss=0.000595, prev_loss=0.000594]
Training epochs on cuda:  40%|████      | 4/10 [00:26<00:40,  6.69s/epoch, loss=0.000595, prev_loss=0.000594]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:10,  7.84batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:09,  9.01batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:08,  9.97batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:07, 10.95batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:06, 11.74batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 12.34batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:00<00:05, 12.82batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 13.20batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 13.49batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:04, 13.67batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:04, 13.79batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 13.93batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 13.96batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:01<00:04, 13.97batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.95batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:03, 13.90batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:03, 13.89batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 13.25batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 13.18batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:02<00:03, 13.20batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 13.14batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.16batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:03, 13.37batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.31batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 13.55batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.70batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:03<00:02, 13.79batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.83batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.83batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:01, 13.91batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 13.87batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 13.70batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.72batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:04<00:01, 13.76batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.78batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 13.69batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 13.63batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 13.75batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 13.85batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 14.01batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:05<00:00, 14.08batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 14.06batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 14.03batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:33<00:40,  6.69s/epoch, loss=0.000595, prev_loss=0.000595]
Training epochs on cuda:  50%|█████     | 5/10 [00:33<00:33,  6.66s/epoch, loss=0.000595, prev_loss=0.000595]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:11,  7.72batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:09,  8.82batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:08,  9.89batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:07, 10.73batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:06, 11.35batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 11.95batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:01<00:05, 12.49batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 12.52batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 12.82batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:05, 12.89batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:04, 13.19batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 13.43batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 13.58batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:02<00:04, 13.61batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.68batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:04, 13.49batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:04, 13.15batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 13.23batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 13.58batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:02<00:03, 13.76batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 13.85batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.89batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:02, 13.91batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.99batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 14.04batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.98batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:03<00:02, 13.86batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.91batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.85batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:01, 13.84batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 13.74batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 13.11batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.48batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:04<00:01, 13.75batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.96batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 14.03batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 13.80batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 13.71batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 13.70batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 13.54batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:05<00:00, 13.64batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 13.93batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 13.99batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:39<00:33,  6.66s/epoch, loss=0.000595, prev_loss=0.000595]
Training epochs on cuda:  60%|██████    | 6/10 [00:39<00:26,  6.65s/epoch, loss=0.000595, prev_loss=0.000595]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:12,  6.89batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:10,  8.17batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:08,  9.34batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:07, 10.11batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:07, 10.82batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 11.58batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:01<00:05, 12.24batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 12.82batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 12.94batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:05, 13.25batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:05, 12.94batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 12.81batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 13.11batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:02<00:04, 13.19batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.25batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:04, 13.45batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:04, 12.63batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:04, 12.72batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 12.88batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:03<00:03, 13.10batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 13.40batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.54batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:03, 13.63batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.49batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 13.45batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.49batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:04<00:02, 13.59batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.67batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.65batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:01, 13.57batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 13.57batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 13.68batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.41batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:05<00:01, 13.08batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.12batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 13.11batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 13.16batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 13.07batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 13.06batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 13.15batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:06<00:00, 13.06batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 13.17batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 13.07batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [00:46<00:26,  6.65s/epoch, loss=0.000595, prev_loss=0.000595]
Training epochs on cuda:  70%|███████   | 7/10 [00:46<00:20,  6.70s/epoch, loss=0.000595, prev_loss=0.000595]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:11,  7.37batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:09,  8.43batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:08,  9.53batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:07, 10.23batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:07, 10.74batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 11.38batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:01<00:06, 11.95batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 12.23batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 12.15batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:05, 12.66batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:05, 12.81batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 12.91batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 13.27batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:02<00:04, 13.38batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.46batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:04, 13.31batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:03, 13.25batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 13.38batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 13.46batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:03<00:03, 13.20batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 12.64batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.05batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:03, 13.28batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.47batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 13.37batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.16batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:04<00:02, 13.12batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.30batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.12batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:02, 13.25batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 13.42batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 13.34batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.56batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:05<00:01, 13.27batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.27batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 13.40batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 13.16batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 13.44batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 13.56batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:06<00:00, 13.58batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:06<00:00, 13.50batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 13.82batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 13.75batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [00:53<00:20,  6.70s/epoch, loss=0.000595, prev_loss=0.000595]
Training epochs on cuda:  80%|████████  | 8/10 [00:53<00:13,  6.73s/epoch, loss=0.000595, prev_loss=0.000595]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:12,  6.57batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:10,  7.82batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:08,  9.01batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:07, 10.15batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:06, 11.20batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 12.05batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:01<00:06, 11.98batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 12.43batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 12.84batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:05, 13.06batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:04, 13.09batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 13.11batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 13.17batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:02<00:04, 13.18batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.46batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:04, 13.60batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:03, 13.66batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 13.86batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 13.69batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:02<00:03, 13.78batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 13.64batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.61batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:03, 13.47batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.46batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 13.42batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.40batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:03<00:02, 13.40batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.51batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.57batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:01, 13.55batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 13.69batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 13.78batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 13.75batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:04<00:01, 13.73batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.60batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 13.44batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 13.17batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 13.07batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 13.29batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 13.37batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:06<00:00, 13.64batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 13.76batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 13.54batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [01:00<00:13,  6.73s/epoch, loss=0.000595, prev_loss=0.000595]
Training epochs on cuda:  90%|█████████ | 9/10 [01:00<00:06,  6.72s/epoch, loss=0.000595, prev_loss=0.000595]

Training batches on cuda:   0%|          | 0/86 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/86 [00:00<00:14,  6.01batch/s]

Training batches on cuda:   3%|▎         | 3/86 [00:00<00:11,  7.19batch/s]

Training batches on cuda:   6%|▌         | 5/86 [00:00<00:09,  8.44batch/s]

Training batches on cuda:   8%|▊         | 7/86 [00:00<00:08,  9.60batch/s]

Training batches on cuda:  10%|█         | 9/86 [00:00<00:07, 10.45batch/s]

Training batches on cuda:  13%|█▎        | 11/86 [00:00<00:06, 11.28batch/s]

Training batches on cuda:  15%|█▌        | 13/86 [00:01<00:06, 12.04batch/s]

Training batches on cuda:  17%|█▋        | 15/86 [00:01<00:05, 12.53batch/s]

Training batches on cuda:  20%|█▉        | 17/86 [00:01<00:05, 12.81batch/s]

Training batches on cuda:  22%|██▏       | 19/86 [00:01<00:05, 13.18batch/s]

Training batches on cuda:  24%|██▍       | 21/86 [00:01<00:04, 13.40batch/s]

Training batches on cuda:  27%|██▋       | 23/86 [00:01<00:04, 13.52batch/s]

Training batches on cuda:  29%|██▉       | 25/86 [00:01<00:04, 13.69batch/s]

Training batches on cuda:  31%|███▏      | 27/86 [00:02<00:04, 13.71batch/s]

Training batches on cuda:  34%|███▎      | 29/86 [00:02<00:04, 13.82batch/s]

Training batches on cuda:  36%|███▌      | 31/86 [00:02<00:03, 13.92batch/s]

Training batches on cuda:  38%|███▊      | 33/86 [00:02<00:03, 14.02batch/s]

Training batches on cuda:  41%|████      | 35/86 [00:02<00:03, 14.00batch/s]

Training batches on cuda:  43%|████▎     | 37/86 [00:02<00:03, 14.02batch/s]

Training batches on cuda:  45%|████▌     | 39/86 [00:02<00:03, 13.95batch/s]

Training batches on cuda:  48%|████▊     | 41/86 [00:03<00:03, 13.90batch/s]

Training batches on cuda:  50%|█████     | 43/86 [00:03<00:03, 13.83batch/s]

Training batches on cuda:  52%|█████▏    | 45/86 [00:03<00:02, 13.86batch/s]

Training batches on cuda:  55%|█████▍    | 47/86 [00:03<00:02, 13.90batch/s]

Training batches on cuda:  57%|█████▋    | 49/86 [00:03<00:02, 13.73batch/s]

Training batches on cuda:  59%|█████▉    | 51/86 [00:03<00:02, 13.81batch/s]

Training batches on cuda:  62%|██████▏   | 53/86 [00:03<00:02, 13.93batch/s]

Training batches on cuda:  64%|██████▍   | 55/86 [00:04<00:02, 13.92batch/s]

Training batches on cuda:  66%|██████▋   | 57/86 [00:04<00:02, 13.71batch/s]

Training batches on cuda:  69%|██████▊   | 59/86 [00:04<00:02, 13.42batch/s]

Training batches on cuda:  71%|███████   | 61/86 [00:04<00:01, 13.31batch/s]

Training batches on cuda:  73%|███████▎  | 63/86 [00:04<00:01, 12.91batch/s]

Training batches on cuda:  76%|███████▌  | 65/86 [00:04<00:01, 12.88batch/s]

Training batches on cuda:  78%|███████▊  | 67/86 [00:04<00:01, 12.91batch/s]

Training batches on cuda:  80%|████████  | 69/86 [00:05<00:01, 13.24batch/s]

Training batches on cuda:  83%|████████▎ | 71/86 [00:05<00:01, 13.41batch/s]

Training batches on cuda:  85%|████████▍ | 73/86 [00:05<00:00, 13.64batch/s]

Training batches on cuda:  87%|████████▋ | 75/86 [00:05<00:00, 13.80batch/s]

Training batches on cuda:  90%|████████▉ | 77/86 [00:05<00:00, 13.66batch/s]

Training batches on cuda:  92%|█████████▏| 79/86 [00:05<00:00, 13.69batch/s]

Training batches on cuda:  94%|█████████▍| 81/86 [00:06<00:00, 13.61batch/s]

Training batches on cuda:  97%|█████████▋| 83/86 [00:06<00:00, 13.79batch/s]

Training batches on cuda:  99%|█████████▉| 85/86 [00:06<00:00, 13.95batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [01:06<00:06,  6.72s/epoch, loss=0.000595, prev_loss=0.000595]2021-09-16:18:34:55,636 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:34:56,685 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:35:11,758 INFO     [evaluator.py:657] Evaluation took 15.07s seconds
2021-09-16:18:35:11,916 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 11.56s/epoch, loss=0.000595, prev_loss=0.000595]
Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00,  8.31s/epoch, loss=0.000595, prev_loss=0.000595]
2021-09-16:18:35:11,917 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpmyk26b1t'
2021-09-16:18:35:12,13 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpmyk26b1t' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.37ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:14, 2.46ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.51ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.54ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.57ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.61ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.62ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.62ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.62ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:35:26,968 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
[I 2021-09-16 18:35:26,974] Trial 50 finished with value: -0.0026091001974926087 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.3327552025424868, 'optimizer.lr': 0.069018207443836, 'negative_sampler.num_negs_per_pos': 31, 'training.batch_size': 3632}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:35:26,997 WARNING  [api.py:823] No random seed is specified. Setting to 852334525.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:17,  6.16batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:14,  7.26batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:12,  8.34batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:11,  9.27batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:10, 10.07batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:09, 10.80batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:09, 10.61batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:08, 11.15batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:08, 11.57batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 11.95batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:07, 12.19batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:07, 12.42batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:02<00:06, 12.72batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 12.88batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 12.99batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:06, 13.07batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:05, 13.11batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 13.24batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 13.31batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 12.91batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 12.96batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 13.05batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:04, 13.07batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 13.24batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 13.16batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:04<00:04, 13.25batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 13.31batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 13.30batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:04, 13.13batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 13.03batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 13.21batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:03, 13.27batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:05<00:03, 13.32batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 13.38batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 13.46batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:02, 13.40batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 13.49batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 13.42batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:06<00:02, 13.55batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 12.73batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 13.04batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 13.15batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 13.32batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 13.33batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:06<00:01, 13.30batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:07<00:01, 13.18batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 13.22batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 13.40batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:00, 13.25batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 13.28batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 13.18batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:07<00:00, 13.24batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 13.26batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 13.38batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.42batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:08<?, ?epoch/s, loss=0.000465, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:08<01:19,  8.80s/epoch, loss=0.000465, prev_loss=nan]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:17,  6.07batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:14,  7.24batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:12,  8.40batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:10,  9.45batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:09, 10.44batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:08, 11.16batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 11.70batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:07, 12.27batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 12.54batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 12.71batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:06, 12.92batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 13.08batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:06, 13.09batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 13.21batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 13.27batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:05, 13.34batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:05, 13.29batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 13.35batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 13.30batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 13.35batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 13.33batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 13.32batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:04, 13.26batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 13.26batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 13.33batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:04, 13.38batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 13.32batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 13.25batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:04, 13.20batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 13.04batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 13.08batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:03, 13.14batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:03, 13.25batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 13.30batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 13.19batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:02, 13.22batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 13.20batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 13.16batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:05<00:02, 13.28batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 13.15batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 12.89batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 12.90batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 12.87batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 12.52batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:06<00:01, 12.83batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:06<00:01, 13.02batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 12.89batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 12.95batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:01, 12.78batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 12.93batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 12.97batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:07<00:00, 12.91batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 12.90batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 13.01batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.02batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:17<01:19,  8.80s/epoch, loss=0.000458, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:17<01:10,  8.77s/epoch, loss=0.000458, prev_loss=nan]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:18,  5.89batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:15,  7.03batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:12,  8.13batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:11,  9.12batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:10, 10.09batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:09, 10.86batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 11.57batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:07, 11.88batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 12.29batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 12.58batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:06, 12.88batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 12.84batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:06, 13.06batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 13.11batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 13.22batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:06, 13.02batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:06, 12.61batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 12.74batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 12.80batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 13.03batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 12.94batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 13.07batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:04, 13.13batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 13.19batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 13.21batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:04, 13.24batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 13.20batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 13.27batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:03, 13.27batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 13.28batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 13.16batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:03, 13.00batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:05<00:03, 13.11batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 13.20batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 13.36batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:02, 13.50batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 13.21batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 13.18batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:05<00:02, 13.32batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 13.12batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 12.91batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 12.82batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 12.75batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 12.94batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:06<00:01, 12.87batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:07<00:01, 13.03batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 12.89batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 12.87batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:01, 12.95batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 12.97batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 12.95batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:07<00:00, 12.96batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 13.06batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 13.08batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.10batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:26<01:10,  8.77s/epoch, loss=0.000442, prev_loss=0.000458]
Training epochs on cuda:  30%|███       | 3/10 [00:26<01:01,  8.78s/epoch, loss=0.000442, prev_loss=0.000458]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:18,  5.79batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:15,  6.91batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:13,  8.01batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:11,  9.06batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:10, 10.00batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:09, 10.78batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 11.42batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:07, 11.97batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 12.44batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 12.56batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:06, 12.78batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 12.92batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:06, 12.98batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 12.98batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 12.90batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:06, 12.79batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:05, 12.97batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 13.02batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 13.16batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 13.19batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 13.12batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 13.03batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:05, 12.95batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 13.04batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 13.07batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:04, 13.00batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 13.05batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 13.07batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:04, 13.11batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 13.02batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 12.75batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:03, 12.84batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:05<00:03, 12.99batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 13.12batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 13.16batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:02, 13.09batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 13.21batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 13.07batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:05<00:02, 13.20batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 13.18batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 13.16batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 13.30batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 13.24batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 13.31batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:06<00:01, 13.34batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:07<00:01, 13.26batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 13.22batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 13.14batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:00, 13.11batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 12.95batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 12.89batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:07<00:00, 12.91batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 12.93batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 12.98batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.06batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:35<01:01,  8.78s/epoch, loss=0.000367, prev_loss=0.000442]
Training epochs on cuda:  40%|████      | 4/10 [00:35<00:52,  8.80s/epoch, loss=0.000367, prev_loss=0.000442]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:21,  5.00batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:17,  6.10batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:14,  7.27batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:12,  8.33batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:10,  9.34batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:09, 10.21batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 10.91batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:08, 11.10batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 11.78batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 11.99batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:07, 12.39batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 12.53batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:02<00:06, 12.70batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 12.92batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 12.97batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:06, 13.04batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:05, 13.05batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 13.03batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 13.06batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 13.01batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 12.98batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 13.07batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:05, 12.97batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 12.82batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 12.84batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:04<00:04, 12.79batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 12.92batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 12.91batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:04, 12.44batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:04, 12.49batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 12.71batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:05<00:03, 12.84batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:05<00:03, 12.90batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 12.84batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 12.67batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:03, 12.87batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 13.01batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 13.03batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:06<00:02, 13.17batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 13.07batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 13.13batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 13.01batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 13.18batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 13.11batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:07<00:01, 13.16batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:07<00:01, 13.04batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 13.17batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 13.07batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:00, 13.27batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 13.44batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 13.22batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:08<00:00, 13.03batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 13.17batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 13.15batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.11batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:44<00:52,  8.80s/epoch, loss=0.000306, prev_loss=0.000367]
Training epochs on cuda:  50%|█████     | 5/10 [00:44<00:44,  8.83s/epoch, loss=0.000306, prev_loss=0.000367]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:18,  5.74batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:15,  6.88batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:13,  8.04batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:11,  9.15batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:09, 10.19batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:08, 11.04batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 11.75batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:07, 12.10batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 12.33batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 12.37batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:07, 12.51batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 12.63batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:02<00:06, 12.62batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 12.55batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 12.56batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:06, 11.96batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:06, 12.19batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:06, 12.30batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 12.27batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 12.20batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 12.06batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 12.09batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:05, 12.07batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:05, 12.42batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 12.76batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:04<00:04, 12.86batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 12.93batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 12.94batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:04, 12.75batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 12.81batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 12.76batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:05<00:03, 12.75batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:05<00:03, 12.75batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 12.77batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 12.73batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:03, 12.81batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 12.81batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 12.63batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:06<00:02, 12.55batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 12.75batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 12.74batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 12.95batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 13.12batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 13.16batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:07<00:01, 13.23batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:07<00:01, 13.22batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 13.14batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 12.99batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:00, 13.01batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 12.96batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 13.12batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:08<00:00, 13.25batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 13.27batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 13.29batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.31batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:53<00:44,  8.83s/epoch, loss=0.000247, prev_loss=0.000306]
Training epochs on cuda:  60%|██████    | 6/10 [00:53<00:35,  8.86s/epoch, loss=0.000247, prev_loss=0.000306]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:20,  5.20batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:16,  6.33batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:13,  7.55batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:11,  8.68batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:10,  9.68batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:09, 10.49batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 11.16batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:08, 11.58batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 12.07batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 12.48batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:06, 12.81batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 13.03batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:06, 13.16batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 13.10batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 13.08batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:05, 13.30batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:05, 13.48batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 13.59batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 13.64batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 13.34batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 13.47batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:04, 13.55batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:04, 13.69batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 12.82batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 13.27batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:04, 13.49batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 13.63batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 13.72batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:03, 13.79batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 13.69batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 13.46batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:03, 13.39batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:03, 13.40batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 13.40batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 13.39batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:02, 13.45batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 13.46batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 13.62batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:05<00:02, 13.65batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:05<00:02, 13.71batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 13.74batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:01, 13.73batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 13.50batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 13.58batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:06<00:01, 13.54batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:06<00:01, 13.54batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 13.28batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 13.19batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:00, 13.26batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 13.27batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 13.38batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:07<00:00, 13.41batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:07<00:00, 13.42batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 13.18batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.06batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [01:01<00:35,  8.86s/epoch, loss=0.000217, prev_loss=0.000247]
Training epochs on cuda:  70%|███████   | 7/10 [01:01<00:26,  8.79s/epoch, loss=0.000217, prev_loss=0.000247]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:16,  6.70batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:13,  7.83batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:11,  8.91batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:10,  9.89batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:09, 10.65batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:08, 11.21batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 11.74batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:07, 12.05batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 12.29batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 12.38batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:07, 12.18batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 12.56batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:02<00:06, 12.66batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 12.64batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 12.70batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:06, 12.90batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:05, 13.00batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 12.94batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 12.91batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 13.06batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 13.13batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 13.28batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:04, 13.20batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 13.29batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 13.14batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:04, 13.04batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 13.18batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 13.04batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:04, 13.24batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 13.09batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 13.16batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:03, 13.09batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:05<00:03, 12.45batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 12.49batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 12.78batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:03, 12.70batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 12.73batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 12.88batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:06<00:02, 13.01batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 13.06batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 13.13batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 13.18batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 13.19batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 13.28batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:06<00:01, 13.28batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:07<00:01, 13.18batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 13.04batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 13.13batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:00, 13.13batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 13.12batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 13.19batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:07<00:00, 13.37batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 13.43batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 13.21batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.37batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [01:10<00:26,  8.79s/epoch, loss=0.0002, prev_loss=0.000217]  
Training epochs on cuda:  80%|████████  | 8/10 [01:10<00:17,  8.80s/epoch, loss=0.0002, prev_loss=0.000217]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:16,  6.79batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:13,  7.93batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:11,  9.02batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:10,  9.93batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:09, 10.70batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:08, 11.31batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 11.73batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:07, 12.23batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 12.50batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 12.64batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:06, 12.75batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 12.85batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:06, 12.99batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 13.01batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 13.15batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:05, 13.33batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:05, 13.33batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 13.29batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 13.26batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 13.33batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 13.47batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 13.16batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:04, 13.04batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 13.12batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 13.14batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:04, 12.87batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 13.08batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 13.18batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:04, 13.14batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 13.24batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 13.24batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:03, 13.29batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:05<00:03, 13.19batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 13.18batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 13.14batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:03, 12.92batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 12.98batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 13.10batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:05<00:02, 13.11batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 12.96batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 12.95batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 13.02batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 13.09batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 13.16batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:06<00:01, 13.27batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:06<00:01, 13.30batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 13.32batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 13.26batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:00, 13.24batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 13.34batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 13.33batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:07<00:00, 13.22batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 13.40batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 13.27batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 13.18batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:19<00:17,  8.80s/epoch, loss=0.000189, prev_loss=0.0002]
Training epochs on cuda:  90%|█████████ | 9/10 [01:19<00:08,  8.79s/epoch, loss=0.000189, prev_loss=0.0002]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:18,  5.95batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:15,  7.10batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:12,  8.15batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:11,  9.16batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:09, 10.12batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:09, 10.94batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:01<00:08, 11.54batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:08, 11.86batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:07, 12.10batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:07, 12.41batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:07, 12.48batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:06, 12.79batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:02<00:06, 12.86batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:02<00:06, 12.98batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:02<00:06, 13.00batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:06, 13.03batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:05, 13.05batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:05, 12.91batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:05, 12.97batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:03<00:05, 13.11batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:03<00:05, 13.18batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:03<00:05, 13.14batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:03<00:04, 13.18batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 13.27batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:04, 13.14batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:04, 13.08batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:04<00:04, 13.01batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:04<00:04, 13.02batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:04<00:04, 13.01batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:04<00:03, 12.99batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:04<00:03, 12.56batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:03, 12.39batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:05<00:03, 11.99batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:05<00:03, 12.46batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:05<00:03, 12.74batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:05<00:03, 12.82batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:05<00:02, 13.04batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:05<00:02, 13.08batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:06<00:02, 13.09batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:06<00:02, 13.32batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:06<00:02, 13.32batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:06<00:02, 13.36batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:06<00:01, 13.20batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:06<00:01, 13.13batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:06<00:01, 13.05batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:07<00:01, 12.96batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:07<00:01, 13.21batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:07<00:01, 13.57batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:07<00:00, 13.83batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:07<00:00, 13.99batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:07<00:00, 14.14batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:07<00:00, 14.19batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:08<00:00, 14.27batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:08<00:00, 14.33batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:08<00:00, 14.37batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:27<00:08,  8.79s/epoch, loss=0.000182, prev_loss=0.000189]2021-09-16:18:36:55,584 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:36:56,562 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:37:11,643 INFO     [evaluator.py:657] Evaluation took 15.08s seconds
2021-09-16:18:37:11,775 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 13.62s/epoch, loss=0.000182, prev_loss=0.000189]
Training epochs on cuda: 100%|██████████| 10/10 [01:44<00:00, 10.41s/epoch, loss=0.000182, prev_loss=0.000189]
2021-09-16:18:37:11,776 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmph21u1w6r'
2021-09-16:18:37:11,862 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmph21u1w6r' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:19, 2.02ktriple/s]
Evaluating on cuda:   1%|▏         | 512/38.8k [00:00<00:17, 2.16ktriple/s]
Evaluating on cuda:   3%|▎         | 1.02k/38.8k [00:00<00:16, 2.27ktriple/s]
Evaluating on cuda:   4%|▍         | 1.54k/38.8k [00:00<00:15, 2.37ktriple/s]
Evaluating on cuda:   5%|▌         | 2.05k/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   7%|▋         | 2.56k/38.8k [00:01<00:14, 2.49ktriple/s]
Evaluating on cuda:   8%|▊         | 3.07k/38.8k [00:01<00:14, 2.53ktriple/s]
Evaluating on cuda:   9%|▉         | 3.58k/38.8k [00:01<00:13, 2.56ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.60ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.61ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.62ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.62ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.59ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.60ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.62ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:37:26,837 INFO     [evaluator.py:657] Evaluation took 14.95s seconds
[I 2021-09-16 18:37:26,843] Trial 51 finished with value: 0.9672825273713663 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.011615403278360166, 'optimizer.lr': 0.041066982439997124, 'negative_sampler.num_negs_per_pos': 56, 'training.batch_size': 2832}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:37:26,872 WARNING  [api.py:823] No random seed is specified. Setting to 3660858888.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:18,  5.58batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:15,  6.74batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  7.97batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.14batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.13batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.66batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 10.96batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.62batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 11.92batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 12.40batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.75batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.79batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 13.03batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.94batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 12.94batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 12.75batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 12.67batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 12.59batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 12.61batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 12.50batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:05, 12.48batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:05, 12.57batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 12.75batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 12.93batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.02batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 13.06batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 13.13batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.22batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.08batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.03batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.14batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.18batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.19batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.37batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 13.42batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 13.53batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 13.66batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 13.75batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 13.70batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:01, 13.73batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 13.76batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 13.72batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 13.71batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 13.53batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 13.64batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 13.65batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.41batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.29batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.36batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.43batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.50batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 13.49batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 13.52batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:08<?, ?epoch/s, loss=0.000452, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:08<01:15,  8.39s/epoch, loss=0.000452, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:20,  5.06batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:16,  6.17batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:13,  7.31batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:11,  8.34batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:10,  9.39batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:09, 10.13batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 10.78batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:08, 11.32batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 11.65batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 11.93batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.37batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.78batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.87batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 13.02batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.07batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.16batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.19batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.13batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.06batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.21batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.25batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.38batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.42batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.45batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.42batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 13.45batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:03, 13.36batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.22batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.22batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.23batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.39batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.38batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.25batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.01batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 12.95batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.88batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 13.06batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 11.65batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:06<00:02, 11.98batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 12.28batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:02, 12.49batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.62batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.80batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.86batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.84batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 12.49batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:01, 12.00batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 11.86batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 12.26batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.22batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 12.51batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:08<00:00, 12.46batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.76batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:17<01:15,  8.39s/epoch, loss=0.000449, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:17<01:07,  8.48s/epoch, loss=0.000449, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:17,  6.03batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.16batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.35batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.31batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.09batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.88batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.35batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.69batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 11.96batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 12.13batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.40batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.54batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.95batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 13.02batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.03batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.17batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.07batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.18batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.22batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.13batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:05, 12.95batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 12.86batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 12.80batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 12.24batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 12.45batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 12.75batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 12.70batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:04, 12.74batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.71batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.51batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.16batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:05<00:03, 12.38batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 12.53batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.86batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 12.82batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.87batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.56batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 12.68batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:06<00:02, 12.94batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 12.88batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.58batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.62batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.69batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.67batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:07<00:01, 12.91batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 13.03batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:01, 12.29batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 12.35batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 12.52batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.83batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.02batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:08<00:00, 12.99batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.78batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:25<01:07,  8.48s/epoch, loss=0.000449, prev_loss=0.000449]
Training epochs on cuda:  30%|███       | 3/10 [00:25<00:59,  8.56s/epoch, loss=0.000449, prev_loss=0.000449]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:16,  6.35batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:13,  7.48batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.40batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.32batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.26batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 11.04batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.54batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.00batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.44batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.78batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.95batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.98batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.85batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.70batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:06, 12.83batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.05batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.32batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 12.42batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 11.86batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 12.12batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:05, 12.11batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:05, 12.33batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 12.52batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 12.80batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 12.82batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 12.63batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 12.79batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:04, 12.68batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.68batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.89batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.63batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:05<00:03, 12.47batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 12.41batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.20batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:03, 12.10batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.38batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.15batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:06<00:02, 12.25batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:06<00:02, 12.60batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 12.57batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.55batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.37batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.59batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.86batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:07<00:01, 13.06batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 13.13batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.29batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.26batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.14batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.17batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:08<00:00, 13.02batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:08<00:00, 13.23batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.97batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:34<00:59,  8.56s/epoch, loss=0.000449, prev_loss=0.000449]
Training epochs on cuda:  40%|████      | 4/10 [00:34<00:51,  8.62s/epoch, loss=0.000449, prev_loss=0.000449]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:17,  6.04batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.14batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.36batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.45batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.31batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 11.06batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.55batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.92batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.46batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.58batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.69batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.97batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 12.70batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.97batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:06, 12.77batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.04batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.09batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.47batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.25batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 12.99batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:05, 12.76batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 12.97batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.26batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.45batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.60batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 13.46batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 13.19batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.03batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.87batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 11.96batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.22batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 12.57batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 12.93batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:03, 12.08batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:03, 12.19batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.38batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.58batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 12.65batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:06<00:02, 12.70batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 12.75batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.61batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.68batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.66batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.57batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.58batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 12.27batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:01, 12.21batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 12.53batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 12.78batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.06batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 12.38batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:08<00:00, 12.52batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.56batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:43<00:51,  8.62s/epoch, loss=0.000449, prev_loss=0.000449]
Training epochs on cuda:  50%|█████     | 5/10 [00:43<00:43,  8.64s/epoch, loss=0.000449, prev_loss=0.000449]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:18,  5.63batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:15,  6.63batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  7.81batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:11,  8.83batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09,  9.83batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:09, 10.34batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.06batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.58batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.10batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.61batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.69batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 13.05batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 13.12batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:05, 13.46batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.14batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.21batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 12.99batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 12.93batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.03batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.24batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.08batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 12.96batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 12.92batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 12.82batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 12.85batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 12.66batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 12.91batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.11batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.13batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 12.84batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 12.61batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 12.82batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.12batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.20batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 13.31batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 13.23batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 13.34batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 13.46batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 13.53batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:01, 13.55batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 13.52batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 13.53batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 13.03batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.85batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.91batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 12.88batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.09batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.21batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.34batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.50batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.55batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 13.54batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 13.18batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:51<00:43,  8.64s/epoch, loss=0.000448, prev_loss=0.000449]
Training epochs on cuda:  60%|██████    | 6/10 [00:51<00:34,  8.61s/epoch, loss=0.000448, prev_loss=0.000449]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:18,  5.82batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  6.95batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.11batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.20batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09, 10.18batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.99batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:07, 11.66batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.09batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.44batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.66batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.88batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 13.03batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 13.12batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 13.14batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 12.92batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.10batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.19batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.34batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.31batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.39batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.44batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.46batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.41batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.44batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.45batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 13.41batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:03, 13.53batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.69batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.77batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.87batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.92batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.96batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:04<00:02, 14.01batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 14.04batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 14.05batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 14.07batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 14.08batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 14.08batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 14.08batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:05<00:01, 14.09batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 14.11batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 14.10batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 14.10batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 14.08batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 14.10batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:06<00:01, 14.09batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:06<00:00, 14.08batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 14.09batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 14.10batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 14.11batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 14.11batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 14.10batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:07<00:00, 14.10batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:59<00:34,  8.61s/epoch, loss=0.000446, prev_loss=0.000448]
Training epochs on cuda:  70%|███████   | 7/10 [00:59<00:25,  8.45s/epoch, loss=0.000446, prev_loss=0.000448]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:15,  6.88batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:12,  8.08batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:10,  9.27batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:09, 10.33batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:08, 11.11batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 11.56batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:07, 12.02batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 12.42batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 12.67batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:06, 12.90batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.75batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.98batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:01<00:06, 13.07batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 13.12batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 13.14batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.28batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.37batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.48batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.33batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:02<00:05, 13.32batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.34batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.29batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.05batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 12.99batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 12.82batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 12.77batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:04, 12.80batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:04, 12.73batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 12.85batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.02batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.11batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.19batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:04<00:03, 13.11batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.01batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 13.18batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 13.12batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 13.15batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 13.16batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 13.11batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 13.15batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.96batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 13.13batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 13.20batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 13.17batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 13.21batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:06<00:01, 13.35batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.40batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.46batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.37batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.28batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 12.29batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 12.43batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.64batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [01:08<00:25,  8.45s/epoch, loss=0.000443, prev_loss=0.000446]
Training epochs on cuda:  80%|████████  | 8/10 [01:08<00:16,  8.47s/epoch, loss=0.000443, prev_loss=0.000446]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:22,  4.65batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:17,  5.78batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:14,  6.97batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:12,  8.11batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:10,  9.15batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:09, 10.04batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 10.71batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:08, 11.32batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 11.75batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 12.18batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.51batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.65batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.50batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.68batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 12.85batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.04batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.15batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.17batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.07batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.12batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.11batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.20batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.14batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.24batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.24batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:04<00:04, 13.31batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:03, 13.45batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.40batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.45batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.48batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.45batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.46batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.49batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.21batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 13.29batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 12.90batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 12.85batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 12.90batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 12.97batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 13.01batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 13.02batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 13.11batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 13.22batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 13.22batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 13.32batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 13.20batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:00, 13.26batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 13.35batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.23batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 13.23batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 13.21batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 13.12batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 13.15batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:16<00:16,  8.47s/epoch, loss=0.000437, prev_loss=0.000443]
Training epochs on cuda:  90%|█████████ | 9/10 [01:16<00:08,  8.45s/epoch, loss=0.000437, prev_loss=0.000443]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/106 [00:00<00:17,  5.89batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:14,  7.03batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:12,  8.20batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:10,  9.24batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:09,  9.93batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:08, 10.72batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:01<00:08, 11.35batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:01<00:07, 11.50batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:01<00:07, 11.82batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:01<00:07, 12.14batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:01<00:06, 12.23batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:01<00:06, 12.47batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:02<00:06, 12.86batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:02<00:06, 12.81batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:02<00:05, 12.95batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:02<00:05, 13.22batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:02<00:05, 13.23batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:02<00:05, 13.17batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:02<00:05, 13.28batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:03<00:05, 13.38batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:03<00:04, 13.40batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:03<00:04, 13.39batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:03<00:04, 13.46batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:03<00:04, 13.50batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:03<00:04, 13.46batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:03<00:04, 13.46batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:04<00:03, 13.50batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:04<00:03, 13.51batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:04<00:03, 13.59batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:04<00:03, 13.58batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:04<00:03, 13.43batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:04<00:03, 13.40batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:05<00:03, 13.31batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:05<00:02, 13.28batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:05<00:02, 13.25batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:05<00:02, 13.26batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:05<00:02, 13.21batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:05<00:02, 13.12batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:05<00:02, 12.60batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:06<00:02, 12.71batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:06<00:01, 12.80batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:06<00:01, 12.93batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:06<00:01, 12.92batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:06<00:01, 12.91batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:06<00:01, 12.85batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:07<00:01, 12.84batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:07<00:01, 12.92batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:07<00:00, 12.99batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:07<00:00, 13.01batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:07<00:00, 12.82batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:07<00:00, 12.83batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:07<00:00, 12.93batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:08<00:00, 12.90batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:25<00:08,  8.45s/epoch, loss=0.000433, prev_loss=0.000437]2021-09-16:18:38:52,898 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:38:54,15 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:39:09,127 INFO     [evaluator.py:657] Evaluation took 15.11s seconds
2021-09-16:18:39:09,298 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:41<00:00, 13.40s/epoch, loss=0.000433, prev_loss=0.000437]
Training epochs on cuda: 100%|██████████| 10/10 [01:41<00:00, 10.18s/epoch, loss=0.000433, prev_loss=0.000437]
2021-09-16:18:39:09,299 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp3o9zr_be'
2021-09-16:18:39:09,368 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3o9zr_be' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.39ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.50ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.59ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.61ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.62ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.60ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.61ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.62ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:39:24,305 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
[I 2021-09-16 18:39:24,313] Trial 52 finished with value: 0.4133899017703794 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.06813020399520647, 'optimizer.lr': 0.037628488252772235, 'negative_sampler.num_negs_per_pos': 52, 'training.batch_size': 2932}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:39:24,334 WARNING  [api.py:823] No random seed is specified. Setting to 621250741.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:20,  5.59batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:19,  5.85batch/s]

Training batches on cuda:   4%|▎         | 4/114 [00:00<00:15,  6.88batch/s]

Training batches on cuda:   5%|▌         | 6/114 [00:00<00:13,  7.84batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:12,  8.66batch/s]

Training batches on cuda:   9%|▉         | 10/114 [00:01<00:11,  9.37batch/s]

Training batches on cuda:  11%|█         | 12/114 [00:01<00:10,  9.94batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:01<00:09, 10.41batch/s]

Training batches on cuda:  14%|█▍        | 16/114 [00:01<00:09, 10.76batch/s]

Training batches on cuda:  16%|█▌        | 18/114 [00:01<00:08, 10.93batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:01<00:08, 11.13batch/s]

Training batches on cuda:  19%|█▉        | 22/114 [00:02<00:08, 11.19batch/s]

Training batches on cuda:  21%|██        | 24/114 [00:02<00:07, 11.33batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:02<00:07, 11.51batch/s]

Training batches on cuda:  25%|██▍       | 28/114 [00:02<00:07, 11.53batch/s]

Training batches on cuda:  26%|██▋       | 30/114 [00:02<00:07, 11.52batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:02<00:07, 11.52batch/s]

Training batches on cuda:  30%|██▉       | 34/114 [00:03<00:06, 11.51batch/s]

Training batches on cuda:  32%|███▏      | 36/114 [00:03<00:06, 11.48batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:03<00:06, 11.49batch/s]

Training batches on cuda:  35%|███▌      | 40/114 [00:03<00:06, 11.49batch/s]

Training batches on cuda:  37%|███▋      | 42/114 [00:03<00:06, 11.44batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:03<00:06, 11.43batch/s]

Training batches on cuda:  40%|████      | 46/114 [00:04<00:05, 11.43batch/s]

Training batches on cuda:  42%|████▏     | 48/114 [00:04<00:05, 11.50batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:04<00:05, 11.63batch/s]

Training batches on cuda:  46%|████▌     | 52/114 [00:04<00:05, 11.70batch/s]

Training batches on cuda:  47%|████▋     | 54/114 [00:04<00:05, 11.78batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:04<00:04, 11.80batch/s]

Training batches on cuda:  51%|█████     | 58/114 [00:05<00:04, 11.84batch/s]

Training batches on cuda:  53%|█████▎    | 60/114 [00:05<00:04, 11.83batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:05<00:04, 11.81batch/s]

Training batches on cuda:  56%|█████▌    | 64/114 [00:05<00:04, 11.86batch/s]

Training batches on cuda:  58%|█████▊    | 66/114 [00:05<00:04, 11.87batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:06<00:03, 11.80batch/s]

Training batches on cuda:  61%|██████▏   | 70/114 [00:06<00:03, 11.83batch/s]

Training batches on cuda:  63%|██████▎   | 72/114 [00:06<00:03, 11.80batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:06<00:03, 11.73batch/s]

Training batches on cuda:  67%|██████▋   | 76/114 [00:06<00:03, 11.66batch/s]

Training batches on cuda:  68%|██████▊   | 78/114 [00:06<00:03, 11.76batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:07<00:02, 11.88batch/s]

Training batches on cuda:  72%|███████▏  | 82/114 [00:07<00:02, 11.98batch/s]

Training batches on cuda:  74%|███████▎  | 84/114 [00:07<00:02, 11.96batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:07<00:02, 12.00batch/s]

Training batches on cuda:  77%|███████▋  | 88/114 [00:07<00:02, 11.91batch/s]

Training batches on cuda:  79%|███████▉  | 90/114 [00:07<00:02, 11.86batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:08<00:01, 11.85batch/s]

Training batches on cuda:  82%|████████▏ | 94/114 [00:08<00:01, 11.77batch/s]

Training batches on cuda:  84%|████████▍ | 96/114 [00:08<00:01, 11.75batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:08<00:01, 11.75batch/s]

Training batches on cuda:  88%|████████▊ | 100/114 [00:08<00:01, 11.77batch/s]

Training batches on cuda:  89%|████████▉ | 102/114 [00:08<00:01, 11.75batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:09<00:00, 11.74batch/s]

Training batches on cuda:  93%|█████████▎| 106/114 [00:09<00:00, 11.73batch/s]

Training batches on cuda:  95%|█████████▍| 108/114 [00:09<00:00, 11.76batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:09<00:00, 11.73batch/s]

Training batches on cuda:  98%|█████████▊| 112/114 [00:09<00:00, 11.75batch/s]

Training batches on cuda: 100%|██████████| 114/114 [00:09<00:00, 11.61batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:10<?, ?epoch/s, loss=0.000806, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:10<01:31, 10.14s/epoch, loss=0.000806, prev_loss=nan]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:25,  4.44batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:20,  5.45batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:16,  6.47batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:14,  7.48batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:12,  8.38batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:01<00:11,  9.17batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:01<00:10,  9.82batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:01<00:09, 10.30batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:01<00:09, 10.69batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:01<00:08, 11.02batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:01<00:08, 11.27batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:02<00:07, 11.44batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:02<00:07, 11.55batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:02<00:07, 11.46batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:02<00:07, 11.57batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:02<00:07, 11.56batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:02<00:07, 11.46batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:03<00:06, 11.53batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:03<00:06, 11.64batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:03<00:06, 11.69batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:03<00:06, 11.69batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:03<00:06, 11.67batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:03<00:05, 11.67batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:04<00:05, 11.78batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:04<00:05, 11.83batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:04<00:05, 11.58batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:04<00:05, 11.67batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:04<00:05, 11.63batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:05<00:04, 11.69batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:05<00:04, 11.76batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:05<00:04, 11.80batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:05<00:04, 11.65batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:05<00:04, 11.69batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:05<00:04, 11.71batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:06<00:03, 11.72batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:06<00:03, 11.79batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:06<00:03, 11.83batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:06<00:03, 11.85batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:06<00:03, 11.86batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:06<00:02, 11.79batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:07<00:02, 11.83batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:07<00:02, 11.83batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:07<00:02, 11.74batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:07<00:02, 11.71batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:07<00:02, 11.70batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:07<00:02, 11.39batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:08<00:01, 11.30batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:08<00:01, 11.45batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:08<00:01, 11.57batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:08<00:01, 11.67batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:08<00:01, 11.63batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:08<00:00, 11.65batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:09<00:00, 11.72batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:09<00:00, 11.78batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:09<00:00, 11.82batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:09<00:00, 11.85batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:09<00:00, 11.86batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:20<01:31, 10.14s/epoch, loss=0.000781, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:20<01:21, 10.16s/epoch, loss=0.000781, prev_loss=nan]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:22,  5.05batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:21,  5.30batch/s]

Training batches on cuda:   4%|▎         | 4/114 [00:00<00:17,  6.30batch/s]

Training batches on cuda:   5%|▌         | 6/114 [00:00<00:14,  7.31batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:12,  8.27batch/s]

Training batches on cuda:   9%|▉         | 10/114 [00:01<00:11,  9.04batch/s]

Training batches on cuda:  11%|█         | 12/114 [00:01<00:10,  9.71batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:01<00:09, 10.24batch/s]

Training batches on cuda:  14%|█▍        | 16/114 [00:01<00:09, 10.63batch/s]

Training batches on cuda:  16%|█▌        | 18/114 [00:01<00:08, 11.01batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:01<00:08, 11.18batch/s]

Training batches on cuda:  19%|█▉        | 22/114 [00:02<00:08, 11.34batch/s]

Training batches on cuda:  21%|██        | 24/114 [00:02<00:07, 11.49batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:02<00:07, 11.59batch/s]

Training batches on cuda:  25%|██▍       | 28/114 [00:02<00:07, 11.66batch/s]

Training batches on cuda:  26%|██▋       | 30/114 [00:02<00:07, 11.71batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:02<00:06, 11.77batch/s]

Training batches on cuda:  30%|██▉       | 34/114 [00:03<00:06, 11.91batch/s]

Training batches on cuda:  32%|███▏      | 36/114 [00:03<00:06, 11.79batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:03<00:06, 11.89batch/s]

Training batches on cuda:  35%|███▌      | 40/114 [00:03<00:06, 11.77batch/s]

Training batches on cuda:  37%|███▋      | 42/114 [00:03<00:06, 11.81batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:03<00:05, 11.79batch/s]

Training batches on cuda:  40%|████      | 46/114 [00:04<00:05, 11.78batch/s]

Training batches on cuda:  42%|████▏     | 48/114 [00:04<00:05, 11.83batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:04<00:05, 11.84batch/s]

Training batches on cuda:  46%|████▌     | 52/114 [00:04<00:05, 11.76batch/s]

Training batches on cuda:  47%|████▋     | 54/114 [00:04<00:05, 11.73batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:04<00:04, 11.75batch/s]

Training batches on cuda:  51%|█████     | 58/114 [00:05<00:04, 11.76batch/s]

Training batches on cuda:  53%|█████▎    | 60/114 [00:05<00:04, 11.70batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:05<00:04, 11.63batch/s]

Training batches on cuda:  56%|█████▌    | 64/114 [00:05<00:04, 11.66batch/s]

Training batches on cuda:  58%|█████▊    | 66/114 [00:05<00:04, 11.66batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:05<00:03, 11.53batch/s]

Training batches on cuda:  61%|██████▏   | 70/114 [00:06<00:03, 11.55batch/s]

Training batches on cuda:  63%|██████▎   | 72/114 [00:06<00:03, 11.66batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:06<00:03, 11.48batch/s]

Training batches on cuda:  67%|██████▋   | 76/114 [00:06<00:03, 11.52batch/s]

Training batches on cuda:  68%|██████▊   | 78/114 [00:06<00:03, 11.50batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:07<00:02, 11.53batch/s]

Training batches on cuda:  72%|███████▏  | 82/114 [00:07<00:02, 11.46batch/s]

Training batches on cuda:  74%|███████▎  | 84/114 [00:07<00:02, 11.36batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:07<00:02, 11.48batch/s]

Training batches on cuda:  77%|███████▋  | 88/114 [00:07<00:02, 11.59batch/s]

Training batches on cuda:  79%|███████▉  | 90/114 [00:07<00:02, 11.72batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:08<00:01, 11.68batch/s]

Training batches on cuda:  82%|████████▏ | 94/114 [00:08<00:01, 11.83batch/s]

Training batches on cuda:  84%|████████▍ | 96/114 [00:08<00:01, 11.81batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:08<00:01, 11.89batch/s]

Training batches on cuda:  88%|████████▊ | 100/114 [00:08<00:01, 11.83batch/s]

Training batches on cuda:  89%|████████▉ | 102/114 [00:08<00:01, 11.79batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:09<00:00, 11.53batch/s]

Training batches on cuda:  93%|█████████▎| 106/114 [00:09<00:00, 11.52batch/s]

Training batches on cuda:  95%|█████████▍| 108/114 [00:09<00:00, 11.64batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:09<00:00, 11.66batch/s]

Training batches on cuda:  98%|█████████▊| 112/114 [00:09<00:00, 11.60batch/s]

Training batches on cuda: 100%|██████████| 114/114 [00:09<00:00, 12.12batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:30<01:21, 10.16s/epoch, loss=0.000781, prev_loss=0.000781]
Training epochs on cuda:  30%|███       | 3/10 [00:30<01:11, 10.18s/epoch, loss=0.000781, prev_loss=0.000781]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:26,  4.22batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:21,  5.24batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:17,  6.27batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:14,  7.31batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:12,  8.25batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:01<00:11,  8.96batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:01<00:10,  9.68batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:01<00:09, 10.19batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:01<00:09, 10.65batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:01<00:08, 10.83batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:01<00:08, 11.10batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:02<00:08, 11.27batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:02<00:07, 11.42batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:02<00:07, 11.45batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:02<00:07, 11.59batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:02<00:07, 11.46batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:02<00:06, 11.62batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:03<00:06, 11.48batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:03<00:06, 11.54batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:03<00:06, 11.55batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:03<00:06, 11.66batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:03<00:06, 11.73batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:04<00:05, 11.76batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:04<00:05, 11.58batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:04<00:05, 11.75batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:04<00:05, 11.83batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:04<00:05, 11.92batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:04<00:04, 11.96batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:05<00:04, 11.79batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:05<00:04, 11.72batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:05<00:05,  9.42batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:05<00:05,  9.98batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:05<00:04, 10.38batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:06<00:04, 10.81batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:06<00:04, 11.17batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:06<00:03, 11.38batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:06<00:03, 11.56batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:06<00:03, 11.48batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:06<00:03, 11.62batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:07<00:02, 11.75batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:07<00:02, 11.84batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:07<00:02, 11.75batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:07<00:02, 11.88batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:07<00:02, 11.95batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:07<00:02, 12.00batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:08<00:01, 11.96batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:08<00:01, 11.60batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:08<00:01, 11.54batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:08<00:01, 11.62batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:08<00:01, 11.45batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:08<00:01, 11.51batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:09<00:00, 11.64batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:09<00:00, 11.66batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:09<00:00, 11.78batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:09<00:00, 11.60batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:09<00:00, 11.53batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:09<00:00, 11.60batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:40<01:11, 10.18s/epoch, loss=0.000781, prev_loss=0.000781]
Training epochs on cuda:  40%|████      | 4/10 [00:40<01:01, 10.24s/epoch, loss=0.000781, prev_loss=0.000781]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:16,  6.67batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:14,  7.60batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:12,  8.45batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:11,  9.23batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:10,  9.77batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:01<00:10, 10.26batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:01<00:09, 10.62batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:01<00:09, 10.98batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:01<00:08, 11.23batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:01<00:08, 11.41batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:01<00:08, 11.55batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:02<00:07, 11.60batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:02<00:07, 11.69batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:02<00:07, 11.75batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:02<00:07, 11.75batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:02<00:07, 11.76batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:02<00:06, 11.78batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:03<00:06, 11.80batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:03<00:06, 11.85batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:03<00:06, 11.86batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:03<00:06, 11.85batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:03<00:05, 11.87batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:03<00:05, 11.85batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:04<00:05, 11.86batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:04<00:05, 11.90batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:04<00:05, 11.96batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:04<00:05, 11.88batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:04<00:04, 11.92batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:04<00:04, 11.78batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:05<00:04, 11.73batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:05<00:04, 11.65batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:05<00:04, 11.68batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:05<00:04, 11.64batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:05<00:04, 11.57batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:05<00:03, 11.51batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:06<00:03, 11.52batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:06<00:03, 11.47batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:06<00:03, 11.41batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:06<00:03, 11.46batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:06<00:03, 11.50batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:06<00:02, 11.47batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:07<00:02, 11.42batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:07<00:02, 11.45batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:07<00:02, 11.49batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:07<00:02, 11.50batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:07<00:01, 11.51batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:08<00:01, 11.44batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:08<00:01, 11.46batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:08<00:01, 11.45batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:08<00:01, 11.43batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:08<00:01, 11.44batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:08<00:00, 11.50batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:09<00:00, 11.59batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:09<00:00, 11.52batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:09<00:00, 11.65batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:09<00:00, 11.54batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:09<00:00, 11.65batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:51<01:01, 10.24s/epoch, loss=0.000781, prev_loss=0.000781]
Training epochs on cuda:  50%|█████     | 5/10 [00:51<00:50, 10.19s/epoch, loss=0.000781, prev_loss=0.000781]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:25,  4.49batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:20,  5.52batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:16,  6.52batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:14,  7.53batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:12,  8.45batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:01<00:11,  9.15batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:01<00:10,  9.84batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:01<00:09, 10.37batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:01<00:09, 10.64batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:01<00:08, 10.81batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:01<00:08, 10.84batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:02<00:08, 11.11batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:02<00:07, 11.32batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:02<00:07, 11.49batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:02<00:07, 11.62batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:02<00:07, 11.62batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:02<00:06, 11.62batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:03<00:06, 11.72batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:03<00:06, 11.81batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:03<00:06, 11.72batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:03<00:06, 11.73batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:03<00:06, 11.78batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:03<00:05, 11.70batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:04<00:05, 11.77batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:04<00:05, 11.76batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:04<00:05, 11.72batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:04<00:05, 11.67batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:04<00:05, 11.70batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:05<00:04, 11.76batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:05<00:04, 11.81batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:05<00:04, 11.75batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:05<00:04, 11.78batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:05<00:04, 11.82batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:05<00:04, 11.64batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:06<00:03, 11.68batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:06<00:03, 11.71batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:06<00:03, 11.64batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:06<00:03, 11.72batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:06<00:03, 11.68batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:06<00:02, 11.74batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:07<00:02, 11.76batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:07<00:02, 11.75batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:07<00:02, 11.74batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:07<00:02, 11.78batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:07<00:02, 11.83batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:07<00:01, 11.76batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:08<00:01, 11.77batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:08<00:01, 11.81batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:08<00:01, 11.86batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:08<00:01, 11.88batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:08<00:01, 11.84batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:08<00:00, 11.87batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:09<00:00, 11.84batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:09<00:00, 11.83batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:09<00:00, 11.72batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:09<00:00, 11.76batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:09<00:00, 11.75batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [01:01<00:50, 10.19s/epoch, loss=0.000781, prev_loss=0.000781]
Training epochs on cuda:  60%|██████    | 6/10 [01:01<00:40, 10.18s/epoch, loss=0.000781, prev_loss=0.000781]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:26,  4.33batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:20,  5.35batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:17,  6.39batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:14,  7.41batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:12,  8.37batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:01<00:11,  9.19batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:01<00:10,  9.87batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:01<00:09, 10.42batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:01<00:08, 10.84batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:01<00:08, 11.13batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:01<00:08, 11.34batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:02<00:07, 11.49batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:02<00:07, 11.65batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:02<00:07, 11.73batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:02<00:07, 11.79batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:02<00:07, 11.77batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:02<00:06, 11.83batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:03<00:06, 11.86batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:03<00:06, 11.87batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:03<00:06, 11.83batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:03<00:06, 11.84batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:03<00:06, 11.79batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:03<00:05, 11.82batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:04<00:05, 11.83batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:04<00:05, 11.84batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:04<00:05, 11.86batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:04<00:05, 11.19batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:04<00:05, 11.35batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:04<00:04, 11.52batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:05<00:04, 11.33batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:05<00:04, 11.49batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:05<00:04, 11.63batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:05<00:04, 11.67batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:05<00:04, 11.66batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:06<00:03, 11.73batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:06<00:03, 11.78batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:06<00:03, 11.83batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:06<00:03, 11.86batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:06<00:03, 11.69batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:06<00:02, 11.75batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:07<00:02, 11.80batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:07<00:02, 11.84batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:07<00:02, 11.87batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:07<00:02, 11.88batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:07<00:02, 11.91batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:07<00:01, 11.83batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:08<00:01, 11.79batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:08<00:01, 11.78batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:08<00:01, 11.80batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:08<00:01, 11.87batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:08<00:01, 11.68batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:08<00:00, 11.72batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:09<00:00, 11.75batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:09<00:00, 11.81batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:09<00:00, 11.86batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:09<00:00, 10.97batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:09<00:00, 11.23batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [01:11<00:40, 10.18s/epoch, loss=0.000781, prev_loss=0.000781]
Training epochs on cuda:  70%|███████   | 7/10 [01:11<00:30, 10.16s/epoch, loss=0.000781, prev_loss=0.000781]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:25,  4.45batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:20,  5.47batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:16,  6.48batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:14,  7.50batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:12,  8.40batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:01<00:11,  9.22batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:01<00:10,  9.88batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:01<00:10,  9.75batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:01<00:10,  9.04batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:01<00:09,  9.72batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:02<00:09, 10.30batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:02<00:08, 10.73batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:02<00:08, 10.90batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:02<00:07, 11.20batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:02<00:07, 11.41batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:02<00:07, 11.53batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:03<00:06, 11.60batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:03<00:06, 11.53batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:03<00:06, 11.49batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:03<00:06, 11.56batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:03<00:06, 11.67batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:03<00:06, 11.76batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:04<00:05, 11.83batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:04<00:05, 11.89batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:04<00:05, 11.88batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:04<00:05, 11.77batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:04<00:05, 11.80batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:04<00:04, 11.83batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:05<00:04, 11.86batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:05<00:04, 11.81batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:05<00:04, 11.82batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:05<00:04, 11.82batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:05<00:04, 11.83batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:05<00:03, 11.88batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:06<00:03, 11.86batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:06<00:03, 11.85batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:06<00:03, 11.87batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:06<00:03, 11.82batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:06<00:03, 11.79batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:06<00:02, 11.77batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:07<00:02, 11.82batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:07<00:02, 11.86batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:07<00:02, 11.81batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:07<00:02, 11.82batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:07<00:02, 11.86batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:07<00:01, 11.86batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:08<00:01, 11.87batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:08<00:01, 11.89batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:08<00:01, 11.90batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:08<00:01, 11.83batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:08<00:01, 11.78batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:08<00:00, 11.76batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:09<00:00, 11.73batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:09<00:00, 11.77batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:09<00:00, 11.77batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:09<00:00, 11.67batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:09<00:00, 11.74batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [01:21<00:30, 10.16s/epoch, loss=0.000781, prev_loss=0.000781]
Training epochs on cuda:  80%|████████  | 8/10 [01:21<00:20, 10.17s/epoch, loss=0.000781, prev_loss=0.000781]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:27,  4.11batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:21,  5.11batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:17,  6.13batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:14,  7.17batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:12,  8.15batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:01<00:11,  8.98batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:01<00:10,  9.68batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:01<00:09, 10.27batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:01<00:09, 10.72batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:01<00:08, 10.89batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:01<00:08, 11.16batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:02<00:08, 11.36batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:02<00:07, 11.49batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:02<00:07, 11.56batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:02<00:07, 11.64batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:02<00:07, 11.65batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:02<00:06, 11.68batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:03<00:06, 11.53batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:03<00:06, 11.69batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:03<00:06, 11.68batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:03<00:06, 11.76batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:03<00:06, 11.79batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:03<00:05, 11.80batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:04<00:05, 11.78batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:04<00:05, 11.78batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:04<00:05, 11.78batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:04<00:05, 11.71batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:04<00:05, 11.68batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:05<00:04, 11.65batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:05<00:04, 11.67batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:05<00:04, 11.52batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:05<00:04, 11.62batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:05<00:04, 11.69batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:05<00:03, 11.77batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:06<00:03, 11.75batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:06<00:03, 11.76batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:06<00:03, 11.80batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:06<00:03, 11.85batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:06<00:03, 11.84batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:06<00:02, 11.71batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:07<00:02, 11.75batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:07<00:02, 11.73batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:07<00:02, 11.77batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:07<00:02, 11.78batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:07<00:02, 11.74batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:07<00:01, 11.75batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:08<00:01, 11.77batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:08<00:01, 11.79batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:08<00:01, 11.81batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:08<00:01, 11.83batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:08<00:01, 11.82batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:08<00:00, 11.85batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:09<00:00, 11.87batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:09<00:00, 11.88batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:09<00:00, 11.88batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:09<00:00, 11.89batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:09<00:00, 11.89batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:31<00:20, 10.17s/epoch, loss=0.000781, prev_loss=0.000781]
Training epochs on cuda:  90%|█████████ | 9/10 [01:31<00:10, 10.15s/epoch, loss=0.000781, prev_loss=0.000781]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:26,  4.19batch/s]

Training batches on cuda:   3%|▎         | 3/114 [00:00<00:21,  5.17batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:17,  6.22batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:14,  7.27batch/s]

Training batches on cuda:   8%|▊         | 9/114 [00:00<00:12,  8.23batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:01<00:11,  9.09batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:01<00:10,  9.78batch/s]

Training batches on cuda:  13%|█▎        | 15/114 [00:01<00:09, 10.32batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:01<00:09, 10.61batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:01<00:08, 10.88batch/s]

Training batches on cuda:  18%|█▊        | 21/114 [00:02<00:09,  9.50batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:02<00:09, 10.08batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:02<00:08, 10.57batch/s]

Training batches on cuda:  24%|██▎       | 27/114 [00:02<00:07, 10.93batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:02<00:07, 11.20batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:02<00:07, 11.29batch/s]

Training batches on cuda:  29%|██▉       | 33/114 [00:03<00:07, 11.39batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:03<00:06, 11.57batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:03<00:06, 11.68batch/s]

Training batches on cuda:  34%|███▍      | 39/114 [00:03<00:06, 11.75batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:03<00:06, 11.78batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:03<00:06, 11.83batch/s]

Training batches on cuda:  39%|███▉      | 45/114 [00:04<00:05, 11.86batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:04<00:05, 11.85batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:04<00:05, 11.84batch/s]

Training batches on cuda:  45%|████▍     | 51/114 [00:04<00:05, 11.86batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:04<00:05, 11.83batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:04<00:04, 11.81batch/s]

Training batches on cuda:  50%|█████     | 57/114 [00:05<00:04, 11.82batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:05<00:04, 11.62batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:05<00:04, 11.72batch/s]

Training batches on cuda:  55%|█████▌    | 63/114 [00:05<00:04, 11.75batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:05<00:04, 11.70batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:05<00:03, 11.76batch/s]

Training batches on cuda:  61%|██████    | 69/114 [00:06<00:03, 11.80batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:06<00:03, 11.83batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:06<00:03, 11.78batch/s]

Training batches on cuda:  66%|██████▌   | 75/114 [00:06<00:03, 11.82batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:06<00:03, 11.84batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:07<00:03, 10.04batch/s]

Training batches on cuda:  71%|███████   | 81/114 [00:07<00:03, 10.53batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:07<00:02, 10.87batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:07<00:02, 10.92batch/s]

Training batches on cuda:  76%|███████▋  | 87/114 [00:07<00:02, 11.14batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:07<00:02, 11.33batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:08<00:02, 11.35batch/s]

Training batches on cuda:  82%|████████▏ | 93/114 [00:08<00:01, 11.48batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:08<00:01, 11.57batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:08<00:01, 11.66batch/s]

Training batches on cuda:  87%|████████▋ | 99/114 [00:08<00:01, 11.77batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:08<00:01, 11.82batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:09<00:00, 11.83batch/s]

Training batches on cuda:  92%|█████████▏| 105/114 [00:09<00:00, 11.80batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:09<00:00, 11.68batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:09<00:00, 11.56batch/s]

Training batches on cuda:  97%|█████████▋| 111/114 [00:09<00:00, 11.62batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:09<00:00, 11.69batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:42<00:10, 10.15s/epoch, loss=0.000781, prev_loss=0.000781]2021-09-16:18:41:07,49 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:41:08,116 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:41:23,128 INFO     [evaluator.py:657] Evaluation took 15.01s seconds
2021-09-16:18:41:23,279 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 15.09s/epoch, loss=0.000781, prev_loss=0.000781]
Training epochs on cuda: 100%|██████████| 10/10 [01:58<00:00, 11.82s/epoch, loss=0.000781, prev_loss=0.000781]
2021-09-16:18:41:23,280 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpovhi97wr'
2021-09-16:18:41:23,335 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpovhi97wr' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.18ktriple/s]
Evaluating on cuda:   1%|▏         | 512/38.8k [00:00<00:16, 2.28ktriple/s]
Evaluating on cuda:   3%|▎         | 1.02k/38.8k [00:00<00:15, 2.38ktriple/s]
Evaluating on cuda:   4%|▍         | 1.54k/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   5%|▌         | 2.05k/38.8k [00:00<00:14, 2.48ktriple/s]
Evaluating on cuda:   7%|▋         | 2.56k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   8%|▊         | 3.07k/38.8k [00:01<00:13, 2.56ktriple/s]
Evaluating on cuda:   9%|▉         | 3.58k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:11, 2.46ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:11, 2.50ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.54ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.57ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:10, 2.59ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.60ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.62ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:41:38,317 INFO     [evaluator.py:657] Evaluation took 14.95s seconds
[I 2021-09-16 18:41:38,324] Trial 53 finished with value: -0.00535024495684433 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.18087042483768104, 'optimizer.lr': 0.004395241149519509, 'negative_sampler.num_negs_per_pos': 71, 'training.batch_size': 2732}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:41:38,348 WARNING  [api.py:823] No random seed is specified. Setting to 2619526013.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:15,  7.20batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:12,  8.60batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:10, 10.12batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:08, 11.48batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:07, 12.73batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:07, 13.68batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:06, 14.52batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:06, 15.20batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:05, 15.65batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:05, 15.97batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 16.22batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 16.51batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 16.53batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 16.53batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:04, 16.58batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:01<00:04, 16.58batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 16.60batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 16.57batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 16.46batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 16.50batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 16.52batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 16.39batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:03, 16.52batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:02<00:03, 16.73batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 16.68batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 16.56batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 16.51batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 16.45batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 16.45batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 16.21batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:03, 16.28batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:03<00:02, 16.12batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 16.17batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 15.80batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 15.93batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 15.99batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 16.03batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 16.02batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 16.15batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:04<00:01, 16.22batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 16.12batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 16.05batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 15.84batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 15.87batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 15.68batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 16.04batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 15.76batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:05<00:00, 15.83batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 16.00batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 16.00batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 15.90batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 16.01batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 16.11batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 16.13batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 16.11batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:07<?, ?epoch/s, loss=0.000468, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:07<01:04,  7.13s/epoch, loss=0.000468, prev_loss=nan]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:13,  7.97batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:11,  9.36batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:09, 10.72batch/s]

Training batches on cuda:   5%|▌         | 6/110 [00:00<00:10, 10.11batch/s]

Training batches on cuda:   7%|▋         | 8/110 [00:00<00:08, 11.39batch/s]

Training batches on cuda:   9%|▉         | 10/110 [00:00<00:08, 12.43batch/s]

Training batches on cuda:  11%|█         | 12/110 [00:00<00:07, 13.46batch/s]

Training batches on cuda:  13%|█▎        | 14/110 [00:00<00:06, 14.18batch/s]

Training batches on cuda:  15%|█▍        | 16/110 [00:01<00:06, 14.73batch/s]

Training batches on cuda:  16%|█▋        | 18/110 [00:01<00:06, 14.95batch/s]

Training batches on cuda:  18%|█▊        | 20/110 [00:01<00:05, 15.32batch/s]

Training batches on cuda:  20%|██        | 22/110 [00:01<00:05, 15.64batch/s]

Training batches on cuda:  22%|██▏       | 24/110 [00:01<00:05, 15.97batch/s]

Training batches on cuda:  24%|██▎       | 26/110 [00:01<00:05, 16.16batch/s]

Training batches on cuda:  25%|██▌       | 28/110 [00:01<00:05, 16.26batch/s]

Training batches on cuda:  27%|██▋       | 30/110 [00:01<00:04, 16.26batch/s]

Training batches on cuda:  29%|██▉       | 32/110 [00:02<00:04, 16.35batch/s]

Training batches on cuda:  31%|███       | 34/110 [00:02<00:04, 16.18batch/s]

Training batches on cuda:  33%|███▎      | 36/110 [00:02<00:04, 16.09batch/s]

Training batches on cuda:  35%|███▍      | 38/110 [00:02<00:04, 16.00batch/s]

Training batches on cuda:  36%|███▋      | 40/110 [00:02<00:04, 15.74batch/s]

Training batches on cuda:  38%|███▊      | 42/110 [00:02<00:04, 15.76batch/s]

Training batches on cuda:  40%|████      | 44/110 [00:02<00:04, 15.79batch/s]

Training batches on cuda:  42%|████▏     | 46/110 [00:02<00:04, 15.77batch/s]

Training batches on cuda:  44%|████▎     | 48/110 [00:03<00:03, 15.65batch/s]

Training batches on cuda:  45%|████▌     | 50/110 [00:03<00:03, 15.65batch/s]

Training batches on cuda:  47%|████▋     | 52/110 [00:03<00:03, 15.88batch/s]

Training batches on cuda:  49%|████▉     | 54/110 [00:03<00:03, 15.50batch/s]

Training batches on cuda:  51%|█████     | 56/110 [00:03<00:03, 15.63batch/s]

Training batches on cuda:  53%|█████▎    | 58/110 [00:03<00:03, 15.85batch/s]

Training batches on cuda:  55%|█████▍    | 60/110 [00:03<00:03, 15.99batch/s]

Training batches on cuda:  56%|█████▋    | 62/110 [00:03<00:03, 15.85batch/s]

Training batches on cuda:  58%|█████▊    | 64/110 [00:04<00:02, 15.53batch/s]

Training batches on cuda:  60%|██████    | 66/110 [00:04<00:02, 15.69batch/s]

Training batches on cuda:  62%|██████▏   | 68/110 [00:04<00:02, 15.72batch/s]

Training batches on cuda:  64%|██████▎   | 70/110 [00:04<00:02, 15.24batch/s]

Training batches on cuda:  65%|██████▌   | 72/110 [00:04<00:02, 15.42batch/s]

Training batches on cuda:  67%|██████▋   | 74/110 [00:04<00:02, 15.54batch/s]

Training batches on cuda:  69%|██████▉   | 76/110 [00:04<00:02, 15.19batch/s]

Training batches on cuda:  71%|███████   | 78/110 [00:05<00:02, 15.37batch/s]

Training batches on cuda:  73%|███████▎  | 80/110 [00:05<00:01, 15.72batch/s]

Training batches on cuda:  75%|███████▍  | 82/110 [00:05<00:01, 15.67batch/s]

Training batches on cuda:  76%|███████▋  | 84/110 [00:05<00:01, 15.66batch/s]

Training batches on cuda:  78%|███████▊  | 86/110 [00:05<00:01, 14.55batch/s]

Training batches on cuda:  80%|████████  | 88/110 [00:05<00:01, 14.87batch/s]

Training batches on cuda:  82%|████████▏ | 90/110 [00:05<00:01, 15.30batch/s]

Training batches on cuda:  84%|████████▎ | 92/110 [00:05<00:01, 15.56batch/s]

Training batches on cuda:  85%|████████▌ | 94/110 [00:06<00:01, 15.86batch/s]

Training batches on cuda:  87%|████████▋ | 96/110 [00:06<00:00, 15.70batch/s]

Training batches on cuda:  89%|████████▉ | 98/110 [00:06<00:00, 15.97batch/s]

Training batches on cuda:  91%|█████████ | 100/110 [00:06<00:00, 16.18batch/s]

Training batches on cuda:  93%|█████████▎| 102/110 [00:06<00:00, 16.33batch/s]

Training batches on cuda:  95%|█████████▍| 104/110 [00:06<00:00, 16.24batch/s]

Training batches on cuda:  96%|█████████▋| 106/110 [00:06<00:00, 16.21batch/s]

Training batches on cuda:  98%|█████████▊| 108/110 [00:06<00:00, 16.04batch/s]

Training batches on cuda: 100%|██████████| 110/110 [00:07<00:00, 16.41batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:14<01:04,  7.13s/epoch, loss=0.000466, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:14<00:57,  7.19s/epoch, loss=0.000466, prev_loss=nan]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:16,  6.65batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:13,  8.06batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:10,  9.55batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:09, 10.75batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:08, 11.94batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:07, 12.83batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:07, 13.51batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:06, 14.00batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:06, 14.71batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:06, 14.95batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 15.27batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 15.25batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 15.30batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 15.33batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:05, 15.52batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:05, 15.74batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 15.87batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 15.95batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 16.09batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 15.96batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 16.13batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 15.86batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:04, 15.75batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:03, 15.89batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 15.99batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 16.16batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 16.37batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 16.47batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 16.43batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 16.48batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:02, 16.52batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:02, 16.53batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 16.22batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 15.92batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 16.09batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 16.30batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 16.21batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 15.93batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 16.17batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:05<00:01, 15.99batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 15.97batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 15.85batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 15.96batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 16.12batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 16.05batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 16.24batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 16.26batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:06<00:00, 16.19batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 16.03batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 16.18batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 16.35batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 16.44batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 16.50batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 16.58batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 16.57batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:21<00:57,  7.19s/epoch, loss=0.000466, prev_loss=0.000466]
Training epochs on cuda:  30%|███       | 3/10 [00:21<00:50,  7.19s/epoch, loss=0.000466, prev_loss=0.000466]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:13,  8.32batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:10,  9.75batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:09, 11.13batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:08, 12.39batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:07, 13.44batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:06, 14.30batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:06, 14.89batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:06, 15.32batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:05, 15.68batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:05, 16.01batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 16.22batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 16.33batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 16.41batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 16.51batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:04, 16.28batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:01<00:04, 16.18batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 16.37batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 16.29batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 16.17batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 15.85batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 15.71batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 15.79batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:04, 15.99batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:02<00:03, 15.89batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 16.02batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 15.99batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 15.67batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 15.70batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 15.93batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 15.79batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:03, 15.67batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:03<00:02, 15.67batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 15.72batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 15.81batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 15.72batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 15.89batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 16.02batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 16.19batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 16.09batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:04<00:01, 16.06batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 15.98batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 15.98batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 15.92batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 16.10batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 16.24batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 15.79batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 15.66batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:05<00:00, 15.79batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 14.75batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 15.17batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 15.36batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 15.64batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 15.97batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 16.11batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 16.27batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:28<00:50,  7.19s/epoch, loss=0.000466, prev_loss=0.000466]
Training epochs on cuda:  40%|████      | 4/10 [00:28<00:43,  7.20s/epoch, loss=0.000466, prev_loss=0.000466]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:13,  8.28batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:11,  9.68batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:09, 11.08batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:08, 12.29batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:07, 13.35batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:07, 14.00batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:06, 14.70batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:06, 15.17batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:05, 15.60batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:05, 15.60batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 15.76batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 16.02batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 16.17batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 16.26batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:04, 16.38batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:01<00:04, 16.40batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 16.08batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 16.18batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 15.87batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 15.98batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 16.05batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 16.16batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:03, 16.28batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:02<00:03, 16.27batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 16.33batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 16.39batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 16.41batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 16.47batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 16.42batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 16.44batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:03, 16.18batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:03<00:02, 16.30batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 16.40batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 16.33batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 16.33batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 16.54batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 16.40batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 16.42batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 16.25batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:04<00:01, 16.26batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 16.36batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 16.28batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 16.35batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 16.26batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 16.09batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 16.20batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 15.91batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:05<00:00, 15.87batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 15.85batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 15.87batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 15.94batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 15.94batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 15.96batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 16.11batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 16.28batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:36<00:43,  7.20s/epoch, loss=0.000463, prev_loss=0.000466]
Training epochs on cuda:  50%|█████     | 5/10 [00:36<00:35,  7.18s/epoch, loss=0.000463, prev_loss=0.000466]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:16,  6.74batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:13,  8.10batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:10,  9.56batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:09, 10.95batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:08, 12.19batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:07, 13.24batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:06, 14.05batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:06, 14.71batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:06, 14.80batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:06, 15.01batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 14.97batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 15.08batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 15.18batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 15.34batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:05, 15.47batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:05, 15.40batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 15.56batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 15.72batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 15.66batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 15.17batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 15.29batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 15.46batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:04, 15.38batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 15.36batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 15.40batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 15.70batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 15.88batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 16.17batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 16.30batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 16.50batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:02, 16.55batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:02, 16.60batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 16.60batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 16.63batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 16.33batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 16.41batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 13.82batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 14.48batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 15.03batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:05<00:02, 15.48batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 15.81batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 15.87batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 15.83batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 16.06batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 15.85batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 15.90batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 15.98batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:06<00:00, 16.05batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 16.09batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 16.11batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 16.28batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 16.32batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 16.03batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 16.00batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 15.38batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:43<00:35,  7.18s/epoch, loss=0.000459, prev_loss=0.000463]
Training epochs on cuda:  60%|██████    | 6/10 [00:43<00:28,  7.22s/epoch, loss=0.000459, prev_loss=0.000463]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:15,  7.07batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:12,  8.41batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:10,  9.84batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:09, 11.17batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:08, 12.11batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:07, 13.01batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:07, 13.49batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:06, 14.30batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:06, 14.56batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:06, 15.11batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 15.46batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 15.69batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 15.95batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 15.91batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:05, 16.12batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:04, 16.22batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 16.31batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 16.26batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 16.36batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 16.39batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 16.45batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 16.47batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:03, 16.46batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:02<00:03, 16.41batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 16.46batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 15.82batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 15.88batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 15.88batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 16.09batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 16.13batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:03, 16.19batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:03<00:02, 16.30batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 16.08batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 16.08batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 15.86batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 15.62batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 15.78batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 16.10batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 16.15batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:04<00:01, 16.05batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 16.06batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 16.01batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 15.94batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 16.12batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 16.28batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 16.24batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 16.34batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:05<00:00, 16.36batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 16.29batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 16.15batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 15.98batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 16.10batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 15.96batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 16.03batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 16.03batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:50<00:28,  7.22s/epoch, loss=0.000454, prev_loss=0.000459]
Training epochs on cuda:  70%|███████   | 7/10 [00:50<00:21,  7.21s/epoch, loss=0.000454, prev_loss=0.000459]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:15,  6.87batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:12,  8.24batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:10,  9.65batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:09, 10.36batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:08, 11.68batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:07, 12.80batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:07, 13.68batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:06, 14.33batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:06, 14.88batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:05, 15.27batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 15.51batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 15.68batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 15.67batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 15.86batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:05, 15.94batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:05, 15.61batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 15.85batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 15.88batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 16.06batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 15.56batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 15.16batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 15.33batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:04, 15.24batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:04, 15.33batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 15.43batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 15.43batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 15.51batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 15.52batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 15.76batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 15.99batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:03, 15.85batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:02, 15.92batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 15.77batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 15.93batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 15.50batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 15.80batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 16.09batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 16.11batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 16.12batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:05<00:01, 16.08batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 16.16batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 16.22batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 16.25batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 16.09batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 16.17batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 16.29batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 16.29batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:06<00:00, 16.35batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 16.47batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 16.52batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 16.18batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 16.19batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 16.18batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 16.21batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 16.29batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:57<00:21,  7.21s/epoch, loss=0.000437, prev_loss=0.000454]
Training epochs on cuda:  80%|████████  | 8/10 [00:57<00:14,  7.25s/epoch, loss=0.000437, prev_loss=0.000454]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:15,  7.21batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:12,  8.66batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:10, 10.15batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:08, 11.53batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:07, 12.74batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:07, 13.36batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:06, 14.15batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:06, 14.57batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:06, 14.98batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:05, 15.42batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 15.81batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 15.92batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 15.78batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 15.92batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:05, 15.50batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:02<00:05, 15.40batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 15.67batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 15.36batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 15.53batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 15.77batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 15.34batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 15.42batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:04, 15.64batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:03<00:03, 15.91batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 16.03batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 16.14batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 16.21batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 16.23batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 16.17batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 16.15batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:03, 16.05batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:04<00:02, 16.04batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 15.99batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 16.05batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 16.11batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 16.17batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 16.14batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 16.16batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 15.95batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:05<00:01, 16.11batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 15.87batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 16.02batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 16.04batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 16.10batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 16.05batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 16.16batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 16.11batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:06<00:00, 16.18batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 16.37batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 16.04batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 16.20batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 16.28batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 16.38batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 16.24batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 16.24batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:05<00:14,  7.25s/epoch, loss=0.000424, prev_loss=0.000437]
Training epochs on cuda:  90%|█████████ | 9/10 [01:05<00:07,  7.27s/epoch, loss=0.000424, prev_loss=0.000437]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:12,  8.48batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:10,  9.75batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:09, 11.01batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:08, 12.13batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:07, 13.05batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:07, 13.84batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:06, 14.48batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:01<00:06, 14.97batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:01<00:06, 15.39batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:01<00:05, 15.69batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:01<00:05, 15.93batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:01<00:05, 16.06batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:01<00:05, 16.24batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:01<00:05, 16.25batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:01<00:04, 16.30batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:01<00:04, 16.03batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:02<00:04, 16.13batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:02<00:04, 16.26batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:02<00:04, 16.38batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:02<00:04, 16.24batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:02<00:04, 16.46batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:02<00:04, 16.50batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:02<00:03, 16.28batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:02<00:03, 16.18batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:03<00:03, 16.04batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:03<00:03, 16.17batch/s]

Training batches on cuda:  48%|████▊     | 53/110 [00:03<00:03, 16.16batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:03<00:03, 16.13batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:03<00:03, 16.07batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:03<00:03, 15.91batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:03<00:03, 16.09batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:03<00:02, 16.22batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:04<00:02, 16.39batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:04<00:02, 16.34batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:04<00:02, 16.29batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:04<00:02, 16.26batch/s]

Training batches on cuda:  66%|██████▋   | 73/110 [00:04<00:02, 16.08batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:04<00:02, 16.01batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:04<00:02, 16.09batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:04<00:01, 16.02batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:05<00:01, 15.98batch/s]

Training batches on cuda:  75%|███████▌  | 83/110 [00:05<00:01, 16.11batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:05<00:01, 16.29batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:05<00:01, 16.19batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:05<00:01, 16.12batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:05<00:01, 16.19batch/s]

Training batches on cuda:  85%|████████▍ | 93/110 [00:05<00:01, 16.13batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:05<00:00, 15.98batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:06<00:00, 16.02batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:06<00:00, 15.60batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:06<00:00, 15.59batch/s]

Training batches on cuda:  94%|█████████▎| 103/110 [00:06<00:00, 15.67batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:06<00:00, 15.73batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:06<00:00, 15.77batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:06<00:00, 15.70batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:12<00:07,  7.27s/epoch, loss=0.000409, prev_loss=0.000424]2021-09-16:18:42:51,321 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:42:52,322 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:43:07,399 INFO     [evaluator.py:657] Evaluation took 15.08s seconds
2021-09-16:18:43:07,532 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00, 12.10s/epoch, loss=0.000409, prev_loss=0.000424]
Training epochs on cuda: 100%|██████████| 10/10 [01:28<00:00,  8.85s/epoch, loss=0.000409, prev_loss=0.000424]
2021-09-16:18:43:07,532 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpja0k108e'
2021-09-16:18:43:07,604 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpja0k108e' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.28ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.37ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.50ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:43:22,495 INFO     [evaluator.py:657] Evaluation took 14.88s seconds
[I 2021-09-16 18:43:22,500] Trial 54 finished with value: 0.7012211840366342 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.04711219995590006, 'optimizer.lr': 0.045980181395423404, 'negative_sampler.num_negs_per_pos': 43, 'training.batch_size': 2832}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:43:22,522 WARNING  [api.py:823] No random seed is specified. Setting to 2404836334.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/94 [00:00<00:03, 28.33batch/s]

Training batches on cuda:  10%|▉         | 9/94 [00:00<00:02, 33.22batch/s]

Training batches on cuda:  16%|█▌        | 15/94 [00:00<00:02, 37.31batch/s]

Training batches on cuda:  22%|██▏       | 21/94 [00:00<00:01, 41.12batch/s]

Training batches on cuda:  29%|██▊       | 27/94 [00:00<00:01, 44.09batch/s]

Training batches on cuda:  35%|███▌      | 33/94 [00:00<00:01, 46.08batch/s]

Training batches on cuda:  41%|████▏     | 39/94 [00:00<00:01, 47.47batch/s]

Training batches on cuda:  48%|████▊     | 45/94 [00:00<00:01, 48.65batch/s]

Training batches on cuda:  54%|█████▍    | 51/94 [00:01<00:00, 50.11batch/s]

Training batches on cuda:  61%|██████    | 57/94 [00:01<00:00, 50.81batch/s]

Training batches on cuda:  67%|██████▋   | 63/94 [00:01<00:00, 52.18batch/s]

Training batches on cuda:  73%|███████▎  | 69/94 [00:01<00:00, 53.00batch/s]

Training batches on cuda:  80%|███████▉  | 75/94 [00:01<00:00, 52.77batch/s]

Training batches on cuda:  86%|████████▌ | 81/94 [00:01<00:00, 53.80batch/s]

Training batches on cuda:  93%|█████████▎| 87/94 [00:01<00:00, 54.72batch/s]

Training batches on cuda:  99%|█████████▉| 93/94 [00:01<00:00, 54.81batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:01<?, ?epoch/s, loss=0.00065, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:01<00:17,  1.98s/epoch, loss=0.00065, prev_loss=nan]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 4/94 [00:00<00:02, 35.83batch/s]

Training batches on cuda:  11%|█         | 10/94 [00:00<00:02, 39.39batch/s]

Training batches on cuda:  16%|█▌        | 15/94 [00:00<00:01, 41.75batch/s]

Training batches on cuda:  22%|██▏       | 21/94 [00:00<00:01, 43.83batch/s]

Training batches on cuda:  29%|██▊       | 27/94 [00:00<00:01, 46.22batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:00<00:01, 46.53batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:00<00:01, 48.38batch/s]

Training batches on cuda:  47%|████▋     | 44/94 [00:00<00:01, 48.93batch/s]

Training batches on cuda:  52%|█████▏    | 49/94 [00:01<00:00, 48.37batch/s]

Training batches on cuda:  59%|█████▊    | 55/94 [00:01<00:00, 48.59batch/s]

Training batches on cuda:  65%|██████▍   | 61/94 [00:01<00:00, 48.83batch/s]

Training batches on cuda:  70%|███████   | 66/94 [00:01<00:00, 47.34batch/s]

Training batches on cuda:  76%|███████▌  | 71/94 [00:01<00:00, 46.86batch/s]

Training batches on cuda:  81%|████████  | 76/94 [00:01<00:00, 46.17batch/s]

Training batches on cuda:  86%|████████▌ | 81/94 [00:01<00:00, 45.78batch/s]

Training batches on cuda:  91%|█████████▏| 86/94 [00:01<00:00, 45.75batch/s]

Training batches on cuda:  97%|█████████▋| 91/94 [00:01<00:00, 45.53batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:17,  1.98s/epoch, loss=0.000646, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:16,  2.06s/epoch, loss=0.000646, prev_loss=nan]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/94 [00:00<00:03, 26.11batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:02, 29.70batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:02, 33.72batch/s]

Training batches on cuda:  19%|█▉        | 18/94 [00:00<00:02, 36.47batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:00<00:01, 37.20batch/s]

Training batches on cuda:  30%|██▉       | 28/94 [00:00<00:01, 39.70batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:00<00:01, 38.98batch/s]

Training batches on cuda:  39%|███▉      | 37/94 [00:00<00:01, 39.81batch/s]

Training batches on cuda:  45%|████▍     | 42/94 [00:01<00:01, 40.26batch/s]

Training batches on cuda:  50%|█████     | 47/94 [00:01<00:01, 41.90batch/s]

Training batches on cuda:  55%|█████▌    | 52/94 [00:01<00:00, 43.80batch/s]

Training batches on cuda:  61%|██████    | 57/94 [00:01<00:00, 43.54batch/s]

Training batches on cuda:  66%|██████▌   | 62/94 [00:01<00:00, 44.45batch/s]

Training batches on cuda:  71%|███████▏  | 67/94 [00:01<00:00, 45.50batch/s]

Training batches on cuda:  77%|███████▋  | 72/94 [00:01<00:00, 45.38batch/s]

Training batches on cuda:  82%|████████▏ | 77/94 [00:01<00:00, 46.26batch/s]

Training batches on cuda:  87%|████████▋ | 82/94 [00:01<00:00, 47.00batch/s]

Training batches on cuda:  93%|█████████▎| 87/94 [00:01<00:00, 47.66batch/s]

Training batches on cuda:  99%|█████████▉| 93/94 [00:02<00:00, 49.00batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:16,  2.06s/epoch, loss=0.000646, prev_loss=0.000646]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:15,  2.16s/epoch, loss=0.000646, prev_loss=0.000646]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/94 [00:00<00:03, 28.02batch/s]

Training batches on cuda:   7%|▋         | 7/94 [00:00<00:02, 30.13batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:02, 32.32batch/s]

Training batches on cuda:  17%|█▋        | 16/94 [00:00<00:02, 35.65batch/s]

Training batches on cuda:  22%|██▏       | 21/94 [00:00<00:01, 37.86batch/s]

Training batches on cuda:  28%|██▊       | 26/94 [00:00<00:01, 40.20batch/s]

Training batches on cuda:  33%|███▎      | 31/94 [00:00<00:01, 42.05batch/s]

Training batches on cuda:  39%|███▉      | 37/94 [00:00<00:01, 44.49batch/s]

Training batches on cuda:  45%|████▍     | 42/94 [00:01<00:01, 41.18batch/s]

Training batches on cuda:  51%|█████     | 48/94 [00:01<00:01, 43.61batch/s]

Training batches on cuda:  57%|█████▋    | 54/94 [00:01<00:00, 46.14batch/s]

Training batches on cuda:  63%|██████▎   | 59/94 [00:01<00:00, 44.37batch/s]

Training batches on cuda:  68%|██████▊   | 64/94 [00:01<00:00, 43.88batch/s]

Training batches on cuda:  73%|███████▎  | 69/94 [00:01<00:00, 44.81batch/s]

Training batches on cuda:  79%|███████▊  | 74/94 [00:01<00:00, 45.01batch/s]

Training batches on cuda:  84%|████████▍ | 79/94 [00:01<00:00, 45.97batch/s]

Training batches on cuda:  89%|████████▉ | 84/94 [00:01<00:00, 46.54batch/s]

Training batches on cuda:  95%|█████████▍| 89/94 [00:02<00:00, 46.20batch/s]

Training batches on cuda: 100%|██████████| 94/94 [00:02<00:00, 44.15batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:15,  2.16s/epoch, loss=0.000646, prev_loss=0.000646]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:13,  2.24s/epoch, loss=0.000646, prev_loss=0.000646]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/94 [00:00<00:03, 29.25batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:02, 33.27batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:02, 36.85batch/s]

Training batches on cuda:  20%|██        | 19/94 [00:00<00:01, 40.04batch/s]

Training batches on cuda:  26%|██▌       | 24/94 [00:00<00:01, 42.31batch/s]

Training batches on cuda:  31%|███       | 29/94 [00:00<00:01, 44.20batch/s]

Training batches on cuda:  36%|███▌      | 34/94 [00:00<00:01, 45.54batch/s]

Training batches on cuda:  41%|████▏     | 39/94 [00:00<00:01, 46.41batch/s]

Training batches on cuda:  47%|████▋     | 44/94 [00:00<00:01, 47.39batch/s]

Training batches on cuda:  52%|█████▏    | 49/94 [00:01<00:00, 47.95batch/s]

Training batches on cuda:  57%|█████▋    | 54/94 [00:01<00:00, 47.68batch/s]

Training batches on cuda:  63%|██████▎   | 59/94 [00:01<00:00, 47.82batch/s]

Training batches on cuda:  68%|██████▊   | 64/94 [00:01<00:00, 47.72batch/s]

Training batches on cuda:  73%|███████▎  | 69/94 [00:01<00:00, 47.06batch/s]

Training batches on cuda:  79%|███████▊  | 74/94 [00:01<00:00, 46.97batch/s]

Training batches on cuda:  84%|████████▍ | 79/94 [00:01<00:00, 46.74batch/s]

Training batches on cuda:  89%|████████▉ | 84/94 [00:01<00:00, 46.96batch/s]

Training batches on cuda:  95%|█████████▍| 89/94 [00:01<00:00, 47.15batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:13,  2.24s/epoch, loss=0.000646, prev_loss=0.000646]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.22s/epoch, loss=0.000646, prev_loss=0.000646]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/94 [00:00<00:03, 27.90batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:02, 31.78batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:02, 35.29batch/s]

Training batches on cuda:  19%|█▉        | 18/94 [00:00<00:01, 38.39batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:00<00:01, 40.80batch/s]

Training batches on cuda:  30%|██▉       | 28/94 [00:00<00:01, 42.99batch/s]

Training batches on cuda:  35%|███▌      | 33/94 [00:00<00:01, 44.12batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:00<00:01, 45.42batch/s]

Training batches on cuda:  46%|████▌     | 43/94 [00:00<00:01, 42.94batch/s]

Training batches on cuda:  51%|█████     | 48/94 [00:01<00:01, 43.65batch/s]

Training batches on cuda:  56%|█████▋    | 53/94 [00:01<00:00, 44.72batch/s]

Training batches on cuda:  62%|██████▏   | 58/94 [00:01<00:00, 44.60batch/s]

Training batches on cuda:  67%|██████▋   | 63/94 [00:01<00:00, 41.73batch/s]

Training batches on cuda:  72%|███████▏  | 68/94 [00:01<00:00, 41.50batch/s]

Training batches on cuda:  78%|███████▊  | 73/94 [00:01<00:00, 42.62batch/s]

Training batches on cuda:  83%|████████▎ | 78/94 [00:01<00:00, 43.19batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:01<00:00, 44.40batch/s]

Training batches on cuda:  94%|█████████▎| 88/94 [00:01<00:00, 44.51batch/s]

Training batches on cuda:  99%|█████████▉| 93/94 [00:02<00:00, 43.14batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:11,  2.22s/epoch, loss=0.000646, prev_loss=0.000646]
Training epochs on cuda:  60%|██████    | 6/10 [00:13<00:09,  2.26s/epoch, loss=0.000646, prev_loss=0.000646]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/94 [00:00<00:04, 18.41batch/s]

Training batches on cuda:   6%|▋         | 6/94 [00:00<00:04, 21.81batch/s]

Training batches on cuda:  12%|█▏        | 11/94 [00:00<00:03, 25.46batch/s]

Training batches on cuda:  17%|█▋        | 16/94 [00:00<00:02, 28.68batch/s]

Training batches on cuda:  22%|██▏       | 21/94 [00:00<00:02, 31.65batch/s]

Training batches on cuda:  27%|██▋       | 25/94 [00:00<00:02, 31.69batch/s]

Training batches on cuda:  32%|███▏      | 30/94 [00:00<00:01, 34.44batch/s]

Training batches on cuda:  37%|███▋      | 35/94 [00:00<00:01, 36.69batch/s]

Training batches on cuda:  43%|████▎     | 40/94 [00:01<00:01, 39.73batch/s]

Training batches on cuda:  48%|████▊     | 45/94 [00:01<00:01, 40.82batch/s]

Training batches on cuda:  54%|█████▍    | 51/94 [00:01<00:00, 43.90batch/s]

Training batches on cuda:  61%|██████    | 57/94 [00:01<00:00, 46.05batch/s]

Training batches on cuda:  67%|██████▋   | 63/94 [00:01<00:00, 47.36batch/s]

Training batches on cuda:  73%|███████▎  | 69/94 [00:01<00:00, 49.24batch/s]

Training batches on cuda:  80%|███████▉  | 75/94 [00:01<00:00, 50.48batch/s]

Training batches on cuda:  86%|████████▌ | 81/94 [00:01<00:00, 51.59batch/s]

Training batches on cuda:  93%|█████████▎| 87/94 [00:01<00:00, 47.60batch/s]

Training batches on cuda:  98%|█████████▊| 92/94 [00:02<00:00, 46.32batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:09,  2.26s/epoch, loss=0.000646, prev_loss=0.000646]
Training epochs on cuda:  70%|███████   | 7/10 [00:15<00:06,  2.28s/epoch, loss=0.000646, prev_loss=0.000646]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 4/94 [00:00<00:02, 34.38batch/s]

Training batches on cuda:  10%|▉         | 9/94 [00:00<00:02, 37.75batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:02, 36.96batch/s]

Training batches on cuda:  18%|█▊        | 17/94 [00:00<00:02, 35.19batch/s]

Training batches on cuda:  22%|██▏       | 21/94 [00:00<00:02, 36.42batch/s]

Training batches on cuda:  28%|██▊       | 26/94 [00:00<00:01, 38.52batch/s]

Training batches on cuda:  32%|███▏      | 30/94 [00:00<00:01, 38.86batch/s]

Training batches on cuda:  37%|███▋      | 35/94 [00:00<00:01, 40.56batch/s]

Training batches on cuda:  43%|████▎     | 40/94 [00:01<00:01, 41.13batch/s]

Training batches on cuda:  48%|████▊     | 45/94 [00:01<00:01, 41.94batch/s]

Training batches on cuda:  53%|█████▎    | 50/94 [00:01<00:01, 43.93batch/s]

Training batches on cuda:  59%|█████▊    | 55/94 [00:01<00:00, 44.03batch/s]

Training batches on cuda:  65%|██████▍   | 61/94 [00:01<00:00, 45.71batch/s]

Training batches on cuda:  71%|███████▏  | 67/94 [00:01<00:00, 47.80batch/s]

Training batches on cuda:  78%|███████▊  | 73/94 [00:01<00:00, 48.66batch/s]

Training batches on cuda:  83%|████████▎ | 78/94 [00:01<00:00, 42.30batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:01<00:00, 43.15batch/s]

Training batches on cuda:  94%|█████████▎| 88/94 [00:02<00:00, 42.16batch/s]

Training batches on cuda:  99%|█████████▉| 93/94 [00:02<00:00, 41.96batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [00:18<00:06,  2.28s/epoch, loss=0.000646, prev_loss=0.000646]
Training epochs on cuda:  80%|████████  | 8/10 [00:18<00:04,  2.36s/epoch, loss=0.000646, prev_loss=0.000646]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/94 [00:00<00:03, 26.92batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:02, 30.91batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:02, 34.63batch/s]

Training batches on cuda:  19%|█▉        | 18/94 [00:00<00:02, 37.53batch/s]

Training batches on cuda:  23%|██▎       | 22/94 [00:00<00:02, 35.60batch/s]

Training batches on cuda:  29%|██▊       | 27/94 [00:00<00:01, 38.85batch/s]

Training batches on cuda:  34%|███▍      | 32/94 [00:00<00:01, 39.86batch/s]

Training batches on cuda:  39%|███▉      | 37/94 [00:00<00:01, 41.13batch/s]

Training batches on cuda:  45%|████▍     | 42/94 [00:00<00:01, 42.68batch/s]

Training batches on cuda:  50%|█████     | 47/94 [00:01<00:01, 43.00batch/s]

Training batches on cuda:  55%|█████▌    | 52/94 [00:01<00:00, 43.83batch/s]

Training batches on cuda:  61%|██████    | 57/94 [00:01<00:00, 43.22batch/s]

Training batches on cuda:  66%|██████▌   | 62/94 [00:01<00:00, 41.12batch/s]

Training batches on cuda:  71%|███████▏  | 67/94 [00:01<00:00, 41.33batch/s]

Training batches on cuda:  77%|███████▋  | 72/94 [00:01<00:00, 39.77batch/s]

Training batches on cuda:  82%|████████▏ | 77/94 [00:01<00:00, 39.56batch/s]

Training batches on cuda:  87%|████████▋ | 82/94 [00:01<00:00, 40.20batch/s]

Training batches on cuda:  93%|█████████▎| 87/94 [00:02<00:00, 40.61batch/s]

Training batches on cuda:  98%|█████████▊| 92/94 [00:02<00:00, 41.01batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [00:20<00:04,  2.36s/epoch, loss=0.000646, prev_loss=0.000646]
Training epochs on cuda:  90%|█████████ | 9/10 [00:20<00:02,  2.39s/epoch, loss=0.000646, prev_loss=0.000646]

Training batches on cuda:   0%|          | 0/94 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/94 [00:00<00:03, 28.30batch/s]

Training batches on cuda:   9%|▊         | 8/94 [00:00<00:02, 31.31batch/s]

Training batches on cuda:  14%|█▍        | 13/94 [00:00<00:02, 34.05batch/s]

Training batches on cuda:  19%|█▉        | 18/94 [00:00<00:02, 37.52batch/s]

Training batches on cuda:  24%|██▍       | 23/94 [00:00<00:01, 39.69batch/s]

Training batches on cuda:  30%|██▉       | 28/94 [00:00<00:01, 41.02batch/s]

Training batches on cuda:  35%|███▌      | 33/94 [00:00<00:01, 42.87batch/s]

Training batches on cuda:  40%|████      | 38/94 [00:00<00:01, 43.99batch/s]

Training batches on cuda:  46%|████▌     | 43/94 [00:00<00:01, 45.07batch/s]

Training batches on cuda:  51%|█████     | 48/94 [00:01<00:01, 44.61batch/s]

Training batches on cuda:  56%|█████▋    | 53/94 [00:01<00:00, 43.77batch/s]

Training batches on cuda:  62%|██████▏   | 58/94 [00:01<00:00, 43.23batch/s]

Training batches on cuda:  67%|██████▋   | 63/94 [00:01<00:00, 43.14batch/s]

Training batches on cuda:  72%|███████▏  | 68/94 [00:01<00:00, 44.37batch/s]

Training batches on cuda:  78%|███████▊  | 73/94 [00:01<00:00, 42.91batch/s]

Training batches on cuda:  83%|████████▎ | 78/94 [00:01<00:00, 38.88batch/s]

Training batches on cuda:  88%|████████▊ | 83/94 [00:01<00:00, 39.78batch/s]

Training batches on cuda:  94%|█████████▎| 88/94 [00:02<00:00, 40.94batch/s]

Training batches on cuda:  99%|█████████▉| 93/94 [00:02<00:00, 42.95batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [00:23<00:02,  2.39s/epoch, loss=0.000646, prev_loss=0.000646]2021-09-16:18:43:46,284 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:43:47,417 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:44:02,433 INFO     [evaluator.py:657] Evaluation took 15.02s seconds
2021-09-16:18:44:02,589 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  7.29s/epoch, loss=0.000646, prev_loss=0.000646]
Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  3.96s/epoch, loss=0.000646, prev_loss=0.000646]
2021-09-16:18:44:02,590 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpt4iopz6q'
2021-09-16:18:44:02,649 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpt4iopz6q' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.41ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:08, 2.57ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.59ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.60ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.62ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:44:17,514 INFO     [evaluator.py:657] Evaluation took 14.84s seconds
[I 2021-09-16 18:44:17,520] Trial 55 finished with value: -0.004147454039588272 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.12749367066065653, 'optimizer.lr': 0.05619386911046546, 'negative_sampler.num_negs_per_pos': 3, 'training.batch_size': 3332}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:44:17,544 WARNING  [api.py:823] No random seed is specified. Setting to 3332245996.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/170 [00:00<00:17,  9.65batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:13, 12.06batch/s]

Training batches on cuda:   4%|▍         | 7/170 [00:00<00:11, 14.56batch/s]

Training batches on cuda:   6%|▌         | 10/170 [00:00<00:09, 17.15batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:08, 19.62batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:07, 21.42batch/s]

Training batches on cuda:  11%|█         | 19/170 [00:00<00:06, 22.76batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:06, 24.41batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:00<00:05, 25.40batch/s]

Training batches on cuda:  16%|█▋        | 28/170 [00:01<00:05, 26.39batch/s]

Training batches on cuda:  18%|█▊        | 31/170 [00:01<00:05, 26.81batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:01<00:04, 27.48batch/s]

Training batches on cuda:  22%|██▏       | 37/170 [00:01<00:04, 27.83batch/s]

Training batches on cuda:  24%|██▎       | 40/170 [00:01<00:04, 28.17batch/s]

Training batches on cuda:  25%|██▌       | 43/170 [00:01<00:04, 28.42batch/s]

Training batches on cuda:  27%|██▋       | 46/170 [00:01<00:04, 28.59batch/s]

Training batches on cuda:  29%|██▉       | 49/170 [00:01<00:04, 28.46batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:01<00:04, 28.63batch/s]

Training batches on cuda:  32%|███▏      | 55/170 [00:01<00:04, 28.63batch/s]

Training batches on cuda:  34%|███▍      | 58/170 [00:02<00:04, 27.74batch/s]

Training batches on cuda:  36%|███▌      | 61/170 [00:02<00:03, 27.81batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:02<00:03, 27.60batch/s]

Training batches on cuda:  39%|███▉      | 67/170 [00:02<00:03, 28.05batch/s]

Training batches on cuda:  41%|████      | 70/170 [00:02<00:03, 27.96batch/s]

Training batches on cuda:  43%|████▎     | 73/170 [00:02<00:03, 28.01batch/s]

Training batches on cuda:  45%|████▍     | 76/170 [00:02<00:03, 27.93batch/s]

Training batches on cuda:  46%|████▋     | 79/170 [00:02<00:03, 28.21batch/s]

Training batches on cuda:  48%|████▊     | 82/170 [00:02<00:03, 28.53batch/s]

Training batches on cuda:  50%|█████     | 85/170 [00:03<00:02, 28.85batch/s]

Training batches on cuda:  52%|█████▏    | 88/170 [00:03<00:02, 29.01batch/s]

Training batches on cuda:  54%|█████▎    | 91/170 [00:03<00:02, 28.94batch/s]

Training batches on cuda:  55%|█████▌    | 94/170 [00:03<00:02, 28.46batch/s]

Training batches on cuda:  57%|█████▋    | 97/170 [00:03<00:02, 28.45batch/s]

Training batches on cuda:  59%|█████▉    | 100/170 [00:03<00:02, 28.55batch/s]

Training batches on cuda:  61%|██████    | 103/170 [00:03<00:02, 28.21batch/s]

Training batches on cuda:  62%|██████▏   | 106/170 [00:03<00:02, 28.43batch/s]

Training batches on cuda:  64%|██████▍   | 109/170 [00:03<00:02, 28.33batch/s]

Training batches on cuda:  66%|██████▌   | 112/170 [00:04<00:02, 27.97batch/s]

Training batches on cuda:  68%|██████▊   | 115/170 [00:04<00:01, 28.36batch/s]

Training batches on cuda:  69%|██████▉   | 118/170 [00:04<00:01, 28.55batch/s]

Training batches on cuda:  71%|███████   | 121/170 [00:04<00:01, 28.89batch/s]

Training batches on cuda:  73%|███████▎  | 124/170 [00:04<00:01, 29.05batch/s]

Training batches on cuda:  75%|███████▍  | 127/170 [00:04<00:01, 28.99batch/s]

Training batches on cuda:  76%|███████▋  | 130/170 [00:04<00:01, 28.57batch/s]

Training batches on cuda:  78%|███████▊  | 133/170 [00:04<00:01, 28.51batch/s]

Training batches on cuda:  80%|████████  | 136/170 [00:04<00:01, 28.12batch/s]

Training batches on cuda:  82%|████████▏ | 139/170 [00:04<00:01, 27.77batch/s]

Training batches on cuda:  84%|████████▎ | 142/170 [00:05<00:00, 28.36batch/s]

Training batches on cuda:  85%|████████▌ | 145/170 [00:05<00:00, 27.73batch/s]

Training batches on cuda:  87%|████████▋ | 148/170 [00:05<00:00, 27.31batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:05<00:00, 27.89batch/s]

Training batches on cuda:  91%|█████████ | 154/170 [00:05<00:00, 27.47batch/s]

Training batches on cuda:  92%|█████████▏| 157/170 [00:05<00:00, 27.86batch/s]

Training batches on cuda:  94%|█████████▍| 160/170 [00:05<00:00, 27.35batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:05<00:00, 27.65batch/s]

Training batches on cuda:  98%|█████████▊| 166/170 [00:05<00:00, 27.27batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:06<00:00, 27.17batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:06<?, ?epoch/s, loss=0.000722, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:56,  6.32s/epoch, loss=0.000722, prev_loss=nan]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/170 [00:00<00:20,  8.32batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:15, 10.53batch/s]

Training batches on cuda:   4%|▍         | 7/170 [00:00<00:12, 12.92batch/s]

Training batches on cuda:   6%|▌         | 10/170 [00:00<00:10, 15.47batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:08, 17.97batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:07, 20.08batch/s]

Training batches on cuda:  11%|█         | 19/170 [00:00<00:06, 21.63batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:06, 23.21batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:00<00:05, 24.25batch/s]

Training batches on cuda:  16%|█▋        | 28/170 [00:01<00:05, 25.39batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:01<00:05, 26.46batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:01<00:04, 27.38batch/s]

Training batches on cuda:  23%|██▎       | 39/170 [00:01<00:04, 28.08batch/s]

Training batches on cuda:  25%|██▍       | 42/170 [00:01<00:04, 28.47batch/s]

Training batches on cuda:  26%|██▋       | 45/170 [00:01<00:04, 28.84batch/s]

Training batches on cuda:  28%|██▊       | 48/170 [00:01<00:04, 29.04batch/s]

Training batches on cuda:  30%|███       | 51/170 [00:01<00:04, 28.93batch/s]

Training batches on cuda:  32%|███▏      | 54/170 [00:01<00:04, 28.82batch/s]

Training batches on cuda:  34%|███▎      | 57/170 [00:02<00:03, 28.87batch/s]

Training batches on cuda:  36%|███▌      | 61/170 [00:02<00:03, 29.00batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:02<00:03, 28.89batch/s]

Training batches on cuda:  39%|███▉      | 67/170 [00:02<00:03, 28.42batch/s]

Training batches on cuda:  41%|████      | 70/170 [00:02<00:03, 28.69batch/s]

Training batches on cuda:  43%|████▎     | 73/170 [00:02<00:03, 29.03batch/s]

Training batches on cuda:  45%|████▍     | 76/170 [00:02<00:03, 29.19batch/s]

Training batches on cuda:  46%|████▋     | 79/170 [00:02<00:03, 28.80batch/s]

Training batches on cuda:  48%|████▊     | 82/170 [00:02<00:03, 28.88batch/s]

Training batches on cuda:  50%|█████     | 85/170 [00:03<00:02, 28.87batch/s]

Training batches on cuda:  52%|█████▏    | 88/170 [00:03<00:02, 28.39batch/s]

Training batches on cuda:  54%|█████▎    | 91/170 [00:03<00:02, 28.10batch/s]

Training batches on cuda:  55%|█████▌    | 94/170 [00:03<00:02, 27.69batch/s]

Training batches on cuda:  57%|█████▋    | 97/170 [00:03<00:02, 27.94batch/s]

Training batches on cuda:  59%|█████▉    | 100/170 [00:03<00:02, 27.94batch/s]

Training batches on cuda:  61%|██████    | 103/170 [00:03<00:02, 28.29batch/s]

Training batches on cuda:  63%|██████▎   | 107/170 [00:03<00:02, 28.79batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:03<00:02, 28.99batch/s]

Training batches on cuda:  66%|██████▋   | 113/170 [00:04<00:01, 29.23batch/s]

Training batches on cuda:  68%|██████▊   | 116/170 [00:04<00:01, 29.23batch/s]

Training batches on cuda:  70%|███████   | 119/170 [00:04<00:01, 29.32batch/s]

Training batches on cuda:  72%|███████▏  | 122/170 [00:04<00:01, 29.28batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:04<00:01, 29.43batch/s]

Training batches on cuda:  75%|███████▌  | 128/170 [00:04<00:01, 29.57batch/s]

Training batches on cuda:  77%|███████▋  | 131/170 [00:04<00:01, 29.62batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:04<00:01, 29.68batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:04<00:01, 29.35batch/s]

Training batches on cuda:  82%|████████▏ | 140/170 [00:04<00:01, 29.05batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:05<00:00, 29.00batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:05<00:00, 29.12batch/s]

Training batches on cuda:  88%|████████▊ | 149/170 [00:05<00:00, 29.08batch/s]

Training batches on cuda:  89%|████████▉ | 152/170 [00:05<00:00, 29.16batch/s]

Training batches on cuda:  91%|█████████ | 155/170 [00:05<00:00, 29.32batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:05<00:00, 29.06batch/s]

Training batches on cuda:  95%|█████████▍| 161/170 [00:05<00:00, 28.80batch/s]

Training batches on cuda:  96%|█████████▋| 164/170 [00:05<00:00, 28.22batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:05<00:00, 28.04batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:12<00:56,  6.32s/epoch, loss=0.000719, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:50,  6.31s/epoch, loss=0.000719, prev_loss=nan]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/170 [00:00<00:09, 17.89batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:08, 20.29batch/s]

Training batches on cuda:   5%|▍         | 8/170 [00:00<00:07, 22.17batch/s]

Training batches on cuda:   6%|▋         | 11/170 [00:00<00:06, 23.74batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:06, 25.01batch/s]

Training batches on cuda:  10%|█         | 17/170 [00:00<00:05, 25.86batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:05, 26.61batch/s]

Training batches on cuda:  14%|█▎        | 23/170 [00:00<00:05, 27.25batch/s]

Training batches on cuda:  15%|█▌        | 26/170 [00:00<00:05, 27.84batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:01<00:04, 28.27batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:01<00:04, 28.62batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:01<00:04, 28.87batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:01<00:04, 28.92batch/s]

Training batches on cuda:  24%|██▍       | 41/170 [00:01<00:04, 29.03batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:01<00:04, 29.12batch/s]

Training batches on cuda:  28%|██▊       | 47/170 [00:01<00:04, 29.14batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:01<00:04, 28.68batch/s]

Training batches on cuda:  31%|███       | 53/170 [00:01<00:04, 28.57batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:02<00:04, 27.25batch/s]

Training batches on cuda:  35%|███▍      | 59/170 [00:02<00:04, 27.35batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:02<00:03, 27.50batch/s]

Training batches on cuda:  38%|███▊      | 65/170 [00:02<00:03, 27.36batch/s]

Training batches on cuda:  40%|████      | 68/170 [00:02<00:03, 26.76batch/s]

Training batches on cuda:  42%|████▏     | 71/170 [00:02<00:03, 27.00batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:02<00:03, 26.20batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:02<00:03, 26.52batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:02<00:03, 26.72batch/s]

Training batches on cuda:  49%|████▉     | 83/170 [00:03<00:03, 26.94batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:03<00:03, 27.18batch/s]

Training batches on cuda:  53%|█████▎    | 90/170 [00:03<00:02, 27.85batch/s]

Training batches on cuda:  55%|█████▍    | 93/170 [00:03<00:02, 27.50batch/s]

Training batches on cuda:  56%|█████▋    | 96/170 [00:03<00:02, 28.18batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:03<00:02, 28.69batch/s]

Training batches on cuda:  60%|██████    | 102/170 [00:03<00:02, 28.40batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:03<00:02, 28.00batch/s]

Training batches on cuda:  64%|██████▎   | 108/170 [00:03<00:02, 27.82batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:03<00:02, 28.08batch/s]

Training batches on cuda:  67%|██████▋   | 114/170 [00:04<00:02, 27.88batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:04<00:01, 27.60batch/s]

Training batches on cuda:  71%|███████   | 120/170 [00:04<00:01, 27.49batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:04<00:01, 27.68batch/s]

Training batches on cuda:  74%|███████▍  | 126/170 [00:04<00:01, 26.68batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:04<00:01, 26.75batch/s]

Training batches on cuda:  78%|███████▊  | 132/170 [00:04<00:01, 27.49batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:04<00:01, 26.93batch/s]

Training batches on cuda:  81%|████████  | 138/170 [00:04<00:01, 26.78batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:05<00:01, 27.12batch/s]

Training batches on cuda:  85%|████████▍ | 144/170 [00:05<00:00, 27.69batch/s]

Training batches on cuda:  86%|████████▋ | 147/170 [00:05<00:00, 28.17batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:05<00:00, 27.80batch/s]

Training batches on cuda:  90%|█████████ | 153/170 [00:05<00:00, 27.46batch/s]

Training batches on cuda:  92%|█████████▏| 156/170 [00:05<00:00, 27.20batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:05<00:00, 27.03batch/s]

Training batches on cuda:  95%|█████████▌| 162/170 [00:05<00:00, 25.93batch/s]

Training batches on cuda:  97%|█████████▋| 165/170 [00:05<00:00, 26.15batch/s]

Training batches on cuda:  99%|█████████▉| 168/170 [00:06<00:00, 27.04batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:18<00:50,  6.31s/epoch, loss=0.000719, prev_loss=0.000719]
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:44,  6.33s/epoch, loss=0.000719, prev_loss=0.000719]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/170 [00:00<00:17,  9.78batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:13, 12.09batch/s]

Training batches on cuda:   4%|▍         | 7/170 [00:00<00:11, 14.51batch/s]

Training batches on cuda:   6%|▌         | 10/170 [00:00<00:09, 16.85batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:08, 19.37batch/s]

Training batches on cuda:  10%|█         | 17/170 [00:00<00:07, 21.47batch/s]

Training batches on cuda:  12%|█▏        | 21/170 [00:00<00:06, 23.50batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:00<00:05, 25.28batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:01<00:05, 26.66batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:01<00:05, 27.36batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:01<00:05, 26.86batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:01<00:04, 27.32batch/s]

Training batches on cuda:  24%|██▍       | 41/170 [00:01<00:04, 27.87batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:01<00:04, 27.84batch/s]

Training batches on cuda:  28%|██▊       | 47/170 [00:01<00:04, 28.24batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:01<00:04, 28.41batch/s]

Training batches on cuda:  31%|███       | 53/170 [00:01<00:04, 28.79batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:02<00:03, 28.78batch/s]

Training batches on cuda:  35%|███▍      | 59/170 [00:02<00:03, 28.19batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:02<00:03, 28.26batch/s]

Training batches on cuda:  38%|███▊      | 65/170 [00:02<00:03, 27.88batch/s]

Training batches on cuda:  40%|████      | 68/170 [00:02<00:03, 27.73batch/s]

Training batches on cuda:  42%|████▏     | 71/170 [00:02<00:03, 28.29batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:02<00:03, 27.42batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:02<00:03, 27.73batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:02<00:03, 28.09batch/s]

Training batches on cuda:  49%|████▉     | 83/170 [00:02<00:03, 27.48batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:03<00:03, 27.42batch/s]

Training batches on cuda:  52%|█████▏    | 89/170 [00:03<00:02, 27.78batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:03<00:02, 27.54batch/s]

Training batches on cuda:  56%|█████▌    | 95/170 [00:03<00:02, 27.44batch/s]

Training batches on cuda:  58%|█████▊    | 98/170 [00:03<00:02, 27.68batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:03<00:02, 27.62batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:03<00:02, 27.39batch/s]

Training batches on cuda:  63%|██████▎   | 107/170 [00:03<00:02, 27.45batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:03<00:02, 27.34batch/s]

Training batches on cuda:  66%|██████▋   | 113/170 [00:04<00:02, 27.13batch/s]

Training batches on cuda:  68%|██████▊   | 116/170 [00:04<00:01, 27.84batch/s]

Training batches on cuda:  70%|███████   | 119/170 [00:04<00:01, 28.33batch/s]

Training batches on cuda:  72%|███████▏  | 122/170 [00:04<00:01, 27.97batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:04<00:01, 27.26batch/s]

Training batches on cuda:  75%|███████▌  | 128/170 [00:04<00:01, 27.02batch/s]

Training batches on cuda:  77%|███████▋  | 131/170 [00:04<00:01, 27.77batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:04<00:01, 27.98batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:04<00:01, 28.01batch/s]

Training batches on cuda:  82%|████████▏ | 140/170 [00:05<00:01, 27.69batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:05<00:00, 27.73batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:05<00:00, 28.24batch/s]

Training batches on cuda:  88%|████████▊ | 149/170 [00:05<00:00, 27.78batch/s]

Training batches on cuda:  89%|████████▉ | 152/170 [00:05<00:00, 27.92batch/s]

Training batches on cuda:  91%|█████████ | 155/170 [00:05<00:00, 28.32batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:05<00:00, 27.56batch/s]

Training batches on cuda:  95%|█████████▍| 161/170 [00:05<00:00, 27.88batch/s]

Training batches on cuda:  96%|█████████▋| 164/170 [00:05<00:00, 28.02batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:06<00:00, 28.10batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:25<00:44,  6.33s/epoch, loss=0.000719, prev_loss=0.000719]
Training epochs on cuda:  40%|████      | 4/10 [00:25<00:38,  6.36s/epoch, loss=0.000719, prev_loss=0.000719]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/170 [00:00<00:10, 16.05batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:08, 18.68batch/s]

Training batches on cuda:   6%|▌         | 10/170 [00:00<00:07, 21.21batch/s]

Training batches on cuda:   7%|▋         | 12/170 [00:00<00:07, 20.26batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:06, 22.62batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:06, 24.74batch/s]

Training batches on cuda:  14%|█▍        | 24/170 [00:00<00:05, 26.34batch/s]

Training batches on cuda:  16%|█▋        | 28/170 [00:01<00:05, 27.71batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:01<00:04, 28.77batch/s]

Training batches on cuda:  21%|██        | 36/170 [00:01<00:04, 29.41batch/s]

Training batches on cuda:  24%|██▎       | 40/170 [00:01<00:04, 29.92batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:01<00:04, 30.29batch/s]

Training batches on cuda:  28%|██▊       | 48/170 [00:01<00:03, 30.52batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:01<00:03, 30.79batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:01<00:03, 30.88batch/s]

Training batches on cuda:  35%|███▌      | 60/170 [00:02<00:03, 30.96batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:02<00:03, 31.00batch/s]

Training batches on cuda:  40%|████      | 68/170 [00:02<00:03, 30.88batch/s]

Training batches on cuda:  42%|████▏     | 72/170 [00:02<00:03, 30.93batch/s]

Training batches on cuda:  45%|████▍     | 76/170 [00:02<00:03, 31.00batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:02<00:02, 30.99batch/s]

Training batches on cuda:  49%|████▉     | 84/170 [00:02<00:02, 30.95batch/s]

Training batches on cuda:  52%|█████▏    | 88/170 [00:02<00:02, 31.00batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:03<00:02, 30.47batch/s]

Training batches on cuda:  56%|█████▋    | 96/170 [00:03<00:02, 29.11batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:03<00:02, 28.89batch/s]

Training batches on cuda:  60%|██████    | 102/170 [00:03<00:02, 29.06batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:03<00:02, 29.07batch/s]

Training batches on cuda:  64%|██████▎   | 108/170 [00:03<00:02, 29.15batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:03<00:02, 28.90batch/s]

Training batches on cuda:  67%|██████▋   | 114/170 [00:03<00:01, 28.77batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:03<00:01, 28.97batch/s]

Training batches on cuda:  71%|███████   | 120/170 [00:04<00:01, 29.11batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:04<00:01, 28.92batch/s]

Training batches on cuda:  74%|███████▍  | 126/170 [00:04<00:01, 28.94batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:04<00:01, 28.28batch/s]

Training batches on cuda:  78%|███████▊  | 132/170 [00:04<00:01, 27.99batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:04<00:01, 27.68batch/s]

Training batches on cuda:  81%|████████  | 138/170 [00:04<00:01, 26.94batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:04<00:01, 27.72batch/s]

Training batches on cuda:  85%|████████▍ | 144/170 [00:04<00:00, 28.32batch/s]

Training batches on cuda:  86%|████████▋ | 147/170 [00:05<00:00, 28.53batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:05<00:00, 27.92batch/s]

Training batches on cuda:  90%|█████████ | 153/170 [00:05<00:00, 28.34batch/s]

Training batches on cuda:  92%|█████████▏| 156/170 [00:05<00:00, 28.52batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:05<00:00, 28.72batch/s]

Training batches on cuda:  95%|█████████▌| 162/170 [00:05<00:00, 29.04batch/s]

Training batches on cuda:  97%|█████████▋| 165/170 [00:05<00:00, 29.16batch/s]

Training batches on cuda:  99%|█████████▉| 168/170 [00:05<00:00, 29.08batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:31<00:38,  6.36s/epoch, loss=0.000718, prev_loss=0.000719]
Training epochs on cuda:  50%|█████     | 5/10 [00:31<00:31,  6.26s/epoch, loss=0.000718, prev_loss=0.000719]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/170 [00:00<00:18,  9.12batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:14, 11.28batch/s]

Training batches on cuda:   4%|▍         | 7/170 [00:00<00:11, 13.63batch/s]

Training batches on cuda:   6%|▌         | 10/170 [00:00<00:10, 15.84batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:08, 18.19batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:07, 20.20batch/s]

Training batches on cuda:  11%|█         | 19/170 [00:00<00:06, 22.12batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:06, 22.96batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:01<00:05, 24.24batch/s]

Training batches on cuda:  16%|█▋        | 28/170 [00:01<00:05, 24.54batch/s]

Training batches on cuda:  18%|█▊        | 31/170 [00:01<00:05, 24.25batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:01<00:05, 25.08batch/s]

Training batches on cuda:  22%|██▏       | 37/170 [00:01<00:05, 25.96batch/s]

Training batches on cuda:  24%|██▎       | 40/170 [00:01<00:04, 26.60batch/s]

Training batches on cuda:  25%|██▌       | 43/170 [00:01<00:04, 26.20batch/s]

Training batches on cuda:  27%|██▋       | 46/170 [00:01<00:04, 26.35batch/s]

Training batches on cuda:  29%|██▉       | 49/170 [00:01<00:04, 25.84batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:02<00:04, 26.38batch/s]

Training batches on cuda:  32%|███▏      | 55/170 [00:02<00:04, 26.54batch/s]

Training batches on cuda:  34%|███▍      | 58/170 [00:02<00:04, 26.55batch/s]

Training batches on cuda:  36%|███▌      | 61/170 [00:02<00:04, 27.21batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:02<00:03, 27.46batch/s]

Training batches on cuda:  39%|███▉      | 67/170 [00:02<00:03, 27.32batch/s]

Training batches on cuda:  41%|████      | 70/170 [00:02<00:03, 27.45batch/s]

Training batches on cuda:  43%|████▎     | 73/170 [00:02<00:03, 27.67batch/s]

Training batches on cuda:  45%|████▍     | 76/170 [00:02<00:03, 28.00batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:03<00:03, 28.63batch/s]

Training batches on cuda:  49%|████▉     | 83/170 [00:03<00:03, 28.17batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:03<00:02, 28.16batch/s]

Training batches on cuda:  52%|█████▏    | 89/170 [00:03<00:02, 28.13batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:03<00:02, 27.39batch/s]

Training batches on cuda:  56%|█████▌    | 95/170 [00:03<00:02, 27.87batch/s]

Training batches on cuda:  58%|█████▊    | 98/170 [00:03<00:02, 27.77batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:03<00:02, 28.01batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:03<00:02, 28.29batch/s]

Training batches on cuda:  63%|██████▎   | 107/170 [00:03<00:02, 28.53batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:04<00:02, 28.24batch/s]

Training batches on cuda:  66%|██████▋   | 113/170 [00:04<00:02, 27.63batch/s]

Training batches on cuda:  68%|██████▊   | 116/170 [00:04<00:01, 27.35batch/s]

Training batches on cuda:  70%|███████   | 119/170 [00:04<00:01, 27.86batch/s]

Training batches on cuda:  72%|███████▏  | 122/170 [00:04<00:01, 28.29batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:04<00:01, 28.41batch/s]

Training batches on cuda:  75%|███████▌  | 128/170 [00:04<00:01, 28.64batch/s]

Training batches on cuda:  77%|███████▋  | 131/170 [00:04<00:01, 28.57batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:04<00:01, 28.54batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:05<00:01, 28.56batch/s]

Training batches on cuda:  82%|████████▏ | 140/170 [00:05<00:01, 28.68batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:05<00:00, 28.31batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:05<00:00, 28.42batch/s]

Training batches on cuda:  88%|████████▊ | 149/170 [00:05<00:00, 28.23batch/s]

Training batches on cuda:  89%|████████▉ | 152/170 [00:05<00:00, 28.47batch/s]

Training batches on cuda:  91%|█████████ | 155/170 [00:05<00:00, 28.08batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:05<00:00, 27.92batch/s]

Training batches on cuda:  95%|█████████▍| 161/170 [00:05<00:00, 27.89batch/s]

Training batches on cuda:  96%|█████████▋| 164/170 [00:06<00:00, 28.37batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:06<00:00, 28.59batch/s]

Training batches on cuda: 100%|██████████| 170/170 [00:06<00:00, 28.92batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:38<00:31,  6.26s/epoch, loss=0.000712, prev_loss=0.000718]
Training epochs on cuda:  60%|██████    | 6/10 [00:38<00:25,  6.36s/epoch, loss=0.000712, prev_loss=0.000718]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/170 [00:00<00:10, 16.74batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:08, 19.03batch/s]

Training batches on cuda:   5%|▍         | 8/170 [00:00<00:07, 20.71batch/s]

Training batches on cuda:   6%|▋         | 11/170 [00:00<00:07, 22.48batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:06, 23.81batch/s]

Training batches on cuda:  10%|█         | 17/170 [00:00<00:06, 25.33batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:05, 26.19batch/s]

Training batches on cuda:  14%|█▎        | 23/170 [00:00<00:05, 26.89batch/s]

Training batches on cuda:  15%|█▌        | 26/170 [00:00<00:05, 26.68batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:01<00:05, 25.88batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:01<00:05, 26.37batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:01<00:05, 26.67batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:01<00:04, 27.33batch/s]

Training batches on cuda:  24%|██▍       | 41/170 [00:01<00:04, 27.75batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:01<00:04, 27.97batch/s]

Training batches on cuda:  28%|██▊       | 47/170 [00:01<00:04, 28.25batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:01<00:04, 28.60batch/s]

Training batches on cuda:  31%|███       | 53/170 [00:01<00:04, 28.60batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:02<00:03, 28.63batch/s]

Training batches on cuda:  35%|███▍      | 59/170 [00:02<00:03, 28.69batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:02<00:03, 28.35batch/s]

Training batches on cuda:  38%|███▊      | 65/170 [00:02<00:03, 28.27batch/s]

Training batches on cuda:  40%|████      | 68/170 [00:02<00:03, 28.29batch/s]

Training batches on cuda:  42%|████▏     | 71/170 [00:02<00:03, 28.56batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:02<00:03, 28.63batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:02<00:03, 28.77batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:02<00:03, 28.74batch/s]

Training batches on cuda:  49%|████▉     | 83/170 [00:03<00:03, 27.34batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:03<00:03, 27.09batch/s]

Training batches on cuda:  52%|█████▏    | 89/170 [00:03<00:02, 27.63batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:03<00:02, 28.19batch/s]

Training batches on cuda:  56%|█████▌    | 95/170 [00:03<00:02, 28.29batch/s]

Training batches on cuda:  58%|█████▊    | 98/170 [00:03<00:02, 28.48batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:03<00:02, 27.99batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:03<00:02, 27.98batch/s]

Training batches on cuda:  63%|██████▎   | 107/170 [00:03<00:02, 28.12batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:03<00:02, 28.21batch/s]

Training batches on cuda:  66%|██████▋   | 113/170 [00:04<00:02, 28.49batch/s]

Training batches on cuda:  68%|██████▊   | 116/170 [00:04<00:01, 28.43batch/s]

Training batches on cuda:  70%|███████   | 119/170 [00:04<00:01, 28.02batch/s]

Training batches on cuda:  72%|███████▏  | 122/170 [00:04<00:01, 27.60batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:04<00:01, 27.95batch/s]

Training batches on cuda:  75%|███████▌  | 128/170 [00:04<00:01, 28.43batch/s]

Training batches on cuda:  77%|███████▋  | 131/170 [00:04<00:01, 28.26batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:04<00:01, 28.69batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:04<00:01, 28.64batch/s]

Training batches on cuda:  82%|████████▏ | 140/170 [00:05<00:01, 28.87batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:05<00:00, 28.97batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:05<00:00, 29.08batch/s]

Training batches on cuda:  88%|████████▊ | 149/170 [00:05<00:00, 27.95batch/s]

Training batches on cuda:  89%|████████▉ | 152/170 [00:05<00:00, 28.42batch/s]

Training batches on cuda:  91%|█████████ | 155/170 [00:05<00:00, 28.52batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:05<00:00, 28.36batch/s]

Training batches on cuda:  95%|█████████▍| 161/170 [00:05<00:00, 28.57batch/s]

Training batches on cuda:  96%|█████████▋| 164/170 [00:05<00:00, 28.29batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:05<00:00, 28.58batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:44<00:25,  6.36s/epoch, loss=0.000691, prev_loss=0.000712]
Training epochs on cuda:  70%|███████   | 7/10 [00:44<00:19,  6.34s/epoch, loss=0.000691, prev_loss=0.000712]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/170 [00:00<00:10, 16.31batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:08, 18.76batch/s]

Training batches on cuda:   5%|▍         | 8/170 [00:00<00:07, 20.84batch/s]

Training batches on cuda:   6%|▋         | 11/170 [00:00<00:06, 22.77batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:06, 24.28batch/s]

Training batches on cuda:  10%|█         | 17/170 [00:00<00:05, 25.50batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:05, 26.31batch/s]

Training batches on cuda:  14%|█▎        | 23/170 [00:00<00:05, 26.79batch/s]

Training batches on cuda:  15%|█▌        | 26/170 [00:00<00:05, 26.93batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:01<00:05, 27.16batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:01<00:04, 27.66batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:01<00:04, 28.07batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:01<00:04, 28.08batch/s]

Training batches on cuda:  24%|██▍       | 41/170 [00:01<00:04, 28.09batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:01<00:04, 28.23batch/s]

Training batches on cuda:  28%|██▊       | 47/170 [00:01<00:04, 25.72batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:01<00:04, 26.30batch/s]

Training batches on cuda:  31%|███       | 53/170 [00:01<00:04, 26.43batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:02<00:04, 27.37batch/s]

Training batches on cuda:  35%|███▍      | 59/170 [00:02<00:04, 27.72batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:02<00:03, 28.07batch/s]

Training batches on cuda:  38%|███▊      | 65/170 [00:02<00:03, 27.95batch/s]

Training batches on cuda:  40%|████      | 68/170 [00:02<00:03, 28.08batch/s]

Training batches on cuda:  42%|████▏     | 71/170 [00:02<00:03, 27.90batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:02<00:03, 28.04batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:02<00:03, 28.44batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:02<00:03, 28.59batch/s]

Training batches on cuda:  49%|████▉     | 83/170 [00:03<00:03, 27.35batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:03<00:03, 27.40batch/s]

Training batches on cuda:  52%|█████▏    | 89/170 [00:03<00:02, 27.84batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:03<00:02, 28.16batch/s]

Training batches on cuda:  56%|█████▌    | 95/170 [00:03<00:02, 28.62batch/s]

Training batches on cuda:  58%|█████▊    | 98/170 [00:03<00:02, 28.34batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:03<00:02, 28.33batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:03<00:02, 28.29batch/s]

Training batches on cuda:  63%|██████▎   | 107/170 [00:03<00:02, 28.31batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:03<00:02, 28.52batch/s]

Training batches on cuda:  66%|██████▋   | 113/170 [00:04<00:02, 28.29batch/s]

Training batches on cuda:  68%|██████▊   | 116/170 [00:04<00:01, 27.61batch/s]

Training batches on cuda:  70%|███████   | 119/170 [00:04<00:01, 27.31batch/s]

Training batches on cuda:  72%|███████▏  | 122/170 [00:04<00:01, 26.92batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:04<00:01, 27.48batch/s]

Training batches on cuda:  75%|███████▌  | 128/170 [00:04<00:01, 27.94batch/s]

Training batches on cuda:  77%|███████▋  | 131/170 [00:04<00:01, 28.17batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:04<00:01, 28.24batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:04<00:01, 28.46batch/s]

Training batches on cuda:  82%|████████▏ | 140/170 [00:05<00:01, 28.72batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:05<00:00, 28.75batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:05<00:00, 28.25batch/s]

Training batches on cuda:  88%|████████▊ | 149/170 [00:05<00:00, 28.42batch/s]

Training batches on cuda:  89%|████████▉ | 152/170 [00:05<00:00, 28.19batch/s]

Training batches on cuda:  91%|█████████ | 155/170 [00:05<00:00, 28.53batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:05<00:00, 28.84batch/s]

Training batches on cuda:  95%|█████████▍| 161/170 [00:05<00:00, 27.90batch/s]

Training batches on cuda:  96%|█████████▋| 164/170 [00:05<00:00, 27.98batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:06<00:00, 27.64batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:50<00:19,  6.34s/epoch, loss=0.000619, prev_loss=0.000691]
Training epochs on cuda:  80%|████████  | 8/10 [00:50<00:12,  6.36s/epoch, loss=0.000619, prev_loss=0.000691]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/170 [00:00<00:10, 15.42batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:09, 18.10batch/s]

Training batches on cuda:   6%|▌         | 10/170 [00:00<00:07, 20.72batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:06, 22.89batch/s]

Training batches on cuda:  11%|█         | 18/170 [00:00<00:06, 24.80batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:05, 26.42batch/s]

Training batches on cuda:  15%|█▌        | 26/170 [00:00<00:05, 27.51batch/s]

Training batches on cuda:  18%|█▊        | 30/170 [00:01<00:04, 28.60batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:01<00:04, 29.41batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:01<00:04, 29.75batch/s]

Training batches on cuda:  25%|██▍       | 42/170 [00:01<00:04, 30.12batch/s]

Training batches on cuda:  27%|██▋       | 46/170 [00:01<00:04, 30.33batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:01<00:03, 30.48batch/s]

Training batches on cuda:  32%|███▏      | 54/170 [00:01<00:03, 30.81batch/s]

Training batches on cuda:  34%|███▍      | 58/170 [00:01<00:03, 30.99batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:02<00:03, 30.89batch/s]

Training batches on cuda:  39%|███▉      | 66/170 [00:02<00:03, 31.11batch/s]

Training batches on cuda:  41%|████      | 70/170 [00:02<00:03, 31.26batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:02<00:03, 30.73batch/s]

Training batches on cuda:  46%|████▌     | 78/170 [00:02<00:02, 30.87batch/s]

Training batches on cuda:  48%|████▊     | 82/170 [00:02<00:02, 30.94batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:02<00:02, 30.98batch/s]

Training batches on cuda:  53%|█████▎    | 90/170 [00:02<00:02, 31.12batch/s]

Training batches on cuda:  55%|█████▌    | 94/170 [00:03<00:02, 31.17batch/s]

Training batches on cuda:  58%|█████▊    | 98/170 [00:03<00:02, 31.29batch/s]

Training batches on cuda:  60%|██████    | 102/170 [00:03<00:02, 31.34batch/s]

Training batches on cuda:  62%|██████▏   | 106/170 [00:03<00:02, 31.28batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:03<00:01, 31.25batch/s]

Training batches on cuda:  67%|██████▋   | 114/170 [00:03<00:01, 31.26batch/s]

Training batches on cuda:  69%|██████▉   | 118/170 [00:03<00:01, 30.07batch/s]

Training batches on cuda:  72%|███████▏  | 122/170 [00:04<00:01, 29.37batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:04<00:01, 29.04batch/s]

Training batches on cuda:  75%|███████▌  | 128/170 [00:04<00:01, 28.43batch/s]

Training batches on cuda:  77%|███████▋  | 131/170 [00:04<00:01, 28.13batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:04<00:01, 27.90batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:04<00:01, 27.77batch/s]

Training batches on cuda:  82%|████████▏ | 140/170 [00:04<00:01, 27.65batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:04<00:00, 27.49batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:04<00:00, 27.43batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:05<00:00, 28.03batch/s]

Training batches on cuda:  90%|█████████ | 153/170 [00:05<00:00, 27.67batch/s]

Training batches on cuda:  92%|█████████▏| 156/170 [00:05<00:00, 27.56batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:05<00:00, 26.87batch/s]

Training batches on cuda:  95%|█████████▌| 162/170 [00:05<00:00, 27.40batch/s]

Training batches on cuda:  97%|█████████▋| 165/170 [00:05<00:00, 27.77batch/s]

Training batches on cuda:  99%|█████████▉| 168/170 [00:05<00:00, 27.74batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:56<00:12,  6.36s/epoch, loss=0.000554, prev_loss=0.000619]
Training epochs on cuda:  90%|█████████ | 9/10 [00:56<00:06,  6.28s/epoch, loss=0.000554, prev_loss=0.000619]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/170 [00:00<00:10, 16.30batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:08, 18.35batch/s]

Training batches on cuda:   5%|▍         | 8/170 [00:00<00:07, 20.56batch/s]

Training batches on cuda:   6%|▋         | 11/170 [00:00<00:07, 22.45batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:06, 23.89batch/s]

Training batches on cuda:  10%|█         | 17/170 [00:00<00:06, 24.74batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:05, 25.33batch/s]

Training batches on cuda:  14%|█▎        | 23/170 [00:00<00:05, 26.15batch/s]

Training batches on cuda:  15%|█▌        | 26/170 [00:00<00:05, 26.43batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:01<00:05, 26.91batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:01<00:05, 27.20batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:01<00:05, 26.98batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:01<00:04, 27.13batch/s]

Training batches on cuda:  24%|██▍       | 41/170 [00:01<00:04, 26.41batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:01<00:04, 26.68batch/s]

Training batches on cuda:  28%|██▊       | 47/170 [00:01<00:04, 26.72batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:01<00:04, 26.92batch/s]

Training batches on cuda:  31%|███       | 53/170 [00:01<00:04, 27.56batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:02<00:04, 27.53batch/s]

Training batches on cuda:  35%|███▍      | 59/170 [00:02<00:04, 27.29batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:02<00:03, 27.60batch/s]

Training batches on cuda:  38%|███▊      | 65/170 [00:02<00:03, 28.14batch/s]

Training batches on cuda:  40%|████      | 68/170 [00:02<00:03, 28.37batch/s]

Training batches on cuda:  42%|████▏     | 71/170 [00:02<00:03, 28.28batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:02<00:03, 28.73batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:02<00:03, 28.62batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:02<00:03, 28.71batch/s]

Training batches on cuda:  49%|████▉     | 83/170 [00:03<00:03, 28.41batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:03<00:02, 28.15batch/s]

Training batches on cuda:  52%|█████▏    | 89/170 [00:03<00:02, 27.97batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:03<00:02, 27.25batch/s]

Training batches on cuda:  56%|█████▌    | 95/170 [00:03<00:02, 26.43batch/s]

Training batches on cuda:  58%|█████▊    | 98/170 [00:03<00:02, 26.39batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:03<00:02, 26.88batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:03<00:02, 27.03batch/s]

Training batches on cuda:  63%|██████▎   | 107/170 [00:03<00:02, 27.20batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:04<00:02, 27.40batch/s]

Training batches on cuda:  66%|██████▋   | 113/170 [00:04<00:02, 27.34batch/s]

Training batches on cuda:  68%|██████▊   | 116/170 [00:04<00:01, 27.26batch/s]

Training batches on cuda:  70%|███████   | 119/170 [00:04<00:01, 27.41batch/s]

Training batches on cuda:  72%|███████▏  | 122/170 [00:04<00:01, 27.83batch/s]

Training batches on cuda:  74%|███████▎  | 125/170 [00:04<00:01, 27.97batch/s]

Training batches on cuda:  75%|███████▌  | 128/170 [00:04<00:01, 28.40batch/s]

Training batches on cuda:  77%|███████▋  | 131/170 [00:04<00:01, 28.69batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:04<00:01, 28.85batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:04<00:01, 28.98batch/s]

Training batches on cuda:  82%|████████▏ | 140/170 [00:05<00:01, 29.00batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:05<00:00, 29.11batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:05<00:00, 29.16batch/s]

Training batches on cuda:  88%|████████▊ | 149/170 [00:05<00:00, 29.21batch/s]

Training batches on cuda:  89%|████████▉ | 152/170 [00:05<00:00, 28.77batch/s]

Training batches on cuda:  91%|█████████ | 155/170 [00:05<00:00, 28.83batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:05<00:00, 28.82batch/s]

Training batches on cuda:  95%|█████████▍| 161/170 [00:05<00:00, 28.84batch/s]

Training batches on cuda:  96%|█████████▋| 164/170 [00:05<00:00, 28.56batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:06<00:00, 28.59batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:03<00:06,  6.28s/epoch, loss=0.000504, prev_loss=0.000554]2021-09-16:18:45:21,326 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:45:22,370 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:45:37,473 INFO     [evaluator.py:657] Evaluation took 15.10s seconds
2021-09-16:18:45:37,615 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00, 11.21s/epoch, loss=0.000504, prev_loss=0.000554]
Training epochs on cuda: 100%|██████████| 10/10 [01:19<00:00,  7.95s/epoch, loss=0.000504, prev_loss=0.000554]
2021-09-16:18:45:37,616 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp12yfdry2'
2021-09-16:18:45:37,727 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp12yfdry2' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.35ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.62ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:03, 2.55ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:12<00:02, 2.57ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.58ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.60ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.60ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.61ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.62ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.61ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.61ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.62ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:45:52,708 INFO     [evaluator.py:657] Evaluation took 14.95s seconds
[I 2021-09-16 18:45:52,715] Trial 56 finished with value: 0.8806886545926426 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.048492375399818675, 'optimizer.lr': 0.02820820030149787, 'negative_sampler.num_negs_per_pos': 49, 'training.batch_size': 1832}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:45:52,741 WARNING  [api.py:823] No random seed is specified. Setting to 3897284946.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:17,  7.00batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:14,  8.31batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:12,  9.48batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:10, 10.60batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:09, 11.57batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:09, 12.30batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 12.91batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:08, 13.39batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 13.71batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 14.00batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:07, 14.14batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:06, 14.46batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.46batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 14.43batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:02<00:06, 14.41batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 14.62batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:06, 14.80batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:05, 14.82batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 14.92batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 15.03batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 15.06batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:02<00:05, 15.06batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 15.06batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 15.07batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:04, 15.07batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 15.14batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 15.14batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 14.96batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:03<00:04, 14.90batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 14.77batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 14.89batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.87batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:03, 14.68batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.87batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.85batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 14.64batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:05<00:03, 14.58batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 14.45batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:03, 14.19batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:03, 14.19batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 14.15batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 14.15batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 14.25batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:06<00:02, 14.29batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:06<00:02, 14.36batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 14.41batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.52batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.62batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 14.54batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.75batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 14.67batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:07<00:01, 14.57batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 14.48batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.26batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 14.34batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 14.35batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 14.17batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 14.43batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:08<00:00, 14.43batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 14.44batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 14.60batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 15.38batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:08<?, ?epoch/s, loss=0.000848, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:08<01:18,  8.76s/epoch, loss=0.000848, prev_loss=nan]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:17,  6.82batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:14,  8.14batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:12,  9.40batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:10, 10.56batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:09, 11.57batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:09, 12.43batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 13.11batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:08, 13.27batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 13.45batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 13.84batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:07, 14.01batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:07, 14.21batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.49batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 14.73batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:02<00:06, 14.86batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 14.82batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:06, 14.73batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:05, 14.87batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 14.94batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 14.80batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 14.78batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:02<00:05, 14.61batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 14.49batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 14.29batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:05, 14.32batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:05, 13.09batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:05, 13.52batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 13.79batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:04<00:04, 13.86batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 14.14batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 14.06batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.30batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:04, 14.26batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.07batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.28batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 14.54batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:05<00:03, 14.49batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 14.64batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:03, 14.75batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 14.74batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 14.62batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 14.65batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 14.61batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:06<00:02, 14.72batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:06<00:02, 14.86batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 14.80batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.64batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.78batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 14.68batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.53batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:07<00:01, 14.35batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:07<00:01, 14.34batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 13.98batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.01batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:01, 13.96batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 13.98batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 14.01batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:08<00:00, 14.04batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:08<00:00, 14.03batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 13.67batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 13.62batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 14.52batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:17<01:18,  8.76s/epoch, loss=0.000843, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:17<01:10,  8.83s/epoch, loss=0.000843, prev_loss=nan]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:19,  6.11batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:16,  7.47batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:13,  8.81batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:11, 10.14batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:10, 11.33batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:09, 12.34batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 13.16batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:07, 13.81batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 14.30batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 14.68batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:06, 14.95batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:06, 15.14batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 15.27batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 15.35batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:06, 15.41batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:05, 15.49batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:05, 15.52batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:05, 15.52batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 15.52batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 15.51batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 15.53batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:02<00:05, 15.56batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:02<00:05, 15.58batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:04, 15.60batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:04, 15.62batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 15.62batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 15.59batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 15.54batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:03<00:04, 15.36batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:03<00:04, 15.21batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 15.09batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.80batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:03, 14.86batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.65batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.64batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 14.70batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:04<00:03, 14.61batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:04<00:03, 14.64batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:03, 14.74batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 14.70batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 14.62batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 14.83batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 14.66batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:05<00:02, 14.34batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:05<00:02, 14.37batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 14.57batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.56batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.62batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 14.41batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.53batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 14.45batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:06<00:01, 14.32batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 14.17batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.37batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 14.40batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 14.47batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 14.57batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 14.61batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:07<00:00, 14.80batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 14.84batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 14.79batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 15.71batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:26<01:10,  8.83s/epoch, loss=0.000843, prev_loss=0.000843]
Training epochs on cuda:  30%|███       | 3/10 [00:26<01:01,  8.74s/epoch, loss=0.000843, prev_loss=0.000843]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:17,  7.06batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:14,  8.43batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:12,  9.63batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:10, 10.70batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:09, 11.69batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:08, 12.48batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 13.07batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:08, 12.99batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 13.38batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 13.84batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:07, 13.98batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:07, 14.19batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.30batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 14.50batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:02<00:06, 14.63batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 14.49batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:06, 14.66batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:05, 14.84batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 14.92batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 14.63batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 14.46batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:03<00:05, 14.60batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 14.60batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 14.84batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:04, 14.83batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 14.87batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 14.86batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 14.86batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:03<00:04, 14.88batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 15.09batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 15.24batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:03, 15.34batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:03, 15.41batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 15.47batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 15.50batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 15.54batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:04<00:03, 15.57batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 15.58batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:02, 15.55batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 15.57batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 15.58batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 15.51batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 15.54batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:05<00:02, 15.56batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:06<00:02, 15.56batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 14.56batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.86batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.81batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 15.03batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.89batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 14.93batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:06<00:01, 14.77batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 14.69batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.72batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 14.29batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 14.19batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 14.34batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 14.60batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:07<00:00, 14.59batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 14.44batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 14.41batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 15.18batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:34<01:01,  8.74s/epoch, loss=0.000843, prev_loss=0.000843]
Training epochs on cuda:  40%|████      | 4/10 [00:34<00:52,  8.73s/epoch, loss=0.000843, prev_loss=0.000843]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:17,  7.03batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:14,  8.29batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:12,  9.39batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:11, 10.40batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:10, 11.28batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:09, 12.17batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 12.77batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:08, 13.14batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 13.68batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 14.00batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:07, 14.28batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:06, 14.54batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.44batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 14.38batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:02<00:06, 14.42batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 14.53batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:06, 14.47batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:06, 14.61batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 14.54batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 14.47batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 14.53batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:03<00:05, 14.76batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 14.79batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 14.88batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:04, 14.87batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 14.84batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 14.87batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 14.86batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:03<00:04, 14.88batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 14.95batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 15.06batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.82batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:03, 14.62batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.62batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.80batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 14.95batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:05<00:03, 14.99batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 15.05batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:03, 15.04batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 15.11batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 15.16batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 15.19batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 15.23batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:05<00:02, 15.15batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:06<00:02, 15.09batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 15.03batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.99batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.94batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 14.86batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.72batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 14.76batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:07<00:01, 14.71batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 14.69batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.65batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 14.62batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 14.65batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 14.63batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 14.48batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:08<00:00, 14.47batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 14.58batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 14.58batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 15.49batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:43<00:52,  8.73s/epoch, loss=0.000843, prev_loss=0.000843]
Training epochs on cuda:  50%|█████     | 5/10 [00:43<00:43,  8.70s/epoch, loss=0.000843, prev_loss=0.000843]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:14,  8.14batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:13,  9.22batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:11, 10.27batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:10, 11.28batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:09, 11.80batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:08, 12.51batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 12.99batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:07, 13.52batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 13.89batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 14.22batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:07, 14.36batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:06, 14.61batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.81batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 14.96batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:02<00:06, 14.85batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 14.42batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:06, 14.26batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:06, 14.19batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:06, 14.07batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 14.27batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 14.41batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:03<00:05, 14.51batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 14.48batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 14.60batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:05, 14.65batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 14.61batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 14.69batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 14.07batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:04<00:04, 13.90batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 14.38batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 14.50batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.58batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:03, 14.83batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.61batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.80batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 14.84batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:05<00:03, 14.89batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 15.00batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:03, 14.99batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 14.96batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 14.78batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 14.67batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 14.61batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:06<00:02, 14.52batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:06<00:02, 14.63batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 14.48batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.47batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.42batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 14.44batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.65batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 14.83batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:07<00:01, 14.80batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 14.68batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.72batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 14.72batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 14.65batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 13.91batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 14.16batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:08<00:00, 14.36batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 14.53batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 14.66batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 15.55batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:52<00:43,  8.70s/epoch, loss=0.000843, prev_loss=0.000843]
Training epochs on cuda:  60%|██████    | 6/10 [00:52<00:34,  8.70s/epoch, loss=0.000843, prev_loss=0.000843]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:19,  6.19batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:15,  7.51batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:13,  8.80batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:11, 10.02batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:10, 11.11batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:09, 12.00batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 12.77batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:08, 13.32batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 13.77batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 14.03batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:07, 14.15batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:07, 14.26batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.40batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 14.32batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:02<00:06, 14.38batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 14.46batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:06, 14.52batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:06, 14.66batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 14.79batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 14.80batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 14.85batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:03<00:05, 14.81batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 14.86batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 14.90batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:04, 14.99batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 14.99batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 14.93batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 14.92batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:03<00:04, 14.86batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 14.86batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 14.66batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.65batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:04, 14.46batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.60batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.69batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 14.80batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:05<00:03, 14.88batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 14.91batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:03, 14.95batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 14.98batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 14.98batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 15.00batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 15.00batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:05<00:02, 15.00batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:06<00:02, 14.91batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 14.94batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.95batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.91batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 14.93batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.91batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 14.79batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:07<00:01, 14.85batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 14.85batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.91batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 14.93batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 14.94batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 14.69batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 14.77batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:07<00:00, 14.84batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 14.89batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 14.94batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 15.78batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [01:00<00:34,  8.70s/epoch, loss=0.000843, prev_loss=0.000843]
Training epochs on cuda:  70%|███████   | 7/10 [01:00<00:26,  8.69s/epoch, loss=0.000843, prev_loss=0.000843]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:14,  8.55batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:12,  9.81batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:10, 10.83batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:09, 11.85batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:09, 12.52batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:08, 13.26batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 13.71batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:07, 14.06batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 14.38batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 14.59batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:06, 14.75batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:06, 14.89batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.90batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 15.02batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:06, 15.05batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 15.10batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:05, 15.03batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:05, 15.05batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 14.82batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 14.70batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 14.71batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:02<00:05, 14.75batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 14.75batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 14.80batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:05, 14.80batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 14.72batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 14.61batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 14.68batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:03<00:04, 14.55batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 14.64batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 14.73batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.68batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:03, 14.71batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.72batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.98batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 15.13batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:04<00:03, 15.27batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 15.39batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:02, 15.37batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 15.44batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 15.46batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 15.48batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 15.50batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:05<00:02, 15.38batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:05<00:02, 15.37batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 15.41batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:01, 15.49batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 15.54batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 15.54batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 15.53batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 15.52batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:06<00:01, 15.50batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 15.50batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 15.54batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 15.56batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 15.58batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 15.58batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 15.58batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:07<00:00, 15.59batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:07<00:00, 15.57batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 15.56batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 16.41batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [01:09<00:26,  8.69s/epoch, loss=0.000843, prev_loss=0.000843]
Training epochs on cuda:  80%|████████  | 8/10 [01:09<00:17,  8.64s/epoch, loss=0.000843, prev_loss=0.000843]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:13,  9.07batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:11, 10.25batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:10, 11.21batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:09, 12.09batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:08, 12.77batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:08, 13.29batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 13.74batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:07, 14.08batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 14.29batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 14.33batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:07, 14.21batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:06, 14.43batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.49batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 14.60batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:02<00:06, 14.59batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 14.74batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:06, 14.75batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:05, 14.73batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 14.75batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 14.71batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 14.74batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:02<00:05, 14.78batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 14.84batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 14.64batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:05, 14.63batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 14.66batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 14.65batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 14.62batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:03<00:04, 14.68batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 14.69batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 14.57batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.60batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:03, 14.52batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.48batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.63batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 14.60batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:05<00:03, 14.49batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 14.62batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:03, 14.60batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 14.68batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 14.73batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 14.75batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 14.69batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:05<00:02, 14.86batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:06<00:02, 14.85batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 14.75batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.40batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.32batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 14.23batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.18batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 14.19batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:07<00:01, 14.15batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 14.06batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.10batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 14.15batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 14.25batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 14.40batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 14.59batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:08<00:00, 14.71batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 14.72batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 14.80batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 15.59batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:18<00:17,  8.64s/epoch, loss=0.000833, prev_loss=0.000843]
Training epochs on cuda:  90%|█████████ | 9/10 [01:18<00:08,  8.68s/epoch, loss=0.000833, prev_loss=0.000843]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:17,  6.92batch/s]

Training batches on cuda:   2%|▏         | 3/123 [00:00<00:14,  8.24batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:12,  9.36batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:11, 10.55batch/s]

Training batches on cuda:   7%|▋         | 9/123 [00:00<00:09, 11.52batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:08, 12.45batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:08, 13.12batch/s]

Training batches on cuda:  12%|█▏        | 15/123 [00:01<00:07, 13.61batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:01<00:07, 13.97batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:01<00:07, 14.16batch/s]

Training batches on cuda:  17%|█▋        | 21/123 [00:01<00:07, 14.30batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:01<00:06, 14.36batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:06, 14.50batch/s]

Training batches on cuda:  22%|██▏       | 27/123 [00:01<00:06, 14.49batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:02<00:06, 14.58batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:02<00:06, 14.62batch/s]

Training batches on cuda:  27%|██▋       | 33/123 [00:02<00:06, 14.55batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:02<00:06, 14.63batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:02<00:05, 14.73batch/s]

Training batches on cuda:  32%|███▏      | 39/123 [00:02<00:05, 14.77batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:02<00:05, 14.78batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:02<00:05, 14.90batch/s]

Training batches on cuda:  37%|███▋      | 45/123 [00:03<00:05, 14.92batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:03<00:05, 14.93batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:03<00:04, 14.89batch/s]

Training batches on cuda:  41%|████▏     | 51/123 [00:03<00:04, 14.96batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:03<00:04, 15.12batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:03<00:04, 15.18batch/s]

Training batches on cuda:  46%|████▋     | 57/123 [00:03<00:04, 15.04batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:04<00:04, 14.84batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:04<00:04, 14.79batch/s]

Training batches on cuda:  51%|█████     | 63/123 [00:04<00:04, 14.68batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:04<00:04, 14.44batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:04<00:03, 14.66batch/s]

Training batches on cuda:  56%|█████▌    | 69/123 [00:04<00:03, 14.73batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:04<00:03, 14.49batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:05<00:03, 14.61batch/s]

Training batches on cuda:  61%|██████    | 75/123 [00:05<00:03, 14.61batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:05<00:03, 14.84batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:05<00:02, 14.91batch/s]

Training batches on cuda:  66%|██████▌   | 81/123 [00:05<00:02, 14.72batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:05<00:02, 14.78batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:05<00:02, 14.83batch/s]

Training batches on cuda:  71%|███████   | 87/123 [00:05<00:02, 14.87batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:06<00:02, 14.85batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:06<00:02, 14.89batch/s]

Training batches on cuda:  76%|███████▌  | 93/123 [00:06<00:02, 14.90batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:06<00:01, 14.86batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:06<00:01, 14.78batch/s]

Training batches on cuda:  80%|████████  | 99/123 [00:06<00:01, 14.26batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:06<00:01, 14.04batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:07<00:01, 14.04batch/s]

Training batches on cuda:  85%|████████▌ | 105/123 [00:07<00:01, 14.00batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:07<00:01, 14.15batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:07<00:00, 14.33batch/s]

Training batches on cuda:  90%|█████████ | 111/123 [00:07<00:00, 14.51batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:07<00:00, 14.69batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:07<00:00, 14.77batch/s]

Training batches on cuda:  95%|█████████▌| 117/123 [00:08<00:00, 14.76batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:08<00:00, 14.58batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:08<00:00, 14.39batch/s]

Training batches on cuda: 100%|██████████| 123/123 [00:08<00:00, 15.26batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:26<00:08,  8.68s/epoch, loss=0.000835, prev_loss=0.000833]2021-09-16:18:47:20,514 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:47:21,734 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:47:36,765 INFO     [evaluator.py:657] Evaluation took 15.03s seconds
2021-09-16:18:47:36,937 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 13.61s/epoch, loss=0.000835, prev_loss=0.000833]
Training epochs on cuda: 100%|██████████| 10/10 [01:43<00:00, 10.34s/epoch, loss=0.000835, prev_loss=0.000833]
2021-09-16:18:47:36,938 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp3au78081'
2021-09-16:18:47:36,993 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp3au78081' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.34ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.42ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.62ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.62ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.62ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.61ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.49ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.53ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:08, 2.56ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.58ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.59ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.60ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.61ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.62ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.62ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.62ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.62ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.61ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:47:51,967 INFO     [evaluator.py:657] Evaluation took 14.96s seconds
[I 2021-09-16 18:47:51,972] Trial 57 finished with value: 0.36350532009741754 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.10357401039345003, 'optimizer.lr': 0.022584766434178973, 'negative_sampler.num_negs_per_pos': 62, 'training.batch_size': 2532}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:47:51,996 WARNING  [api.py:823] No random seed is specified. Setting to 526543468.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/302 [00:00<00:15, 19.66batch/s]

Training batches on cuda:   2%|▏         | 7/302 [00:00<00:12, 23.54batch/s]

Training batches on cuda:   4%|▍         | 12/302 [00:00<00:10, 27.00batch/s]

Training batches on cuda:   6%|▌         | 17/302 [00:00<00:09, 30.83batch/s]

Training batches on cuda:   7%|▋         | 22/302 [00:00<00:08, 34.04batch/s]

Training batches on cuda:   9%|▉         | 27/302 [00:00<00:07, 36.85batch/s]

Training batches on cuda:  11%|█         | 32/302 [00:00<00:06, 39.00batch/s]

Training batches on cuda:  12%|█▏        | 37/302 [00:00<00:06, 40.82batch/s]

Training batches on cuda:  14%|█▍        | 42/302 [00:01<00:06, 39.36batch/s]

Training batches on cuda:  16%|█▌        | 47/302 [00:01<00:06, 40.66batch/s]

Training batches on cuda:  17%|█▋        | 52/302 [00:01<00:05, 41.90batch/s]

Training batches on cuda:  19%|█▉        | 57/302 [00:01<00:05, 42.96batch/s]

Training batches on cuda:  21%|██        | 62/302 [00:01<00:05, 43.45batch/s]

Training batches on cuda:  22%|██▏       | 67/302 [00:01<00:05, 43.79batch/s]

Training batches on cuda:  24%|██▍       | 72/302 [00:01<00:05, 43.77batch/s]

Training batches on cuda:  25%|██▌       | 77/302 [00:01<00:05, 44.20batch/s]

Training batches on cuda:  27%|██▋       | 82/302 [00:01<00:04, 44.48batch/s]

Training batches on cuda:  29%|██▉       | 87/302 [00:02<00:04, 44.63batch/s]

Training batches on cuda:  30%|███       | 92/302 [00:02<00:04, 45.07batch/s]

Training batches on cuda:  32%|███▏      | 97/302 [00:02<00:04, 45.30batch/s]

Training batches on cuda:  34%|███▍      | 102/302 [00:02<00:04, 45.19batch/s]

Training batches on cuda:  35%|███▌      | 107/302 [00:02<00:04, 44.97batch/s]

Training batches on cuda:  37%|███▋      | 112/302 [00:02<00:04, 45.06batch/s]

Training batches on cuda:  39%|███▊      | 117/302 [00:02<00:04, 44.43batch/s]

Training batches on cuda:  40%|████      | 122/302 [00:02<00:04, 44.70batch/s]

Training batches on cuda:  42%|████▏     | 127/302 [00:02<00:03, 44.08batch/s]

Training batches on cuda:  44%|████▎     | 132/302 [00:03<00:03, 44.00batch/s]

Training batches on cuda:  45%|████▌     | 137/302 [00:03<00:03, 44.61batch/s]

Training batches on cuda:  47%|████▋     | 142/302 [00:03<00:03, 44.92batch/s]

Training batches on cuda:  49%|████▊     | 147/302 [00:03<00:03, 45.29batch/s]

Training batches on cuda:  50%|█████     | 152/302 [00:03<00:03, 45.04batch/s]

Training batches on cuda:  52%|█████▏    | 157/302 [00:03<00:03, 44.82batch/s]

Training batches on cuda:  54%|█████▎    | 162/302 [00:03<00:03, 45.00batch/s]

Training batches on cuda:  55%|█████▌    | 167/302 [00:03<00:02, 45.30batch/s]

Training batches on cuda:  57%|█████▋    | 172/302 [00:03<00:02, 44.99batch/s]

Training batches on cuda:  59%|█████▊    | 177/302 [00:04<00:02, 45.20batch/s]

Training batches on cuda:  60%|██████    | 182/302 [00:04<00:02, 45.30batch/s]

Training batches on cuda:  62%|██████▏   | 187/302 [00:04<00:02, 45.62batch/s]

Training batches on cuda:  64%|██████▎   | 192/302 [00:04<00:02, 43.75batch/s]

Training batches on cuda:  65%|██████▌   | 197/302 [00:04<00:02, 44.41batch/s]

Training batches on cuda:  67%|██████▋   | 202/302 [00:04<00:02, 45.08batch/s]

Training batches on cuda:  69%|██████▊   | 207/302 [00:04<00:02, 44.89batch/s]

Training batches on cuda:  70%|███████   | 212/302 [00:04<00:01, 45.22batch/s]

Training batches on cuda:  72%|███████▏  | 217/302 [00:04<00:01, 45.12batch/s]

Training batches on cuda:  74%|███████▎  | 222/302 [00:05<00:01, 45.25batch/s]

Training batches on cuda:  75%|███████▌  | 227/302 [00:05<00:01, 44.18batch/s]

Training batches on cuda:  77%|███████▋  | 232/302 [00:05<00:01, 43.54batch/s]

Training batches on cuda:  78%|███████▊  | 237/302 [00:05<00:01, 43.86batch/s]

Training batches on cuda:  80%|████████  | 242/302 [00:05<00:01, 43.95batch/s]

Training batches on cuda:  82%|████████▏ | 247/302 [00:05<00:01, 43.82batch/s]

Training batches on cuda:  83%|████████▎ | 252/302 [00:05<00:01, 44.16batch/s]

Training batches on cuda:  85%|████████▌ | 257/302 [00:05<00:01, 44.24batch/s]

Training batches on cuda:  87%|████████▋ | 262/302 [00:05<00:00, 43.73batch/s]

Training batches on cuda:  88%|████████▊ | 267/302 [00:06<00:00, 41.24batch/s]

Training batches on cuda:  90%|█████████ | 272/302 [00:06<00:00, 41.69batch/s]

Training batches on cuda:  92%|█████████▏| 277/302 [00:06<00:00, 42.21batch/s]

Training batches on cuda:  93%|█████████▎| 282/302 [00:06<00:00, 43.13batch/s]

Training batches on cuda:  95%|█████████▌| 287/302 [00:06<00:00, 43.25batch/s]

Training batches on cuda:  97%|█████████▋| 292/302 [00:06<00:00, 43.29batch/s]

Training batches on cuda:  98%|█████████▊| 297/302 [00:06<00:00, 43.16batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:07<?, ?epoch/s, loss=0.00129, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:07<01:03,  7.09s/epoch, loss=0.00129, prev_loss=nan]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/302 [00:00<00:11, 25.78batch/s]

Training batches on cuda:   3%|▎         | 8/302 [00:00<00:09, 29.49batch/s]

Training batches on cuda:   4%|▍         | 13/302 [00:00<00:08, 32.84batch/s]

Training batches on cuda:   6%|▌         | 18/302 [00:00<00:07, 36.25batch/s]

Training batches on cuda:   8%|▊         | 23/302 [00:00<00:07, 39.04batch/s]

Training batches on cuda:   9%|▉         | 28/302 [00:00<00:06, 41.27batch/s]

Training batches on cuda:  11%|█         | 33/302 [00:00<00:06, 42.44batch/s]

Training batches on cuda:  13%|█▎        | 38/302 [00:00<00:06, 43.75batch/s]

Training batches on cuda:  14%|█▍        | 43/302 [00:00<00:05, 44.75batch/s]

Training batches on cuda:  16%|█▌        | 48/302 [00:01<00:05, 45.50batch/s]

Training batches on cuda:  18%|█▊        | 53/302 [00:01<00:05, 46.04batch/s]

Training batches on cuda:  19%|█▉        | 58/302 [00:01<00:05, 46.43batch/s]

Training batches on cuda:  21%|██        | 63/302 [00:01<00:05, 46.31batch/s]

Training batches on cuda:  23%|██▎       | 68/302 [00:01<00:05, 45.81batch/s]

Training batches on cuda:  24%|██▍       | 73/302 [00:01<00:05, 45.80batch/s]

Training batches on cuda:  26%|██▌       | 78/302 [00:01<00:04, 44.93batch/s]

Training batches on cuda:  27%|██▋       | 83/302 [00:01<00:04, 44.78batch/s]

Training batches on cuda:  29%|██▉       | 88/302 [00:01<00:04, 45.01batch/s]

Training batches on cuda:  31%|███       | 93/302 [00:02<00:04, 45.25batch/s]

Training batches on cuda:  32%|███▏      | 98/302 [00:02<00:04, 45.07batch/s]

Training batches on cuda:  34%|███▍      | 103/302 [00:02<00:04, 45.02batch/s]

Training batches on cuda:  36%|███▌      | 108/302 [00:02<00:04, 45.03batch/s]

Training batches on cuda:  37%|███▋      | 113/302 [00:02<00:04, 44.91batch/s]

Training batches on cuda:  39%|███▉      | 118/302 [00:02<00:04, 44.22batch/s]

Training batches on cuda:  41%|████      | 123/302 [00:02<00:04, 42.88batch/s]

Training batches on cuda:  42%|████▏     | 128/302 [00:02<00:04, 42.48batch/s]

Training batches on cuda:  44%|████▍     | 133/302 [00:02<00:03, 43.25batch/s]

Training batches on cuda:  46%|████▌     | 138/302 [00:03<00:03, 43.44batch/s]

Training batches on cuda:  47%|████▋     | 143/302 [00:03<00:03, 43.58batch/s]

Training batches on cuda:  49%|████▉     | 148/302 [00:03<00:03, 43.57batch/s]

Training batches on cuda:  51%|█████     | 153/302 [00:03<00:03, 43.70batch/s]

Training batches on cuda:  52%|█████▏    | 158/302 [00:03<00:03, 43.92batch/s]

Training batches on cuda:  54%|█████▍    | 163/302 [00:03<00:03, 43.89batch/s]

Training batches on cuda:  56%|█████▌    | 168/302 [00:03<00:03, 43.32batch/s]

Training batches on cuda:  57%|█████▋    | 173/302 [00:03<00:03, 42.75batch/s]

Training batches on cuda:  59%|█████▉    | 178/302 [00:04<00:03, 40.11batch/s]

Training batches on cuda:  61%|██████    | 183/302 [00:04<00:02, 41.24batch/s]

Training batches on cuda:  62%|██████▏   | 188/302 [00:04<00:02, 41.27batch/s]

Training batches on cuda:  64%|██████▍   | 193/302 [00:04<00:02, 42.08batch/s]

Training batches on cuda:  66%|██████▌   | 198/302 [00:04<00:02, 41.97batch/s]

Training batches on cuda:  67%|██████▋   | 203/302 [00:04<00:02, 42.51batch/s]

Training batches on cuda:  69%|██████▉   | 208/302 [00:04<00:02, 41.74batch/s]

Training batches on cuda:  71%|███████   | 213/302 [00:04<00:02, 41.04batch/s]

Training batches on cuda:  72%|███████▏  | 218/302 [00:05<00:02, 41.99batch/s]

Training batches on cuda:  74%|███████▍  | 223/302 [00:05<00:01, 42.68batch/s]

Training batches on cuda:  75%|███████▌  | 228/302 [00:05<00:01, 42.04batch/s]

Training batches on cuda:  77%|███████▋  | 233/302 [00:05<00:01, 41.65batch/s]

Training batches on cuda:  79%|███████▉  | 238/302 [00:05<00:01, 41.82batch/s]

Training batches on cuda:  80%|████████  | 243/302 [00:05<00:01, 42.72batch/s]

Training batches on cuda:  82%|████████▏ | 248/302 [00:05<00:01, 42.26batch/s]

Training batches on cuda:  84%|████████▍ | 253/302 [00:05<00:01, 40.92batch/s]

Training batches on cuda:  85%|████████▌ | 258/302 [00:05<00:01, 40.78batch/s]

Training batches on cuda:  87%|████████▋ | 263/302 [00:06<00:00, 39.89batch/s]

Training batches on cuda:  89%|████████▊ | 268/302 [00:06<00:00, 40.25batch/s]

Training batches on cuda:  90%|█████████ | 273/302 [00:06<00:00, 40.48batch/s]

Training batches on cuda:  92%|█████████▏| 278/302 [00:06<00:00, 39.94batch/s]

Training batches on cuda:  94%|█████████▎| 283/302 [00:06<00:00, 39.43batch/s]

Training batches on cuda:  95%|█████████▌| 287/302 [00:06<00:00, 38.05batch/s]

Training batches on cuda:  97%|█████████▋| 292/302 [00:06<00:00, 38.80batch/s]

Training batches on cuda:  98%|█████████▊| 297/302 [00:06<00:00, 39.84batch/s]

Training batches on cuda: 100%|██████████| 302/302 [00:07<00:00, 41.85batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:14<01:03,  7.09s/epoch, loss=0.00128, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:14<00:57,  7.17s/epoch, loss=0.00128, prev_loss=nan]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 4/302 [00:00<00:08, 33.82batch/s]

Training batches on cuda:   3%|▎         | 9/302 [00:00<00:08, 36.49batch/s]

Training batches on cuda:   5%|▍         | 14/302 [00:00<00:07, 38.17batch/s]

Training batches on cuda:   6%|▋         | 19/302 [00:00<00:07, 40.01batch/s]

Training batches on cuda:   8%|▊         | 24/302 [00:00<00:06, 40.11batch/s]

Training batches on cuda:  10%|▉         | 29/302 [00:00<00:06, 41.31batch/s]

Training batches on cuda:  11%|█▏        | 34/302 [00:00<00:06, 42.24batch/s]

Training batches on cuda:  13%|█▎        | 39/302 [00:00<00:06, 42.79batch/s]

Training batches on cuda:  15%|█▍        | 44/302 [00:01<00:05, 43.74batch/s]

Training batches on cuda:  16%|█▌        | 49/302 [00:01<00:05, 43.35batch/s]

Training batches on cuda:  18%|█▊        | 54/302 [00:01<00:05, 42.70batch/s]

Training batches on cuda:  20%|█▉        | 59/302 [00:01<00:05, 43.35batch/s]

Training batches on cuda:  21%|██        | 64/302 [00:01<00:05, 43.45batch/s]

Training batches on cuda:  23%|██▎       | 69/302 [00:01<00:05, 42.90batch/s]

Training batches on cuda:  25%|██▍       | 74/302 [00:01<00:05, 43.67batch/s]

Training batches on cuda:  26%|██▌       | 79/302 [00:01<00:05, 43.35batch/s]

Training batches on cuda:  28%|██▊       | 84/302 [00:01<00:05, 43.48batch/s]

Training batches on cuda:  29%|██▉       | 89/302 [00:02<00:04, 43.18batch/s]

Training batches on cuda:  31%|███       | 94/302 [00:02<00:04, 43.64batch/s]

Training batches on cuda:  33%|███▎      | 99/302 [00:02<00:04, 43.95batch/s]

Training batches on cuda:  34%|███▍      | 104/302 [00:02<00:04, 43.78batch/s]

Training batches on cuda:  36%|███▌      | 109/302 [00:02<00:04, 43.92batch/s]

Training batches on cuda:  38%|███▊      | 114/302 [00:02<00:04, 44.60batch/s]

Training batches on cuda:  39%|███▉      | 119/302 [00:02<00:04, 44.33batch/s]

Training batches on cuda:  41%|████      | 124/302 [00:02<00:04, 44.01batch/s]

Training batches on cuda:  43%|████▎     | 129/302 [00:02<00:03, 43.70batch/s]

Training batches on cuda:  44%|████▍     | 134/302 [00:03<00:03, 42.98batch/s]

Training batches on cuda:  46%|████▌     | 139/302 [00:03<00:03, 42.46batch/s]

Training batches on cuda:  48%|████▊     | 144/302 [00:03<00:03, 41.77batch/s]

Training batches on cuda:  49%|████▉     | 149/302 [00:03<00:03, 40.90batch/s]

Training batches on cuda:  51%|█████     | 154/302 [00:03<00:03, 41.28batch/s]

Training batches on cuda:  53%|█████▎    | 159/302 [00:03<00:03, 42.05batch/s]

Training batches on cuda:  54%|█████▍    | 164/302 [00:03<00:03, 42.21batch/s]

Training batches on cuda:  56%|█████▌    | 169/302 [00:03<00:03, 42.29batch/s]

Training batches on cuda:  58%|█████▊    | 174/302 [00:04<00:03, 41.27batch/s]

Training batches on cuda:  59%|█████▉    | 179/302 [00:04<00:02, 41.78batch/s]

Training batches on cuda:  61%|██████    | 184/302 [00:04<00:02, 41.67batch/s]

Training batches on cuda:  63%|██████▎   | 189/302 [00:04<00:02, 42.35batch/s]

Training batches on cuda:  64%|██████▍   | 194/302 [00:04<00:02, 43.00batch/s]

Training batches on cuda:  66%|██████▌   | 199/302 [00:04<00:02, 44.09batch/s]

Training batches on cuda:  68%|██████▊   | 204/302 [00:04<00:02, 44.69batch/s]

Training batches on cuda:  69%|██████▉   | 209/302 [00:04<00:02, 45.01batch/s]

Training batches on cuda:  71%|███████   | 214/302 [00:04<00:01, 45.29batch/s]

Training batches on cuda:  73%|███████▎  | 219/302 [00:05<00:01, 44.75batch/s]

Training batches on cuda:  74%|███████▍  | 224/302 [00:05<00:01, 44.87batch/s]

Training batches on cuda:  76%|███████▌  | 229/302 [00:05<00:01, 44.44batch/s]

Training batches on cuda:  77%|███████▋  | 234/302 [00:05<00:01, 43.78batch/s]

Training batches on cuda:  79%|███████▉  | 239/302 [00:05<00:01, 43.11batch/s]

Training batches on cuda:  81%|████████  | 244/302 [00:05<00:01, 43.35batch/s]

Training batches on cuda:  82%|████████▏ | 249/302 [00:05<00:01, 41.79batch/s]

Training batches on cuda:  84%|████████▍ | 254/302 [00:05<00:01, 42.80batch/s]

Training batches on cuda:  86%|████████▌ | 259/302 [00:06<00:01, 42.26batch/s]

Training batches on cuda:  87%|████████▋ | 264/302 [00:06<00:00, 43.19batch/s]

Training batches on cuda:  89%|████████▉ | 269/302 [00:06<00:00, 44.06batch/s]

Training batches on cuda:  91%|█████████ | 274/302 [00:06<00:00, 44.17batch/s]

Training batches on cuda:  92%|█████████▏| 279/302 [00:06<00:00, 44.25batch/s]

Training batches on cuda:  94%|█████████▍| 284/302 [00:06<00:00, 43.99batch/s]

Training batches on cuda:  96%|█████████▌| 289/302 [00:06<00:00, 44.69batch/s]

Training batches on cuda:  97%|█████████▋| 294/302 [00:06<00:00, 45.01batch/s]

Training batches on cuda:  99%|█████████▉| 299/302 [00:06<00:00, 44.07batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:21<00:57,  7.17s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda:  30%|███       | 3/10 [00:21<00:50,  7.19s/epoch, loss=0.00128, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/302 [00:00<00:10, 29.61batch/s]

Training batches on cuda:   3%|▎         | 8/302 [00:00<00:08, 33.18batch/s]

Training batches on cuda:   4%|▍         | 13/302 [00:00<00:08, 35.54batch/s]

Training batches on cuda:   6%|▌         | 18/302 [00:00<00:07, 37.65batch/s]

Training batches on cuda:   8%|▊         | 23/302 [00:00<00:07, 38.98batch/s]

Training batches on cuda:   9%|▉         | 28/302 [00:00<00:06, 39.96batch/s]

Training batches on cuda:  11%|█         | 33/302 [00:00<00:06, 40.27batch/s]

Training batches on cuda:  13%|█▎        | 38/302 [00:00<00:06, 41.05batch/s]

Training batches on cuda:  14%|█▍        | 43/302 [00:01<00:06, 42.48batch/s]

Training batches on cuda:  16%|█▌        | 48/302 [00:01<00:05, 43.01batch/s]

Training batches on cuda:  18%|█▊        | 53/302 [00:01<00:05, 43.70batch/s]

Training batches on cuda:  19%|█▉        | 58/302 [00:01<00:05, 43.37batch/s]

Training batches on cuda:  21%|██        | 63/302 [00:01<00:05, 43.80batch/s]

Training batches on cuda:  23%|██▎       | 68/302 [00:01<00:05, 43.82batch/s]

Training batches on cuda:  24%|██▍       | 73/302 [00:01<00:05, 43.33batch/s]

Training batches on cuda:  26%|██▌       | 78/302 [00:01<00:05, 41.53batch/s]

Training batches on cuda:  27%|██▋       | 83/302 [00:01<00:05, 41.79batch/s]

Training batches on cuda:  29%|██▉       | 88/302 [00:02<00:05, 42.26batch/s]

Training batches on cuda:  31%|███       | 93/302 [00:02<00:04, 43.15batch/s]

Training batches on cuda:  32%|███▏      | 98/302 [00:02<00:04, 42.01batch/s]

Training batches on cuda:  34%|███▍      | 103/302 [00:02<00:04, 41.88batch/s]

Training batches on cuda:  36%|███▌      | 108/302 [00:02<00:04, 42.03batch/s]

Training batches on cuda:  37%|███▋      | 113/302 [00:02<00:04, 43.34batch/s]

Training batches on cuda:  39%|███▉      | 118/302 [00:02<00:04, 43.60batch/s]

Training batches on cuda:  41%|████      | 123/302 [00:02<00:04, 43.54batch/s]

Training batches on cuda:  42%|████▏     | 128/302 [00:02<00:03, 43.59batch/s]

Training batches on cuda:  44%|████▍     | 133/302 [00:03<00:03, 43.90batch/s]

Training batches on cuda:  46%|████▌     | 138/302 [00:03<00:03, 43.56batch/s]

Training batches on cuda:  47%|████▋     | 143/302 [00:03<00:03, 43.46batch/s]

Training batches on cuda:  49%|████▉     | 148/302 [00:03<00:03, 44.02batch/s]

Training batches on cuda:  51%|█████     | 153/302 [00:03<00:03, 44.94batch/s]

Training batches on cuda:  52%|█████▏    | 158/302 [00:03<00:03, 44.17batch/s]

Training batches on cuda:  54%|█████▍    | 163/302 [00:03<00:03, 44.63batch/s]

Training batches on cuda:  56%|█████▌    | 168/302 [00:03<00:02, 44.96batch/s]

Training batches on cuda:  57%|█████▋    | 173/302 [00:04<00:02, 45.42batch/s]

Training batches on cuda:  59%|█████▉    | 178/302 [00:04<00:02, 45.12batch/s]

Training batches on cuda:  61%|██████    | 183/302 [00:04<00:02, 43.67batch/s]

Training batches on cuda:  62%|██████▏   | 188/302 [00:04<00:02, 43.45batch/s]

Training batches on cuda:  64%|██████▍   | 193/302 [00:04<00:02, 39.45batch/s]

Training batches on cuda:  66%|██████▌   | 198/302 [00:04<00:02, 39.53batch/s]

Training batches on cuda:  67%|██████▋   | 203/302 [00:04<00:02, 41.32batch/s]

Training batches on cuda:  69%|██████▉   | 208/302 [00:04<00:02, 41.85batch/s]

Training batches on cuda:  71%|███████   | 213/302 [00:04<00:02, 42.56batch/s]

Training batches on cuda:  72%|███████▏  | 218/302 [00:05<00:01, 42.13batch/s]

Training batches on cuda:  74%|███████▍  | 223/302 [00:05<00:01, 42.44batch/s]

Training batches on cuda:  75%|███████▌  | 228/302 [00:05<00:01, 42.55batch/s]

Training batches on cuda:  77%|███████▋  | 233/302 [00:05<00:01, 42.68batch/s]

Training batches on cuda:  79%|███████▉  | 238/302 [00:05<00:01, 43.56batch/s]

Training batches on cuda:  80%|████████  | 243/302 [00:05<00:01, 43.86batch/s]

Training batches on cuda:  82%|████████▏ | 248/302 [00:05<00:01, 44.33batch/s]

Training batches on cuda:  84%|████████▍ | 253/302 [00:05<00:01, 44.60batch/s]

Training batches on cuda:  85%|████████▌ | 258/302 [00:05<00:00, 44.49batch/s]

Training batches on cuda:  87%|████████▋ | 263/302 [00:06<00:00, 44.35batch/s]

Training batches on cuda:  89%|████████▊ | 268/302 [00:06<00:00, 44.17batch/s]

Training batches on cuda:  90%|█████████ | 273/302 [00:06<00:00, 43.57batch/s]

Training batches on cuda:  92%|█████████▏| 278/302 [00:06<00:00, 43.74batch/s]

Training batches on cuda:  94%|█████████▎| 283/302 [00:06<00:00, 44.30batch/s]

Training batches on cuda:  95%|█████████▌| 288/302 [00:06<00:00, 43.63batch/s]

Training batches on cuda:  97%|█████████▋| 293/302 [00:06<00:00, 43.40batch/s]

Training batches on cuda:  99%|█████████▊| 298/302 [00:06<00:00, 43.85batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:28<00:50,  7.19s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda:  40%|████      | 4/10 [00:28<00:43,  7.22s/epoch, loss=0.00128, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/302 [00:00<00:10, 28.45batch/s]

Training batches on cuda:   3%|▎         | 8/302 [00:00<00:09, 31.83batch/s]

Training batches on cuda:   4%|▍         | 12/302 [00:00<00:08, 33.74batch/s]

Training batches on cuda:   6%|▌         | 17/302 [00:00<00:07, 35.89batch/s]

Training batches on cuda:   7%|▋         | 22/302 [00:00<00:07, 38.14batch/s]

Training batches on cuda:   9%|▉         | 27/302 [00:00<00:06, 39.44batch/s]

Training batches on cuda:  11%|█         | 32/302 [00:00<00:06, 40.58batch/s]

Training batches on cuda:  12%|█▏        | 37/302 [00:00<00:06, 41.30batch/s]

Training batches on cuda:  14%|█▍        | 42/302 [00:01<00:06, 41.50batch/s]

Training batches on cuda:  16%|█▌        | 47/302 [00:01<00:06, 42.25batch/s]

Training batches on cuda:  17%|█▋        | 52/302 [00:01<00:05, 41.83batch/s]

Training batches on cuda:  19%|█▉        | 57/302 [00:01<00:06, 39.81batch/s]

Training batches on cuda:  21%|██        | 62/302 [00:01<00:05, 40.24batch/s]

Training batches on cuda:  22%|██▏       | 67/302 [00:01<00:05, 40.74batch/s]

Training batches on cuda:  24%|██▍       | 72/302 [00:01<00:05, 42.14batch/s]

Training batches on cuda:  25%|██▌       | 77/302 [00:01<00:05, 43.12batch/s]

Training batches on cuda:  27%|██▋       | 82/302 [00:01<00:05, 42.70batch/s]

Training batches on cuda:  29%|██▉       | 87/302 [00:02<00:04, 43.22batch/s]

Training batches on cuda:  30%|███       | 92/302 [00:02<00:05, 39.98batch/s]

Training batches on cuda:  32%|███▏      | 97/302 [00:02<00:05, 40.58batch/s]

Training batches on cuda:  34%|███▍      | 102/302 [00:02<00:04, 41.23batch/s]

Training batches on cuda:  35%|███▌      | 107/302 [00:02<00:04, 41.11batch/s]

Training batches on cuda:  37%|███▋      | 112/302 [00:02<00:04, 40.30batch/s]

Training batches on cuda:  39%|███▊      | 117/302 [00:02<00:04, 40.04batch/s]

Training batches on cuda:  40%|████      | 122/302 [00:02<00:04, 41.25batch/s]

Training batches on cuda:  42%|████▏     | 127/302 [00:03<00:04, 41.21batch/s]

Training batches on cuda:  44%|████▎     | 132/302 [00:03<00:03, 42.66batch/s]

Training batches on cuda:  45%|████▌     | 137/302 [00:03<00:03, 42.55batch/s]

Training batches on cuda:  47%|████▋     | 142/302 [00:03<00:03, 43.49batch/s]

Training batches on cuda:  49%|████▊     | 147/302 [00:03<00:03, 44.34batch/s]

Training batches on cuda:  50%|█████     | 152/302 [00:03<00:03, 43.78batch/s]

Training batches on cuda:  52%|█████▏    | 157/302 [00:03<00:03, 42.75batch/s]

Training batches on cuda:  54%|█████▎    | 162/302 [00:03<00:03, 42.47batch/s]

Training batches on cuda:  55%|█████▌    | 167/302 [00:03<00:03, 42.89batch/s]

Training batches on cuda:  57%|█████▋    | 172/302 [00:04<00:03, 42.69batch/s]

Training batches on cuda:  59%|█████▊    | 177/302 [00:04<00:02, 42.62batch/s]

Training batches on cuda:  60%|██████    | 182/302 [00:04<00:02, 42.24batch/s]

Training batches on cuda:  62%|██████▏   | 187/302 [00:04<00:02, 42.65batch/s]

Training batches on cuda:  64%|██████▎   | 192/302 [00:04<00:02, 43.22batch/s]

Training batches on cuda:  65%|██████▌   | 197/302 [00:04<00:02, 42.83batch/s]

Training batches on cuda:  67%|██████▋   | 202/302 [00:04<00:02, 43.26batch/s]

Training batches on cuda:  69%|██████▊   | 207/302 [00:04<00:02, 43.70batch/s]

Training batches on cuda:  70%|███████   | 212/302 [00:05<00:02, 43.58batch/s]

Training batches on cuda:  72%|███████▏  | 217/302 [00:05<00:01, 44.27batch/s]

Training batches on cuda:  74%|███████▎  | 222/302 [00:05<00:01, 43.67batch/s]

Training batches on cuda:  75%|███████▌  | 227/302 [00:05<00:01, 44.32batch/s]

Training batches on cuda:  77%|███████▋  | 232/302 [00:05<00:01, 43.79batch/s]

Training batches on cuda:  78%|███████▊  | 237/302 [00:05<00:01, 44.18batch/s]

Training batches on cuda:  80%|████████  | 242/302 [00:05<00:01, 44.00batch/s]

Training batches on cuda:  82%|████████▏ | 247/302 [00:05<00:01, 43.81batch/s]

Training batches on cuda:  83%|████████▎ | 252/302 [00:05<00:01, 44.12batch/s]

Training batches on cuda:  85%|████████▌ | 257/302 [00:06<00:01, 44.73batch/s]

Training batches on cuda:  87%|████████▋ | 262/302 [00:06<00:00, 44.92batch/s]

Training batches on cuda:  88%|████████▊ | 267/302 [00:06<00:00, 45.22batch/s]

Training batches on cuda:  90%|█████████ | 272/302 [00:06<00:00, 45.29batch/s]

Training batches on cuda:  92%|█████████▏| 277/302 [00:06<00:00, 45.12batch/s]

Training batches on cuda:  93%|█████████▎| 282/302 [00:06<00:00, 44.55batch/s]

Training batches on cuda:  95%|█████████▌| 287/302 [00:06<00:00, 44.63batch/s]

Training batches on cuda:  97%|█████████▋| 292/302 [00:06<00:00, 44.50batch/s]

Training batches on cuda:  98%|█████████▊| 297/302 [00:06<00:00, 42.27batch/s]

Training batches on cuda: 100%|██████████| 302/302 [00:07<00:00, 44.27batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:36<00:43,  7.22s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda:  50%|█████     | 5/10 [00:36<00:36,  7.24s/epoch, loss=0.00128, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/302 [00:00<00:10, 27.25batch/s]

Training batches on cuda:   3%|▎         | 8/302 [00:00<00:09, 30.79batch/s]

Training batches on cuda:   4%|▍         | 13/302 [00:00<00:08, 33.97batch/s]

Training batches on cuda:   6%|▌         | 18/302 [00:00<00:07, 36.39batch/s]

Training batches on cuda:   8%|▊         | 23/302 [00:00<00:07, 38.39batch/s]

Training batches on cuda:   9%|▉         | 28/302 [00:00<00:06, 40.29batch/s]

Training batches on cuda:  11%|█         | 33/302 [00:00<00:06, 40.76batch/s]

Training batches on cuda:  13%|█▎        | 38/302 [00:00<00:06, 41.57batch/s]

Training batches on cuda:  14%|█▍        | 43/302 [00:01<00:06, 42.28batch/s]

Training batches on cuda:  16%|█▌        | 48/302 [00:01<00:06, 42.05batch/s]

Training batches on cuda:  18%|█▊        | 53/302 [00:01<00:05, 43.13batch/s]

Training batches on cuda:  19%|█▉        | 58/302 [00:01<00:05, 43.05batch/s]

Training batches on cuda:  21%|██        | 63/302 [00:01<00:05, 43.48batch/s]

Training batches on cuda:  23%|██▎       | 68/302 [00:01<00:05, 44.56batch/s]

Training batches on cuda:  24%|██▍       | 73/302 [00:01<00:05, 44.21batch/s]

Training batches on cuda:  26%|██▌       | 78/302 [00:01<00:04, 44.83batch/s]

Training batches on cuda:  27%|██▋       | 83/302 [00:01<00:04, 45.17batch/s]

Training batches on cuda:  29%|██▉       | 88/302 [00:02<00:04, 44.23batch/s]

Training batches on cuda:  31%|███       | 93/302 [00:02<00:04, 42.44batch/s]

Training batches on cuda:  32%|███▏      | 98/302 [00:02<00:04, 41.85batch/s]

Training batches on cuda:  34%|███▍      | 103/302 [00:02<00:04, 41.55batch/s]

Training batches on cuda:  36%|███▌      | 108/302 [00:02<00:04, 42.17batch/s]

Training batches on cuda:  37%|███▋      | 113/302 [00:02<00:04, 42.07batch/s]

Training batches on cuda:  39%|███▉      | 118/302 [00:02<00:04, 43.01batch/s]

Training batches on cuda:  41%|████      | 123/302 [00:02<00:04, 43.53batch/s]

Training batches on cuda:  42%|████▏     | 128/302 [00:02<00:04, 42.47batch/s]

Training batches on cuda:  44%|████▍     | 133/302 [00:03<00:04, 40.62batch/s]

Training batches on cuda:  46%|████▌     | 138/302 [00:03<00:04, 40.82batch/s]

Training batches on cuda:  47%|████▋     | 143/302 [00:03<00:03, 41.72batch/s]

Training batches on cuda:  49%|████▉     | 148/302 [00:03<00:03, 42.55batch/s]

Training batches on cuda:  51%|█████     | 153/302 [00:03<00:03, 43.48batch/s]

Training batches on cuda:  52%|█████▏    | 158/302 [00:03<00:03, 43.31batch/s]

Training batches on cuda:  54%|█████▍    | 163/302 [00:03<00:03, 42.94batch/s]

Training batches on cuda:  56%|█████▌    | 168/302 [00:03<00:03, 42.93batch/s]

Training batches on cuda:  57%|█████▋    | 173/302 [00:04<00:02, 43.60batch/s]

Training batches on cuda:  59%|█████▉    | 178/302 [00:04<00:02, 44.31batch/s]

Training batches on cuda:  61%|██████    | 183/302 [00:04<00:02, 44.78batch/s]

Training batches on cuda:  62%|██████▏   | 188/302 [00:04<00:02, 45.00batch/s]

Training batches on cuda:  64%|██████▍   | 193/302 [00:04<00:02, 45.00batch/s]

Training batches on cuda:  66%|██████▌   | 198/302 [00:04<00:02, 44.55batch/s]

Training batches on cuda:  67%|██████▋   | 203/302 [00:04<00:02, 44.09batch/s]

Training batches on cuda:  69%|██████▉   | 208/302 [00:04<00:02, 44.35batch/s]

Training batches on cuda:  71%|███████   | 213/302 [00:04<00:02, 40.58batch/s]

Training batches on cuda:  72%|███████▏  | 218/302 [00:05<00:01, 42.03batch/s]

Training batches on cuda:  74%|███████▍  | 223/302 [00:05<00:01, 43.13batch/s]

Training batches on cuda:  75%|███████▌  | 228/302 [00:05<00:01, 43.79batch/s]

Training batches on cuda:  77%|███████▋  | 233/302 [00:05<00:01, 44.09batch/s]

Training batches on cuda:  79%|███████▉  | 238/302 [00:05<00:01, 44.23batch/s]

Training batches on cuda:  80%|████████  | 243/302 [00:05<00:01, 43.33batch/s]

Training batches on cuda:  82%|████████▏ | 248/302 [00:05<00:01, 42.65batch/s]

Training batches on cuda:  84%|████████▍ | 253/302 [00:05<00:01, 42.95batch/s]

Training batches on cuda:  85%|████████▌ | 258/302 [00:05<00:01, 42.67batch/s]

Training batches on cuda:  87%|████████▋ | 263/302 [00:06<00:00, 43.24batch/s]

Training batches on cuda:  89%|████████▊ | 268/302 [00:06<00:00, 43.60batch/s]

Training batches on cuda:  90%|█████████ | 273/302 [00:06<00:00, 44.11batch/s]

Training batches on cuda:  92%|█████████▏| 278/302 [00:06<00:00, 43.02batch/s]

Training batches on cuda:  94%|█████████▎| 283/302 [00:06<00:00, 43.51batch/s]

Training batches on cuda:  95%|█████████▌| 288/302 [00:06<00:00, 43.69batch/s]

Training batches on cuda:  97%|█████████▋| 293/302 [00:06<00:00, 44.06batch/s]

Training batches on cuda:  99%|█████████▊| 298/302 [00:06<00:00, 44.33batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:43<00:36,  7.24s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda:  60%|██████    | 6/10 [00:43<00:28,  7.22s/epoch, loss=0.00128, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/302 [00:00<00:10, 27.60batch/s]

Training batches on cuda:   3%|▎         | 8/302 [00:00<00:09, 31.21batch/s]

Training batches on cuda:   4%|▍         | 13/302 [00:00<00:08, 33.92batch/s]

Training batches on cuda:   6%|▌         | 18/302 [00:00<00:07, 36.07batch/s]

Training batches on cuda:   8%|▊         | 23/302 [00:00<00:07, 38.26batch/s]

Training batches on cuda:   9%|▉         | 28/302 [00:00<00:06, 39.53batch/s]

Training batches on cuda:  11%|█         | 33/302 [00:00<00:06, 40.88batch/s]

Training batches on cuda:  13%|█▎        | 38/302 [00:00<00:06, 41.98batch/s]

Training batches on cuda:  14%|█▍        | 43/302 [00:01<00:06, 42.21batch/s]

Training batches on cuda:  16%|█▌        | 48/302 [00:01<00:05, 43.30batch/s]

Training batches on cuda:  18%|█▊        | 53/302 [00:01<00:05, 43.15batch/s]

Training batches on cuda:  19%|█▉        | 58/302 [00:01<00:05, 43.15batch/s]

Training batches on cuda:  21%|██        | 63/302 [00:01<00:05, 42.80batch/s]

Training batches on cuda:  23%|██▎       | 68/302 [00:01<00:05, 42.61batch/s]

Training batches on cuda:  24%|██▍       | 73/302 [00:01<00:05, 42.92batch/s]

Training batches on cuda:  26%|██▌       | 78/302 [00:01<00:05, 41.91batch/s]

Training batches on cuda:  27%|██▋       | 83/302 [00:01<00:05, 42.36batch/s]

Training batches on cuda:  29%|██▉       | 88/302 [00:02<00:05, 42.51batch/s]

Training batches on cuda:  31%|███       | 93/302 [00:02<00:04, 43.25batch/s]

Training batches on cuda:  32%|███▏      | 98/302 [00:02<00:04, 44.11batch/s]

Training batches on cuda:  34%|███▍      | 103/302 [00:02<00:04, 44.07batch/s]

Training batches on cuda:  36%|███▌      | 108/302 [00:02<00:04, 43.05batch/s]

Training batches on cuda:  37%|███▋      | 113/302 [00:02<00:04, 42.42batch/s]

Training batches on cuda:  39%|███▉      | 118/302 [00:02<00:05, 35.79batch/s]

Training batches on cuda:  40%|████      | 122/302 [00:02<00:04, 36.19batch/s]

Training batches on cuda:  42%|████▏     | 127/302 [00:03<00:04, 37.38batch/s]

Training batches on cuda:  44%|████▎     | 132/302 [00:03<00:04, 39.33batch/s]

Training batches on cuda:  45%|████▌     | 137/302 [00:03<00:04, 40.20batch/s]

Training batches on cuda:  47%|████▋     | 142/302 [00:03<00:03, 40.46batch/s]

Training batches on cuda:  49%|████▊     | 147/302 [00:03<00:03, 41.17batch/s]

Training batches on cuda:  50%|█████     | 152/302 [00:03<00:03, 42.06batch/s]

Training batches on cuda:  52%|█████▏    | 157/302 [00:03<00:03, 43.10batch/s]

Training batches on cuda:  54%|█████▎    | 162/302 [00:03<00:03, 43.10batch/s]

Training batches on cuda:  55%|█████▌    | 167/302 [00:03<00:03, 43.60batch/s]

Training batches on cuda:  57%|█████▋    | 172/302 [00:04<00:02, 43.58batch/s]

Training batches on cuda:  59%|█████▊    | 177/302 [00:04<00:02, 44.08batch/s]

Training batches on cuda:  60%|██████    | 182/302 [00:04<00:02, 44.27batch/s]

Training batches on cuda:  62%|██████▏   | 187/302 [00:04<00:02, 44.31batch/s]

Training batches on cuda:  64%|██████▎   | 192/302 [00:04<00:02, 43.24batch/s]

Training batches on cuda:  65%|██████▌   | 197/302 [00:04<00:02, 43.66batch/s]

Training batches on cuda:  67%|██████▋   | 202/302 [00:04<00:02, 43.08batch/s]

Training batches on cuda:  69%|██████▊   | 207/302 [00:04<00:02, 43.88batch/s]

Training batches on cuda:  70%|███████   | 212/302 [00:05<00:02, 43.75batch/s]

Training batches on cuda:  72%|███████▏  | 217/302 [00:05<00:01, 44.21batch/s]

Training batches on cuda:  74%|███████▎  | 222/302 [00:05<00:01, 43.73batch/s]

Training batches on cuda:  75%|███████▌  | 227/302 [00:05<00:01, 43.70batch/s]

Training batches on cuda:  77%|███████▋  | 232/302 [00:05<00:01, 43.40batch/s]

Training batches on cuda:  78%|███████▊  | 237/302 [00:05<00:01, 43.55batch/s]

Training batches on cuda:  80%|████████  | 242/302 [00:05<00:01, 43.48batch/s]

Training batches on cuda:  82%|████████▏ | 247/302 [00:05<00:01, 43.93batch/s]

Training batches on cuda:  83%|████████▎ | 252/302 [00:05<00:01, 44.07batch/s]

Training batches on cuda:  85%|████████▌ | 257/302 [00:06<00:01, 43.94batch/s]

Training batches on cuda:  87%|████████▋ | 262/302 [00:06<00:00, 44.31batch/s]

Training batches on cuda:  88%|████████▊ | 267/302 [00:06<00:00, 43.94batch/s]

Training batches on cuda:  90%|█████████ | 272/302 [00:06<00:00, 43.49batch/s]

Training batches on cuda:  92%|█████████▏| 277/302 [00:06<00:00, 43.67batch/s]

Training batches on cuda:  93%|█████████▎| 282/302 [00:06<00:00, 43.86batch/s]

Training batches on cuda:  95%|█████████▌| 287/302 [00:06<00:00, 44.08batch/s]

Training batches on cuda:  97%|█████████▋| 292/302 [00:06<00:00, 44.07batch/s]

Training batches on cuda:  98%|█████████▊| 297/302 [00:06<00:00, 44.33batch/s]

Training batches on cuda: 100%|██████████| 302/302 [00:07<00:00, 45.76batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:50<00:28,  7.22s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda:  70%|███████   | 7/10 [00:50<00:21,  7.26s/epoch, loss=0.00128, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/302 [00:00<00:10, 28.96batch/s]

Training batches on cuda:   3%|▎         | 8/302 [00:00<00:09, 32.21batch/s]

Training batches on cuda:   4%|▍         | 13/302 [00:00<00:08, 34.54batch/s]

Training batches on cuda:   6%|▌         | 17/302 [00:00<00:07, 35.75batch/s]

Training batches on cuda:   7%|▋         | 22/302 [00:00<00:07, 38.14batch/s]

Training batches on cuda:   9%|▉         | 27/302 [00:00<00:06, 39.67batch/s]

Training batches on cuda:  11%|█         | 32/302 [00:00<00:06, 40.69batch/s]

Training batches on cuda:  12%|█▏        | 37/302 [00:00<00:06, 42.04batch/s]

Training batches on cuda:  14%|█▍        | 42/302 [00:01<00:06, 42.18batch/s]

Training batches on cuda:  16%|█▌        | 47/302 [00:01<00:05, 42.65batch/s]

Training batches on cuda:  17%|█▋        | 52/302 [00:01<00:05, 43.47batch/s]

Training batches on cuda:  19%|█▉        | 57/302 [00:01<00:05, 43.79batch/s]

Training batches on cuda:  21%|██        | 62/302 [00:01<00:05, 43.02batch/s]

Training batches on cuda:  22%|██▏       | 67/302 [00:01<00:05, 42.73batch/s]

Training batches on cuda:  24%|██▍       | 72/302 [00:01<00:05, 42.71batch/s]

Training batches on cuda:  25%|██▌       | 77/302 [00:01<00:05, 43.52batch/s]

Training batches on cuda:  27%|██▋       | 82/302 [00:01<00:05, 43.99batch/s]

Training batches on cuda:  29%|██▉       | 87/302 [00:02<00:04, 43.98batch/s]

Training batches on cuda:  30%|███       | 92/302 [00:02<00:04, 43.75batch/s]

Training batches on cuda:  32%|███▏      | 97/302 [00:02<00:04, 43.40batch/s]

Training batches on cuda:  34%|███▍      | 102/302 [00:02<00:04, 40.28batch/s]

Training batches on cuda:  35%|███▌      | 107/302 [00:02<00:04, 41.56batch/s]

Training batches on cuda:  37%|███▋      | 112/302 [00:02<00:04, 42.07batch/s]

Training batches on cuda:  39%|███▊      | 117/302 [00:02<00:04, 41.40batch/s]

Training batches on cuda:  40%|████      | 122/302 [00:02<00:04, 40.11batch/s]

Training batches on cuda:  42%|████▏     | 127/302 [00:03<00:04, 40.49batch/s]

Training batches on cuda:  44%|████▎     | 132/302 [00:03<00:04, 41.02batch/s]

Training batches on cuda:  45%|████▌     | 137/302 [00:03<00:03, 41.41batch/s]

Training batches on cuda:  47%|████▋     | 142/302 [00:03<00:03, 41.93batch/s]

Training batches on cuda:  49%|████▊     | 147/302 [00:03<00:03, 42.19batch/s]

Training batches on cuda:  50%|█████     | 152/302 [00:03<00:03, 41.27batch/s]

Training batches on cuda:  52%|█████▏    | 157/302 [00:03<00:03, 41.73batch/s]

Training batches on cuda:  54%|█████▎    | 162/302 [00:03<00:03, 42.05batch/s]

Training batches on cuda:  55%|█████▌    | 167/302 [00:03<00:03, 42.75batch/s]

Training batches on cuda:  57%|█████▋    | 172/302 [00:04<00:03, 43.20batch/s]

Training batches on cuda:  59%|█████▊    | 177/302 [00:04<00:02, 43.10batch/s]

Training batches on cuda:  60%|██████    | 182/302 [00:04<00:02, 43.21batch/s]

Training batches on cuda:  62%|██████▏   | 187/302 [00:04<00:02, 43.56batch/s]

Training batches on cuda:  64%|██████▎   | 192/302 [00:04<00:02, 43.21batch/s]

Training batches on cuda:  65%|██████▌   | 197/302 [00:04<00:02, 42.89batch/s]

Training batches on cuda:  67%|██████▋   | 202/302 [00:04<00:02, 43.17batch/s]

Training batches on cuda:  69%|██████▊   | 207/302 [00:04<00:02, 43.43batch/s]

Training batches on cuda:  70%|███████   | 212/302 [00:05<00:02, 42.46batch/s]

Training batches on cuda:  72%|███████▏  | 217/302 [00:05<00:02, 41.64batch/s]

Training batches on cuda:  74%|███████▎  | 222/302 [00:05<00:01, 42.45batch/s]

Training batches on cuda:  75%|███████▌  | 227/302 [00:05<00:01, 41.82batch/s]

Training batches on cuda:  77%|███████▋  | 232/302 [00:05<00:01, 42.63batch/s]

Training batches on cuda:  78%|███████▊  | 237/302 [00:05<00:01, 42.90batch/s]

Training batches on cuda:  80%|████████  | 242/302 [00:05<00:01, 42.67batch/s]

Training batches on cuda:  82%|████████▏ | 247/302 [00:05<00:01, 41.91batch/s]

Training batches on cuda:  83%|████████▎ | 252/302 [00:05<00:01, 40.79batch/s]

Training batches on cuda:  85%|████████▌ | 257/302 [00:06<00:01, 41.32batch/s]

Training batches on cuda:  87%|████████▋ | 262/302 [00:06<00:00, 41.61batch/s]

Training batches on cuda:  88%|████████▊ | 267/302 [00:06<00:00, 41.24batch/s]

Training batches on cuda:  90%|█████████ | 272/302 [00:06<00:00, 40.40batch/s]

Training batches on cuda:  92%|█████████▏| 277/302 [00:06<00:00, 40.34batch/s]

Training batches on cuda:  93%|█████████▎| 282/302 [00:06<00:00, 41.97batch/s]

Training batches on cuda:  95%|█████████▌| 287/302 [00:06<00:00, 43.06batch/s]

Training batches on cuda:  97%|█████████▋| 292/302 [00:06<00:00, 43.85batch/s]

Training batches on cuda:  98%|█████████▊| 297/302 [00:07<00:00, 44.26batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:58<00:21,  7.26s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda:  80%|████████  | 8/10 [00:58<00:14,  7.30s/epoch, loss=0.00128, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/302 [00:00<00:16, 17.88batch/s]

Training batches on cuda:   2%|▏         | 7/302 [00:00<00:13, 21.63batch/s]

Training batches on cuda:   4%|▍         | 12/302 [00:00<00:11, 25.45batch/s]

Training batches on cuda:   6%|▌         | 17/302 [00:00<00:09, 29.11batch/s]

Training batches on cuda:   7%|▋         | 22/302 [00:00<00:08, 32.74batch/s]

Training batches on cuda:   9%|▉         | 27/302 [00:00<00:07, 35.60batch/s]

Training batches on cuda:  11%|█         | 32/302 [00:00<00:07, 37.08batch/s]

Training batches on cuda:  12%|█▏        | 37/302 [00:00<00:06, 39.40batch/s]

Training batches on cuda:  14%|█▍        | 42/302 [00:01<00:06, 39.83batch/s]

Training batches on cuda:  16%|█▌        | 47/302 [00:01<00:06, 40.56batch/s]

Training batches on cuda:  17%|█▋        | 52/302 [00:01<00:06, 40.93batch/s]

Training batches on cuda:  19%|█▉        | 57/302 [00:01<00:05, 42.24batch/s]

Training batches on cuda:  21%|██        | 62/302 [00:01<00:05, 43.09batch/s]

Training batches on cuda:  22%|██▏       | 67/302 [00:01<00:05, 43.58batch/s]

Training batches on cuda:  24%|██▍       | 72/302 [00:01<00:05, 42.40batch/s]

Training batches on cuda:  25%|██▌       | 77/302 [00:01<00:05, 41.94batch/s]

Training batches on cuda:  27%|██▋       | 82/302 [00:01<00:05, 42.93batch/s]

Training batches on cuda:  29%|██▉       | 87/302 [00:02<00:04, 43.16batch/s]

Training batches on cuda:  30%|███       | 92/302 [00:02<00:04, 44.09batch/s]

Training batches on cuda:  32%|███▏      | 97/302 [00:02<00:04, 44.05batch/s]

Training batches on cuda:  34%|███▍      | 102/302 [00:02<00:04, 44.23batch/s]

Training batches on cuda:  35%|███▌      | 107/302 [00:02<00:04, 44.44batch/s]

Training batches on cuda:  37%|███▋      | 112/302 [00:02<00:04, 44.15batch/s]

Training batches on cuda:  39%|███▊      | 117/302 [00:02<00:04, 43.77batch/s]

Training batches on cuda:  40%|████      | 122/302 [00:02<00:04, 43.17batch/s]

Training batches on cuda:  42%|████▏     | 127/302 [00:02<00:03, 43.91batch/s]

Training batches on cuda:  44%|████▎     | 132/302 [00:03<00:04, 39.53batch/s]

Training batches on cuda:  45%|████▌     | 137/302 [00:03<00:04, 40.40batch/s]

Training batches on cuda:  47%|████▋     | 142/302 [00:03<00:03, 41.33batch/s]

Training batches on cuda:  49%|████▊     | 147/302 [00:03<00:03, 41.68batch/s]

Training batches on cuda:  50%|█████     | 152/302 [00:03<00:03, 40.97batch/s]

Training batches on cuda:  52%|█████▏    | 157/302 [00:03<00:03, 41.84batch/s]

Training batches on cuda:  54%|█████▎    | 162/302 [00:03<00:03, 43.16batch/s]

Training batches on cuda:  55%|█████▌    | 167/302 [00:03<00:03, 43.48batch/s]

Training batches on cuda:  57%|█████▋    | 172/302 [00:04<00:02, 44.18batch/s]

Training batches on cuda:  59%|█████▊    | 177/302 [00:04<00:02, 44.19batch/s]

Training batches on cuda:  60%|██████    | 182/302 [00:04<00:02, 44.41batch/s]

Training batches on cuda:  62%|██████▏   | 187/302 [00:04<00:02, 43.99batch/s]

Training batches on cuda:  64%|██████▎   | 192/302 [00:04<00:02, 42.92batch/s]

Training batches on cuda:  65%|██████▌   | 197/302 [00:04<00:02, 42.54batch/s]

Training batches on cuda:  67%|██████▋   | 202/302 [00:04<00:02, 42.40batch/s]

Training batches on cuda:  69%|██████▊   | 207/302 [00:04<00:02, 42.69batch/s]

Training batches on cuda:  70%|███████   | 212/302 [00:04<00:02, 42.79batch/s]

Training batches on cuda:  72%|███████▏  | 217/302 [00:05<00:01, 43.02batch/s]

Training batches on cuda:  74%|███████▎  | 222/302 [00:05<00:01, 43.74batch/s]

Training batches on cuda:  75%|███████▌  | 227/302 [00:05<00:01, 44.19batch/s]

Training batches on cuda:  77%|███████▋  | 232/302 [00:05<00:01, 44.07batch/s]

Training batches on cuda:  78%|███████▊  | 237/302 [00:05<00:01, 44.44batch/s]

Training batches on cuda:  80%|████████  | 242/302 [00:05<00:01, 44.94batch/s]

Training batches on cuda:  82%|████████▏ | 247/302 [00:05<00:01, 44.53batch/s]

Training batches on cuda:  83%|████████▎ | 252/302 [00:05<00:01, 44.90batch/s]

Training batches on cuda:  85%|████████▌ | 257/302 [00:05<00:01, 45.00batch/s]

Training batches on cuda:  87%|████████▋ | 262/302 [00:06<00:00, 45.45batch/s]

Training batches on cuda:  88%|████████▊ | 267/302 [00:06<00:00, 45.66batch/s]

Training batches on cuda:  90%|█████████ | 272/302 [00:06<00:00, 45.72batch/s]

Training batches on cuda:  92%|█████████▏| 277/302 [00:06<00:00, 45.81batch/s]

Training batches on cuda:  93%|█████████▎| 282/302 [00:06<00:00, 45.07batch/s]

Training batches on cuda:  95%|█████████▌| 287/302 [00:06<00:00, 45.25batch/s]

Training batches on cuda:  97%|█████████▋| 292/302 [00:06<00:00, 45.37batch/s]

Training batches on cuda:  98%|█████████▊| 297/302 [00:06<00:00, 44.78batch/s]

Training batches on cuda: 100%|██████████| 302/302 [00:06<00:00, 45.79batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:05<00:14,  7.30s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda:  90%|█████████ | 9/10 [01:05<00:07,  7.29s/epoch, loss=0.00128, prev_loss=0.00128]

Training batches on cuda:   0%|          | 0/302 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 1/302 [00:00<00:30,  9.83batch/s]

Training batches on cuda:   2%|▏         | 6/302 [00:00<00:23, 12.83batch/s]

Training batches on cuda:   4%|▎         | 11/302 [00:00<00:17, 16.36batch/s]

Training batches on cuda:   5%|▌         | 16/302 [00:00<00:14, 20.15batch/s]

Training batches on cuda:   7%|▋         | 21/302 [00:00<00:11, 24.14batch/s]

Training batches on cuda:   9%|▊         | 26/302 [00:00<00:09, 27.76batch/s]

Training batches on cuda:  10%|█         | 31/302 [00:00<00:08, 31.10batch/s]

Training batches on cuda:  12%|█▏        | 36/302 [00:00<00:07, 34.00batch/s]

Training batches on cuda:  14%|█▎        | 41/302 [00:01<00:07, 35.97batch/s]

Training batches on cuda:  15%|█▌        | 46/302 [00:01<00:06, 37.75batch/s]

Training batches on cuda:  17%|█▋        | 51/302 [00:01<00:06, 39.16batch/s]

Training batches on cuda:  19%|█▊        | 56/302 [00:01<00:06, 40.31batch/s]

Training batches on cuda:  20%|██        | 61/302 [00:01<00:05, 41.25batch/s]

Training batches on cuda:  22%|██▏       | 66/302 [00:01<00:05, 42.44batch/s]

Training batches on cuda:  24%|██▎       | 71/302 [00:01<00:05, 43.28batch/s]

Training batches on cuda:  25%|██▌       | 76/302 [00:01<00:05, 43.77batch/s]

Training batches on cuda:  27%|██▋       | 81/302 [00:01<00:05, 43.52batch/s]

Training batches on cuda:  28%|██▊       | 86/302 [00:02<00:04, 43.92batch/s]

Training batches on cuda:  30%|███       | 91/302 [00:02<00:04, 44.32batch/s]

Training batches on cuda:  32%|███▏      | 96/302 [00:02<00:04, 44.50batch/s]

Training batches on cuda:  33%|███▎      | 101/302 [00:02<00:04, 44.65batch/s]

Training batches on cuda:  35%|███▌      | 106/302 [00:02<00:04, 44.46batch/s]

Training batches on cuda:  37%|███▋      | 111/302 [00:02<00:04, 44.78batch/s]

Training batches on cuda:  38%|███▊      | 116/302 [00:02<00:04, 44.19batch/s]

Training batches on cuda:  40%|████      | 121/302 [00:02<00:04, 44.76batch/s]

Training batches on cuda:  42%|████▏     | 126/302 [00:02<00:03, 44.62batch/s]

Training batches on cuda:  43%|████▎     | 131/302 [00:03<00:03, 43.87batch/s]

Training batches on cuda:  45%|████▌     | 136/302 [00:03<00:03, 43.74batch/s]

Training batches on cuda:  47%|████▋     | 141/302 [00:03<00:03, 43.70batch/s]

Training batches on cuda:  48%|████▊     | 146/302 [00:03<00:03, 43.35batch/s]

Training batches on cuda:  50%|█████     | 151/302 [00:03<00:03, 44.08batch/s]

Training batches on cuda:  52%|█████▏    | 156/302 [00:03<00:03, 44.76batch/s]

Training batches on cuda:  53%|█████▎    | 161/302 [00:03<00:03, 44.96batch/s]

Training batches on cuda:  55%|█████▍    | 166/302 [00:03<00:03, 43.74batch/s]

Training batches on cuda:  57%|█████▋    | 171/302 [00:03<00:03, 43.17batch/s]

Training batches on cuda:  58%|█████▊    | 176/302 [00:04<00:03, 41.98batch/s]

Training batches on cuda:  60%|█████▉    | 181/302 [00:04<00:02, 42.22batch/s]

Training batches on cuda:  62%|██████▏   | 186/302 [00:04<00:02, 42.61batch/s]

Training batches on cuda:  63%|██████▎   | 191/302 [00:04<00:02, 42.27batch/s]

Training batches on cuda:  65%|██████▍   | 196/302 [00:04<00:02, 41.53batch/s]

Training batches on cuda:  67%|██████▋   | 201/302 [00:04<00:02, 40.75batch/s]

Training batches on cuda:  68%|██████▊   | 206/302 [00:04<00:02, 40.80batch/s]

Training batches on cuda:  70%|██████▉   | 211/302 [00:04<00:02, 40.32batch/s]

Training batches on cuda:  72%|███████▏  | 216/302 [00:05<00:02, 40.07batch/s]

Training batches on cuda:  73%|███████▎  | 221/302 [00:05<00:02, 40.45batch/s]

Training batches on cuda:  75%|███████▍  | 226/302 [00:05<00:01, 40.43batch/s]

Training batches on cuda:  76%|███████▋  | 231/302 [00:05<00:01, 40.88batch/s]

Training batches on cuda:  78%|███████▊  | 236/302 [00:05<00:01, 41.37batch/s]

Training batches on cuda:  80%|███████▉  | 241/302 [00:05<00:01, 41.87batch/s]

Training batches on cuda:  81%|████████▏ | 246/302 [00:05<00:01, 42.59batch/s]

Training batches on cuda:  83%|████████▎ | 251/302 [00:05<00:01, 38.96batch/s]

Training batches on cuda:  85%|████████▍ | 256/302 [00:06<00:01, 39.80batch/s]

Training batches on cuda:  86%|████████▋ | 261/302 [00:06<00:01, 40.55batch/s]

Training batches on cuda:  88%|████████▊ | 266/302 [00:06<00:00, 42.28batch/s]

Training batches on cuda:  90%|████████▉ | 271/302 [00:06<00:00, 43.45batch/s]

Training batches on cuda:  91%|█████████▏| 276/302 [00:06<00:00, 42.94batch/s]

Training batches on cuda:  93%|█████████▎| 281/302 [00:06<00:00, 43.11batch/s]

Training batches on cuda:  95%|█████████▍| 286/302 [00:06<00:00, 42.43batch/s]

Training batches on cuda:  96%|█████████▋| 291/302 [00:06<00:00, 42.46batch/s]

Training batches on cuda:  98%|█████████▊| 296/302 [00:06<00:00, 42.71batch/s]

Training batches on cuda: 100%|█████████▉| 301/302 [00:07<00:00, 42.51batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:12<00:07,  7.29s/epoch, loss=0.00128, prev_loss=0.00128]2021-09-16:18:49:05,377 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:49:06,471 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:49:21,564 INFO     [evaluator.py:657] Evaluation took 15.09s seconds
2021-09-16:18:49:21,696 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00, 12.21s/epoch, loss=0.00128, prev_loss=0.00128]
Training epochs on cuda: 100%|██████████| 10/10 [01:29<00:00,  8.92s/epoch, loss=0.00128, prev_loss=0.00128]
2021-09-16:18:49:21,696 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp91kf4hrs'
2021-09-16:18:49:21,795 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp91kf4hrs' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.36ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.62ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.62ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.62ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.62ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:49:36,721 INFO     [evaluator.py:657] Evaluation took 14.91s seconds
[I 2021-09-16 18:49:36,726] Trial 58 finished with value: 0.004257488979701574 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.17393277454310851, 'optimizer.lr': 0.035464963249566106, 'negative_sampler.num_negs_per_pos': 86, 'training.batch_size': 1032}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:49:36,749 WARNING  [api.py:823] No random seed is specified. Setting to 572802776.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:04, 18.88batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:03, 20.92batch/s]

Training batches on cuda:  11%|█▏        | 9/80 [00:00<00:03, 22.98batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:02, 25.03batch/s]

Training batches on cuda:  20%|██        | 16/80 [00:00<00:02, 25.93batch/s]

Training batches on cuda:  24%|██▍       | 19/80 [00:00<00:02, 25.97batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:00<00:02, 26.75batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:00<00:02, 27.28batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:01, 27.63batch/s]

Training batches on cuda:  40%|████      | 32/80 [00:01<00:01, 28.89batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:01<00:01, 29.32batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:01<00:01, 29.04batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:01<00:01, 29.63batch/s]

Training batches on cuda:  57%|█████▊    | 46/80 [00:01<00:01, 29.41batch/s]

Training batches on cuda:  62%|██████▎   | 50/80 [00:01<00:01, 29.76batch/s]

Training batches on cuda:  68%|██████▊   | 54/80 [00:01<00:00, 30.46batch/s]

Training batches on cuda:  72%|███████▎  | 58/80 [00:02<00:00, 25.99batch/s]

Training batches on cuda:  76%|███████▋  | 61/80 [00:02<00:00, 25.99batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:02<00:00, 25.92batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:02<00:00, 27.46batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:02<00:00, 28.19batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:02<00:00, 28.08batch/s]

Training batches on cuda:  98%|█████████▊| 78/80 [00:02<00:00, 28.36batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=2.69e-6, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:26,  3.00s/epoch, loss=2.69e-6, prev_loss=nan]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:04, 18.36batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:03, 20.92batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:03, 23.33batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:02, 25.13batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:00<00:02, 26.06batch/s]

Training batches on cuda:  25%|██▌       | 20/80 [00:00<00:02, 26.90batch/s]

Training batches on cuda:  29%|██▉       | 23/80 [00:00<00:02, 26.34batch/s]

Training batches on cuda:  32%|███▎      | 26/80 [00:00<00:01, 27.25batch/s]

Training batches on cuda:  38%|███▊      | 30/80 [00:01<00:01, 28.30batch/s]

Training batches on cuda:  42%|████▎     | 34/80 [00:01<00:01, 28.90batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:01<00:01, 29.17batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:01<00:01, 30.02batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:01<00:01, 28.68batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:01<00:01, 28.52batch/s]

Training batches on cuda:  65%|██████▌   | 52/80 [00:01<00:01, 27.29batch/s]

Training batches on cuda:  70%|███████   | 56/80 [00:01<00:00, 28.83batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:02<00:00, 30.22batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:02<00:00, 31.01batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:02<00:00, 30.69batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:02<00:00, 31.09batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:02<00:00, 31.83batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:26,  3.00s/epoch, loss=7.04e-7, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:23,  2.96s/epoch, loss=7.04e-7, prev_loss=nan]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 3/80 [00:00<00:03, 24.09batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:02, 25.53batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:02, 27.49batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:02, 28.49batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:00<00:02, 28.65batch/s]

Training batches on cuda:  25%|██▌       | 20/80 [00:00<00:02, 28.34batch/s]

Training batches on cuda:  29%|██▉       | 23/80 [00:00<00:01, 28.71batch/s]

Training batches on cuda:  34%|███▍      | 27/80 [00:00<00:01, 29.67batch/s]

Training batches on cuda:  39%|███▉      | 31/80 [00:01<00:01, 30.15batch/s]

Training batches on cuda:  42%|████▎     | 34/80 [00:01<00:01, 27.76batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:01<00:01, 28.14batch/s]

Training batches on cuda:  50%|█████     | 40/80 [00:01<00:01, 28.16batch/s]

Training batches on cuda:  55%|█████▌    | 44/80 [00:01<00:01, 28.84batch/s]

Training batches on cuda:  60%|██████    | 48/80 [00:01<00:01, 29.48batch/s]

Training batches on cuda:  65%|██████▌   | 52/80 [00:01<00:00, 30.20batch/s]

Training batches on cuda:  70%|███████   | 56/80 [00:01<00:00, 30.46batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:02<00:00, 30.17batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:02<00:00, 30.36batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:02<00:00, 29.65batch/s]

Training batches on cuda:  89%|████████▉ | 71/80 [00:02<00:00, 29.52batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:02<00:00, 29.36batch/s]

Training batches on cuda:  98%|█████████▊| 78/80 [00:02<00:00, 27.23batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:23,  2.96s/epoch, loss=7.04e-7, prev_loss=7.04e-7]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:20,  2.98s/epoch, loss=7.04e-7, prev_loss=7.04e-7]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:04, 17.10batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:03, 19.59batch/s]

Training batches on cuda:  11%|█▏        | 9/80 [00:00<00:03, 22.15batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:02, 24.03batch/s]

Training batches on cuda:  20%|██        | 16/80 [00:00<00:02, 25.40batch/s]

Training batches on cuda:  24%|██▍       | 19/80 [00:00<00:02, 26.21batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:00<00:02, 26.37batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:00<00:02, 26.99batch/s]

Training batches on cuda:  36%|███▋      | 29/80 [00:01<00:01, 28.38batch/s]

Training batches on cuda:  41%|████▏     | 33/80 [00:01<00:01, 29.44batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:01<00:01, 29.22batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:01<00:01, 29.01batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:01<00:01, 29.77batch/s]

Training batches on cuda:  59%|█████▉    | 47/80 [00:01<00:01, 30.13batch/s]

Training batches on cuda:  64%|██████▍   | 51/80 [00:01<00:00, 30.44batch/s]

Training batches on cuda:  69%|██████▉   | 55/80 [00:01<00:00, 30.76batch/s]

Training batches on cuda:  74%|███████▍  | 59/80 [00:01<00:00, 30.99batch/s]

Training batches on cuda:  79%|███████▉  | 63/80 [00:02<00:00, 30.46batch/s]

Training batches on cuda:  84%|████████▍ | 67/80 [00:02<00:00, 30.10batch/s]

Training batches on cuda:  89%|████████▉ | 71/80 [00:02<00:00, 30.50batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:02<00:00, 30.77batch/s]

Training batches on cuda:  99%|█████████▉| 79/80 [00:02<00:00, 30.55batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:20,  2.98s/epoch, loss=7.16e-7, prev_loss=7.04e-7]
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:17,  2.95s/epoch, loss=7.16e-7, prev_loss=7.04e-7]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:04, 16.23batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:04, 18.44batch/s]

Training batches on cuda:  10%|█         | 8/80 [00:00<00:03, 19.52batch/s]

Training batches on cuda:  14%|█▍        | 11/80 [00:00<00:03, 21.64batch/s]

Training batches on cuda:  19%|█▉        | 15/80 [00:00<00:02, 23.80batch/s]

Training batches on cuda:  22%|██▎       | 18/80 [00:00<00:02, 25.12batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:00<00:02, 26.61batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:00<00:02, 26.85batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:01, 26.77batch/s]

Training batches on cuda:  40%|████      | 32/80 [00:01<00:01, 28.03batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:01<00:01, 28.92batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:01<00:01, 27.37batch/s]

Training batches on cuda:  52%|█████▎    | 42/80 [00:01<00:01, 26.65batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:01<00:01, 26.71batch/s]

Training batches on cuda:  60%|██████    | 48/80 [00:01<00:01, 27.41batch/s]

Training batches on cuda:  64%|██████▍   | 51/80 [00:01<00:01, 27.78batch/s]

Training batches on cuda:  69%|██████▉   | 55/80 [00:02<00:00, 28.53batch/s]

Training batches on cuda:  74%|███████▍  | 59/80 [00:02<00:00, 29.25batch/s]

Training batches on cuda:  78%|███████▊  | 62/80 [00:02<00:00, 28.92batch/s]

Training batches on cuda:  82%|████████▎ | 66/80 [00:02<00:00, 29.01batch/s]

Training batches on cuda:  86%|████████▋ | 69/80 [00:02<00:00, 29.19batch/s]

Training batches on cuda:  91%|█████████▏| 73/80 [00:02<00:00, 29.71batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:02<00:00, 29.41batch/s]

Training batches on cuda:  99%|█████████▉| 79/80 [00:02<00:00, 29.46batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:17,  2.95s/epoch, loss=7.49e-7, prev_loss=7.16e-7]
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:14,  2.99s/epoch, loss=7.49e-7, prev_loss=7.16e-7]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 3/80 [00:00<00:03, 22.92batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:03, 24.30batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:02, 25.58batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:02, 26.71batch/s]

Training batches on cuda:  20%|██        | 16/80 [00:00<00:02, 27.06batch/s]

Training batches on cuda:  24%|██▍       | 19/80 [00:00<00:02, 27.25batch/s]

Training batches on cuda:  28%|██▊       | 22/80 [00:00<00:02, 27.66batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:00<00:01, 27.64batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:01, 27.86batch/s]

Training batches on cuda:  39%|███▉      | 31/80 [00:01<00:01, 27.97batch/s]

Training batches on cuda:  42%|████▎     | 34/80 [00:01<00:01, 28.20batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:01<00:01, 28.27batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:01<00:01, 28.72batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:01<00:01, 29.25batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:01<00:01, 29.95batch/s]

Training batches on cuda:  66%|██████▋   | 53/80 [00:01<00:00, 30.46batch/s]

Training batches on cuda:  71%|███████▏  | 57/80 [00:01<00:00, 30.31batch/s]

Training batches on cuda:  76%|███████▋  | 61/80 [00:02<00:00, 30.86batch/s]

Training batches on cuda:  81%|████████▏ | 65/80 [00:02<00:00, 29.21batch/s]

Training batches on cuda:  86%|████████▋ | 69/80 [00:02<00:00, 29.48batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:02<00:00, 29.58batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:02<00:00, 30.10batch/s]

Training batches on cuda: 100%|██████████| 80/80 [00:02<00:00, 32.06batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:14,  2.99s/epoch, loss=7.9e-7, prev_loss=7.49e-7] 
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:11,  2.98s/epoch, loss=7.9e-7, prev_loss=7.49e-7]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:05, 14.91batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:04, 17.39batch/s]

Training batches on cuda:  11%|█▏        | 9/80 [00:00<00:03, 19.81batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:03, 21.62batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:00<00:02, 24.04batch/s]

Training batches on cuda:  26%|██▋       | 21/80 [00:00<00:02, 26.37batch/s]

Training batches on cuda:  30%|███       | 24/80 [00:00<00:02, 26.19batch/s]

Training batches on cuda:  35%|███▌      | 28/80 [00:01<00:01, 27.73batch/s]

Training batches on cuda:  40%|████      | 32/80 [00:01<00:01, 28.77batch/s]

Training batches on cuda:  45%|████▌     | 36/80 [00:01<00:01, 29.63batch/s]

Training batches on cuda:  50%|█████     | 40/80 [00:01<00:01, 29.44batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:01<00:01, 29.47batch/s]

Training batches on cuda:  59%|█████▉    | 47/80 [00:01<00:01, 30.27batch/s]

Training batches on cuda:  64%|██████▍   | 51/80 [00:01<00:00, 31.25batch/s]

Training batches on cuda:  69%|██████▉   | 55/80 [00:01<00:00, 30.14batch/s]

Training batches on cuda:  74%|███████▍  | 59/80 [00:02<00:00, 30.52batch/s]

Training batches on cuda:  79%|███████▉  | 63/80 [00:02<00:00, 29.76batch/s]

Training batches on cuda:  84%|████████▍ | 67/80 [00:02<00:00, 29.49batch/s]

Training batches on cuda:  89%|████████▉ | 71/80 [00:02<00:00, 30.12batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:02<00:00, 30.40batch/s]

Training batches on cuda:  99%|█████████▉| 79/80 [00:02<00:00, 30.96batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [00:20<00:11,  2.98s/epoch, loss=7.29e-7, prev_loss=7.9e-7]
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:08,  2.97s/epoch, loss=7.29e-7, prev_loss=7.9e-7]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:04, 18.75batch/s]

Training batches on cuda:   8%|▊         | 6/80 [00:00<00:03, 21.38batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:02, 23.93batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:02, 25.90batch/s]

Training batches on cuda:  22%|██▎       | 18/80 [00:00<00:02, 27.43batch/s]

Training batches on cuda:  26%|██▋       | 21/80 [00:00<00:02, 27.76batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:00<00:01, 28.65batch/s]

Training batches on cuda:  36%|███▋      | 29/80 [00:00<00:01, 29.15batch/s]

Training batches on cuda:  41%|████▏     | 33/80 [00:01<00:01, 29.90batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:01<00:01, 30.35batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:01<00:01, 28.72batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:01<00:01, 28.96batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:01<00:01, 29.43batch/s]

Training batches on cuda:  65%|██████▌   | 52/80 [00:01<00:00, 28.47batch/s]

Training batches on cuda:  70%|███████   | 56/80 [00:01<00:00, 29.42batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:02<00:00, 30.03batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:02<00:00, 29.57batch/s]

Training batches on cuda:  85%|████████▌ | 68/80 [00:02<00:00, 30.02batch/s]

Training batches on cuda:  90%|█████████ | 72/80 [00:02<00:00, 28.32batch/s]

Training batches on cuda:  94%|█████████▍| 75/80 [00:02<00:00, 27.69batch/s]

Training batches on cuda:  98%|█████████▊| 78/80 [00:02<00:00, 27.33batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [00:23<00:08,  2.97s/epoch, loss=7.76e-7, prev_loss=7.29e-7]
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:05,  2.96s/epoch, loss=7.76e-7, prev_loss=7.29e-7]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 1/80 [00:00<00:08,  9.48batch/s]

Training batches on cuda:   5%|▌         | 4/80 [00:00<00:06, 11.91batch/s]

Training batches on cuda:  10%|█         | 8/80 [00:00<00:04, 14.71batch/s]

Training batches on cuda:  14%|█▍        | 11/80 [00:00<00:04, 17.06batch/s]

Training batches on cuda:  18%|█▊        | 14/80 [00:00<00:03, 19.44batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:00<00:02, 21.60batch/s]

Training batches on cuda:  25%|██▌       | 20/80 [00:00<00:02, 23.31batch/s]

Training batches on cuda:  29%|██▉       | 23/80 [00:00<00:02, 24.96batch/s]

Training batches on cuda:  34%|███▍      | 27/80 [00:00<00:02, 26.28batch/s]

Training batches on cuda:  39%|███▉      | 31/80 [00:01<00:01, 27.49batch/s]

Training batches on cuda:  44%|████▍     | 35/80 [00:01<00:01, 28.37batch/s]

Training batches on cuda:  49%|████▉     | 39/80 [00:01<00:01, 28.89batch/s]

Training batches on cuda:  54%|█████▍    | 43/80 [00:01<00:01, 29.35batch/s]

Training batches on cuda:  57%|█████▊    | 46/80 [00:01<00:01, 28.53batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:01<00:01, 28.81batch/s]

Training batches on cuda:  66%|██████▋   | 53/80 [00:01<00:00, 29.80batch/s]

Training batches on cuda:  71%|███████▏  | 57/80 [00:01<00:00, 29.40batch/s]

Training batches on cuda:  75%|███████▌  | 60/80 [00:02<00:00, 28.79batch/s]

Training batches on cuda:  80%|████████  | 64/80 [00:02<00:00, 29.24batch/s]

Training batches on cuda:  84%|████████▍ | 67/80 [00:02<00:00, 29.44batch/s]

Training batches on cuda:  88%|████████▊ | 70/80 [00:02<00:00, 29.34batch/s]

Training batches on cuda:  91%|█████████▏| 73/80 [00:02<00:00, 28.93batch/s]

Training batches on cuda:  95%|█████████▌| 76/80 [00:02<00:00, 28.85batch/s]

Training batches on cuda: 100%|██████████| 80/80 [00:02<00:00, 31.16batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:05,  2.96s/epoch, loss=7.68e-7, prev_loss=7.76e-7]
Training epochs on cuda:  90%|█████████ | 9/10 [00:26<00:02,  2.97s/epoch, loss=7.68e-7, prev_loss=7.76e-7]

Training batches on cuda:   0%|          | 0/80 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▎         | 2/80 [00:00<00:04, 18.66batch/s]

Training batches on cuda:   6%|▋         | 5/80 [00:00<00:03, 20.85batch/s]

Training batches on cuda:  10%|█         | 8/80 [00:00<00:03, 22.91batch/s]

Training batches on cuda:  12%|█▎        | 10/80 [00:00<00:03, 21.65batch/s]

Training batches on cuda:  16%|█▋        | 13/80 [00:00<00:02, 22.46batch/s]

Training batches on cuda:  21%|██▏       | 17/80 [00:00<00:02, 24.19batch/s]

Training batches on cuda:  26%|██▋       | 21/80 [00:00<00:02, 26.37batch/s]

Training batches on cuda:  31%|███▏      | 25/80 [00:00<00:01, 28.09batch/s]

Training batches on cuda:  36%|███▋      | 29/80 [00:01<00:01, 29.28batch/s]

Training batches on cuda:  41%|████▏     | 33/80 [00:01<00:01, 30.40batch/s]

Training batches on cuda:  46%|████▋     | 37/80 [00:01<00:01, 30.92batch/s]

Training batches on cuda:  51%|█████▏    | 41/80 [00:01<00:01, 31.47batch/s]

Training batches on cuda:  56%|█████▋    | 45/80 [00:01<00:01, 32.01batch/s]

Training batches on cuda:  61%|██████▏   | 49/80 [00:01<00:00, 32.29batch/s]

Training batches on cuda:  66%|██████▋   | 53/80 [00:01<00:00, 32.57batch/s]

Training batches on cuda:  71%|███████▏  | 57/80 [00:01<00:00, 32.88batch/s]

Training batches on cuda:  76%|███████▋  | 61/80 [00:02<00:00, 32.90batch/s]

Training batches on cuda:  81%|████████▏ | 65/80 [00:02<00:00, 33.13batch/s]

Training batches on cuda:  86%|████████▋ | 69/80 [00:02<00:00, 33.01batch/s]

Training batches on cuda:  91%|█████████▏| 73/80 [00:02<00:00, 33.11batch/s]

Training batches on cuda:  96%|█████████▋| 77/80 [00:02<00:00, 32.84batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [00:29<00:02,  2.97s/epoch, loss=7.65e-7, prev_loss=7.68e-7]2021-09-16:18:50:06,796 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:50:07,841 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:50:22,940 INFO     [evaluator.py:657] Evaluation took 15.10s seconds
2021-09-16:18:50:23,81 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  7.81s/epoch, loss=7.65e-7, prev_loss=7.68e-7]
Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  4.58s/epoch, loss=7.65e-7, prev_loss=7.68e-7]
2021-09-16:18:50:23,81 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpy38v1hbk'
2021-09-16:18:50:23,146 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpy38v1hbk' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.36ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▉         | 3.58k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.59ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.60ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.60ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.61ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.62ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.61ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.61ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.62ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:50:38,63 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
[I 2021-09-16 18:50:38,068] Trial 59 finished with value: -0.0003238674773831107 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.050799821667851575, 'optimizer.lr': 0.051730502216242556, 'negative_sampler.num_negs_per_pos': 6, 'training.batch_size': 3932}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:50:38,91 WARNING  [api.py:823] No random seed is specified. Setting to 757466107.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/84 [00:00<00:04, 16.65batch/s]

Training batches on cuda:   7%|▋         | 6/84 [00:00<00:03, 20.16batch/s]

Training batches on cuda:  12%|█▏        | 10/84 [00:00<00:03, 23.21batch/s]

Training batches on cuda:  17%|█▋        | 14/84 [00:00<00:02, 26.08batch/s]

Training batches on cuda:  21%|██▏       | 18/84 [00:00<00:02, 28.40batch/s]

Training batches on cuda:  26%|██▌       | 22/84 [00:00<00:02, 30.49batch/s]

Training batches on cuda:  31%|███       | 26/84 [00:00<00:01, 32.45batch/s]

Training batches on cuda:  36%|███▌      | 30/84 [00:00<00:01, 33.56batch/s]

Training batches on cuda:  40%|████      | 34/84 [00:01<00:01, 31.51batch/s]

Training batches on cuda:  45%|████▌     | 38/84 [00:01<00:01, 30.28batch/s]

Training batches on cuda:  50%|█████     | 42/84 [00:01<00:01, 32.15batch/s]

Training batches on cuda:  55%|█████▍    | 46/84 [00:01<00:01, 32.67batch/s]

Training batches on cuda:  60%|█████▉    | 50/84 [00:01<00:00, 34.38batch/s]

Training batches on cuda:  64%|██████▍   | 54/84 [00:01<00:00, 35.33batch/s]

Training batches on cuda:  69%|██████▉   | 58/84 [00:01<00:00, 35.42batch/s]

Training batches on cuda:  74%|███████▍  | 62/84 [00:01<00:00, 35.78batch/s]

Training batches on cuda:  79%|███████▊  | 66/84 [00:01<00:00, 35.87batch/s]

Training batches on cuda:  83%|████████▎ | 70/84 [00:02<00:00, 35.92batch/s]

Training batches on cuda:  88%|████████▊ | 74/84 [00:02<00:00, 35.43batch/s]

Training batches on cuda:  93%|█████████▎| 78/84 [00:02<00:00, 36.26batch/s]

Training batches on cuda:  98%|█████████▊| 82/84 [00:02<00:00, 35.87batch/s]

                                                                            
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000576, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:23,  2.64s/epoch, loss=0.000576, prev_loss=nan]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/84 [00:00<00:04, 19.15batch/s]

Training batches on cuda:   7%|▋         | 6/84 [00:00<00:03, 22.11batch/s]

Training batches on cuda:  12%|█▏        | 10/84 [00:00<00:03, 24.45batch/s]

Training batches on cuda:  17%|█▋        | 14/84 [00:00<00:02, 27.19batch/s]

Training batches on cuda:  21%|██▏       | 18/84 [00:00<00:02, 28.75batch/s]

Training batches on cuda:  26%|██▌       | 22/84 [00:00<00:02, 29.70batch/s]

Training batches on cuda:  30%|██▉       | 25/84 [00:00<00:02, 27.77batch/s]

Training batches on cuda:  35%|███▍      | 29/84 [00:00<00:01, 29.41batch/s]

Training batches on cuda:  39%|███▉      | 33/84 [00:01<00:01, 31.56batch/s]

Training batches on cuda:  44%|████▍     | 37/84 [00:01<00:01, 33.26batch/s]

Training batches on cuda:  49%|████▉     | 41/84 [00:01<00:01, 34.62batch/s]

Training batches on cuda:  54%|█████▎    | 45/84 [00:01<00:01, 34.51batch/s]

Training batches on cuda:  58%|█████▊    | 49/84 [00:01<00:00, 35.30batch/s]

Training batches on cuda:  63%|██████▎   | 53/84 [00:01<00:00, 36.30batch/s]

Training batches on cuda:  68%|██████▊   | 57/84 [00:01<00:00, 36.49batch/s]

Training batches on cuda:  73%|███████▎  | 61/84 [00:01<00:00, 36.20batch/s]

Training batches on cuda:  77%|███████▋  | 65/84 [00:01<00:00, 36.68batch/s]

Training batches on cuda:  82%|████████▏ | 69/84 [00:02<00:00, 37.04batch/s]

Training batches on cuda:  87%|████████▋ | 73/84 [00:02<00:00, 37.19batch/s]

Training batches on cuda:  92%|█████████▏| 77/84 [00:02<00:00, 37.66batch/s]

Training batches on cuda:  96%|█████████▋| 81/84 [00:02<00:00, 36.95batch/s]

                                                                            
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:23,  2.64s/epoch, loss=0.000571, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:21,  2.66s/epoch, loss=0.000571, prev_loss=nan]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/84 [00:00<00:04, 18.48batch/s]

Training batches on cuda:   7%|▋         | 6/84 [00:00<00:03, 21.66batch/s]

Training batches on cuda:  12%|█▏        | 10/84 [00:00<00:03, 24.60batch/s]

Training batches on cuda:  15%|█▌        | 13/84 [00:00<00:02, 25.83batch/s]

Training batches on cuda:  20%|██        | 17/84 [00:00<00:02, 28.02batch/s]

Training batches on cuda:  24%|██▍       | 20/84 [00:00<00:02, 26.33batch/s]

Training batches on cuda:  29%|██▊       | 24/84 [00:00<00:02, 28.91batch/s]

Training batches on cuda:  33%|███▎      | 28/84 [00:00<00:01, 30.41batch/s]

Training batches on cuda:  38%|███▊      | 32/84 [00:01<00:01, 32.36batch/s]

Training batches on cuda:  43%|████▎     | 36/84 [00:01<00:01, 32.98batch/s]

Training batches on cuda:  48%|████▊     | 40/84 [00:01<00:01, 33.01batch/s]

Training batches on cuda:  52%|█████▏    | 44/84 [00:01<00:01, 32.56batch/s]

Training batches on cuda:  57%|█████▋    | 48/84 [00:01<00:01, 31.49batch/s]

Training batches on cuda:  62%|██████▏   | 52/84 [00:01<00:00, 33.09batch/s]

Training batches on cuda:  67%|██████▋   | 56/84 [00:01<00:00, 33.91batch/s]

Training batches on cuda:  71%|███████▏  | 60/84 [00:01<00:00, 33.78batch/s]

Training batches on cuda:  76%|███████▌  | 64/84 [00:01<00:00, 34.52batch/s]

Training batches on cuda:  81%|████████  | 68/84 [00:02<00:00, 35.11batch/s]

Training batches on cuda:  86%|████████▌ | 72/84 [00:02<00:00, 36.04batch/s]

Training batches on cuda:  90%|█████████ | 76/84 [00:02<00:00, 36.38batch/s]

Training batches on cuda:  95%|█████████▌| 80/84 [00:02<00:00, 37.13batch/s]

                                                                            
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:21,  2.66s/epoch, loss=0.000568, prev_loss=0.000571]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:18,  2.67s/epoch, loss=0.000568, prev_loss=0.000571]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/84 [00:00<00:05, 16.00batch/s]

Training batches on cuda:   7%|▋         | 6/84 [00:00<00:04, 19.17batch/s]

Training batches on cuda:  12%|█▏        | 10/84 [00:00<00:03, 22.42batch/s]

Training batches on cuda:  17%|█▋        | 14/84 [00:00<00:02, 25.33batch/s]

Training batches on cuda:  21%|██▏       | 18/84 [00:00<00:02, 26.89batch/s]

Training batches on cuda:  26%|██▌       | 22/84 [00:00<00:02, 28.72batch/s]

Training batches on cuda:  31%|███       | 26/84 [00:00<00:01, 30.04batch/s]

Training batches on cuda:  36%|███▌      | 30/84 [00:00<00:01, 31.99batch/s]

Training batches on cuda:  40%|████      | 34/84 [00:01<00:01, 33.08batch/s]

Training batches on cuda:  45%|████▌     | 38/84 [00:01<00:01, 34.14batch/s]

Training batches on cuda:  50%|█████     | 42/84 [00:01<00:01, 34.88batch/s]

Training batches on cuda:  55%|█████▍    | 46/84 [00:01<00:01, 35.19batch/s]

Training batches on cuda:  60%|█████▉    | 50/84 [00:01<00:00, 36.15batch/s]

Training batches on cuda:  64%|██████▍   | 54/84 [00:01<00:00, 37.00batch/s]

Training batches on cuda:  69%|██████▉   | 58/84 [00:01<00:00, 37.54batch/s]

Training batches on cuda:  74%|███████▍  | 62/84 [00:01<00:00, 37.33batch/s]

Training batches on cuda:  79%|███████▊  | 66/84 [00:01<00:00, 37.39batch/s]

Training batches on cuda:  83%|████████▎ | 70/84 [00:01<00:00, 37.42batch/s]

Training batches on cuda:  88%|████████▊ | 74/84 [00:02<00:00, 37.56batch/s]

Training batches on cuda:  93%|█████████▎| 78/84 [00:02<00:00, 37.52batch/s]

Training batches on cuda:  98%|█████████▊| 82/84 [00:02<00:00, 37.19batch/s]

                                                                            
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:18,  2.67s/epoch, loss=0.000515, prev_loss=0.000568]
Training epochs on cuda:  40%|████      | 4/10 [00:10<00:15,  2.65s/epoch, loss=0.000515, prev_loss=0.000568]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 3/84 [00:00<00:03, 26.55batch/s]

Training batches on cuda:   8%|▊         | 7/84 [00:00<00:02, 29.04batch/s]

Training batches on cuda:  13%|█▎        | 11/84 [00:00<00:02, 31.15batch/s]

Training batches on cuda:  18%|█▊        | 15/84 [00:00<00:02, 32.79batch/s]

Training batches on cuda:  23%|██▎       | 19/84 [00:00<00:01, 34.45batch/s]

Training batches on cuda:  27%|██▋       | 23/84 [00:00<00:01, 34.55batch/s]

Training batches on cuda:  32%|███▏      | 27/84 [00:00<00:01, 34.52batch/s]

Training batches on cuda:  37%|███▋      | 31/84 [00:00<00:01, 35.21batch/s]

Training batches on cuda:  42%|████▏     | 35/84 [00:00<00:01, 35.85batch/s]

Training batches on cuda:  46%|████▋     | 39/84 [00:01<00:01, 36.90batch/s]

Training batches on cuda:  51%|█████     | 43/84 [00:01<00:01, 36.40batch/s]

Training batches on cuda:  56%|█████▌    | 47/84 [00:01<00:01, 35.53batch/s]

Training batches on cuda:  61%|██████    | 51/84 [00:01<00:00, 34.69batch/s]

Training batches on cuda:  65%|██████▌   | 55/84 [00:01<00:00, 34.15batch/s]

Training batches on cuda:  70%|███████   | 59/84 [00:01<00:00, 34.05batch/s]

Training batches on cuda:  75%|███████▌  | 63/84 [00:01<00:00, 33.63batch/s]

Training batches on cuda:  80%|███████▉  | 67/84 [00:01<00:00, 33.68batch/s]

Training batches on cuda:  85%|████████▍ | 71/84 [00:02<00:00, 33.81batch/s]

Training batches on cuda:  89%|████████▉ | 75/84 [00:02<00:00, 33.91batch/s]

Training batches on cuda:  94%|█████████▍| 79/84 [00:02<00:00, 33.14batch/s]

Training batches on cuda:  99%|█████████▉| 83/84 [00:02<00:00, 33.48batch/s]

                                                                            
Training epochs on cuda:  40%|████      | 4/10 [00:13<00:15,  2.65s/epoch, loss=0.000436, prev_loss=0.000515]
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:13,  2.64s/epoch, loss=0.000436, prev_loss=0.000515]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/84 [00:00<00:04, 17.80batch/s]

Training batches on cuda:   7%|▋         | 6/84 [00:00<00:03, 21.16batch/s]

Training batches on cuda:  12%|█▏        | 10/84 [00:00<00:03, 24.16batch/s]

Training batches on cuda:  17%|█▋        | 14/84 [00:00<00:02, 26.33batch/s]

Training batches on cuda:  21%|██▏       | 18/84 [00:00<00:02, 29.10batch/s]

Training batches on cuda:  26%|██▌       | 22/84 [00:00<00:01, 31.35batch/s]

Training batches on cuda:  31%|███       | 26/84 [00:00<00:01, 32.85batch/s]

Training batches on cuda:  36%|███▌      | 30/84 [00:00<00:01, 34.20batch/s]

Training batches on cuda:  42%|████▏     | 35/84 [00:00<00:01, 35.72batch/s]

Training batches on cuda:  46%|████▋     | 39/84 [00:01<00:01, 34.52batch/s]

Training batches on cuda:  51%|█████     | 43/84 [00:01<00:01, 35.50batch/s]

Training batches on cuda:  56%|█████▌    | 47/84 [00:01<00:01, 36.40batch/s]

Training batches on cuda:  61%|██████    | 51/84 [00:01<00:00, 36.70batch/s]

Training batches on cuda:  65%|██████▌   | 55/84 [00:01<00:00, 36.26batch/s]

Training batches on cuda:  70%|███████   | 59/84 [00:01<00:00, 36.40batch/s]

Training batches on cuda:  75%|███████▌  | 63/84 [00:01<00:00, 36.26batch/s]

Training batches on cuda:  80%|███████▉  | 67/84 [00:01<00:00, 35.14batch/s]

Training batches on cuda:  85%|████████▍ | 71/84 [00:02<00:00, 34.49batch/s]

Training batches on cuda:  89%|████████▉ | 75/84 [00:02<00:00, 34.17batch/s]

Training batches on cuda:  94%|█████████▍| 79/84 [00:02<00:00, 35.37batch/s]

Training batches on cuda:  99%|█████████▉| 83/84 [00:02<00:00, 35.71batch/s]

                                                                            
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:13,  2.64s/epoch, loss=0.000399, prev_loss=0.000436]
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:10,  2.67s/epoch, loss=0.000399, prev_loss=0.000436]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/84 [00:00<00:04, 19.55batch/s]

Training batches on cuda:   7%|▋         | 6/84 [00:00<00:03, 21.99batch/s]

Training batches on cuda:  12%|█▏        | 10/84 [00:00<00:02, 25.23batch/s]

Training batches on cuda:  18%|█▊        | 15/84 [00:00<00:02, 28.63batch/s]

Training batches on cuda:  24%|██▍       | 20/84 [00:00<00:01, 32.01batch/s]

Training batches on cuda:  30%|██▉       | 25/84 [00:00<00:01, 34.84batch/s]

Training batches on cuda:  36%|███▌      | 30/84 [00:00<00:01, 36.95batch/s]

Training batches on cuda:  42%|████▏     | 35/84 [00:00<00:01, 38.38batch/s]

Training batches on cuda:  48%|████▊     | 40/84 [00:01<00:01, 39.49batch/s]

Training batches on cuda:  54%|█████▎    | 45/84 [00:01<00:00, 40.75batch/s]

Training batches on cuda:  60%|█████▉    | 50/84 [00:01<00:00, 42.06batch/s]

Training batches on cuda:  65%|██████▌   | 55/84 [00:01<00:00, 42.35batch/s]

Training batches on cuda:  71%|███████▏  | 60/84 [00:01<00:00, 43.20batch/s]

Training batches on cuda:  77%|███████▋  | 65/84 [00:01<00:00, 40.20batch/s]

Training batches on cuda:  83%|████████▎ | 70/84 [00:01<00:00, 40.88batch/s]

Training batches on cuda:  89%|████████▉ | 75/84 [00:01<00:00, 41.04batch/s]

Training batches on cuda:  95%|█████████▌| 80/84 [00:02<00:00, 39.57batch/s]

Training batches on cuda: 100%|██████████| 84/84 [00:02<00:00, 38.25batch/s]

                                                                            
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:10,  2.67s/epoch, loss=0.000353, prev_loss=0.000399]
Training epochs on cuda:  70%|███████   | 7/10 [00:18<00:07,  2.56s/epoch, loss=0.000353, prev_loss=0.000399]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/84 [00:00<00:04, 16.70batch/s]

Training batches on cuda:   7%|▋         | 6/84 [00:00<00:03, 19.98batch/s]

Training batches on cuda:  12%|█▏        | 10/84 [00:00<00:03, 22.37batch/s]

Training batches on cuda:  17%|█▋        | 14/84 [00:00<00:02, 24.91batch/s]

Training batches on cuda:  21%|██▏       | 18/84 [00:00<00:02, 27.89batch/s]

Training batches on cuda:  26%|██▌       | 22/84 [00:00<00:02, 29.20batch/s]

Training batches on cuda:  31%|███       | 26/84 [00:00<00:01, 31.72batch/s]

Training batches on cuda:  37%|███▋      | 31/84 [00:00<00:01, 33.87batch/s]

Training batches on cuda:  42%|████▏     | 35/84 [00:01<00:01, 34.94batch/s]

Training batches on cuda:  46%|████▋     | 39/84 [00:01<00:01, 35.47batch/s]

Training batches on cuda:  51%|█████     | 43/84 [00:01<00:01, 36.09batch/s]

Training batches on cuda:  56%|█████▌    | 47/84 [00:01<00:01, 36.40batch/s]

Training batches on cuda:  61%|██████    | 51/84 [00:01<00:00, 36.92batch/s]

Training batches on cuda:  65%|██████▌   | 55/84 [00:01<00:00, 36.84batch/s]

Training batches on cuda:  70%|███████   | 59/84 [00:01<00:00, 37.03batch/s]

Training batches on cuda:  75%|███████▌  | 63/84 [00:01<00:00, 37.27batch/s]

Training batches on cuda:  80%|███████▉  | 67/84 [00:01<00:00, 37.08batch/s]

Training batches on cuda:  85%|████████▍ | 71/84 [00:01<00:00, 37.56batch/s]

Training batches on cuda:  89%|████████▉ | 75/84 [00:02<00:00, 37.12batch/s]

Training batches on cuda:  94%|█████████▍| 79/84 [00:02<00:00, 37.46batch/s]

Training batches on cuda:  99%|█████████▉| 83/84 [00:02<00:00, 36.83batch/s]

                                                                            
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:07,  2.56s/epoch, loss=0.000317, prev_loss=0.000353]
Training epochs on cuda:  80%|████████  | 8/10 [00:20<00:05,  2.58s/epoch, loss=0.000317, prev_loss=0.000353]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/84 [00:00<00:04, 19.86batch/s]

Training batches on cuda:   7%|▋         | 6/84 [00:00<00:03, 22.84batch/s]

Training batches on cuda:  12%|█▏        | 10/84 [00:00<00:02, 25.78batch/s]

Training batches on cuda:  17%|█▋        | 14/84 [00:00<00:02, 27.99batch/s]

Training batches on cuda:  21%|██▏       | 18/84 [00:00<00:02, 28.98batch/s]

Training batches on cuda:  26%|██▌       | 22/84 [00:00<00:02, 29.52batch/s]

Training batches on cuda:  31%|███       | 26/84 [00:00<00:01, 31.60batch/s]

Training batches on cuda:  36%|███▌      | 30/84 [00:00<00:01, 33.16batch/s]

Training batches on cuda:  40%|████      | 34/84 [00:01<00:01, 34.79batch/s]

Training batches on cuda:  45%|████▌     | 38/84 [00:01<00:01, 35.24batch/s]

Training batches on cuda:  50%|█████     | 42/84 [00:01<00:01, 35.85batch/s]

Training batches on cuda:  55%|█████▍    | 46/84 [00:01<00:01, 36.66batch/s]

Training batches on cuda:  60%|█████▉    | 50/84 [00:01<00:00, 37.38batch/s]

Training batches on cuda:  64%|██████▍   | 54/84 [00:01<00:00, 37.43batch/s]

Training batches on cuda:  69%|██████▉   | 58/84 [00:01<00:00, 37.68batch/s]

Training batches on cuda:  74%|███████▍  | 62/84 [00:01<00:00, 37.93batch/s]

Training batches on cuda:  79%|███████▊  | 66/84 [00:01<00:00, 38.09batch/s]

Training batches on cuda:  83%|████████▎ | 70/84 [00:01<00:00, 37.90batch/s]

Training batches on cuda:  88%|████████▊ | 74/84 [00:02<00:00, 38.13batch/s]

Training batches on cuda:  93%|█████████▎| 78/84 [00:02<00:00, 37.87batch/s]

Training batches on cuda:  98%|█████████▊| 82/84 [00:02<00:00, 37.43batch/s]

                                                                            
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:05,  2.58s/epoch, loss=0.000306, prev_loss=0.000317]
Training epochs on cuda:  90%|█████████ | 9/10 [00:23<00:02,  2.58s/epoch, loss=0.000306, prev_loss=0.000317]

Training batches on cuda:   0%|          | 0/84 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 3/84 [00:00<00:03, 25.92batch/s]

Training batches on cuda:   8%|▊         | 7/84 [00:00<00:02, 28.16batch/s]

Training batches on cuda:  13%|█▎        | 11/84 [00:00<00:02, 30.40batch/s]

Training batches on cuda:  18%|█▊        | 15/84 [00:00<00:02, 30.77batch/s]

Training batches on cuda:  23%|██▎       | 19/84 [00:00<00:02, 31.66batch/s]

Training batches on cuda:  27%|██▋       | 23/84 [00:00<00:01, 32.02batch/s]

Training batches on cuda:  32%|███▏      | 27/84 [00:00<00:01, 32.08batch/s]

Training batches on cuda:  37%|███▋      | 31/84 [00:00<00:01, 33.25batch/s]

Training batches on cuda:  42%|████▏     | 35/84 [00:01<00:01, 33.87batch/s]

Training batches on cuda:  46%|████▋     | 39/84 [00:01<00:01, 35.23batch/s]

Training batches on cuda:  51%|█████     | 43/84 [00:01<00:01, 36.11batch/s]

Training batches on cuda:  56%|█████▌    | 47/84 [00:01<00:01, 34.10batch/s]

Training batches on cuda:  61%|██████    | 51/84 [00:01<00:00, 34.65batch/s]

Training batches on cuda:  65%|██████▌   | 55/84 [00:01<00:00, 35.47batch/s]

Training batches on cuda:  70%|███████   | 59/84 [00:01<00:00, 35.86batch/s]

Training batches on cuda:  75%|███████▌  | 63/84 [00:01<00:00, 35.95batch/s]

Training batches on cuda:  80%|███████▉  | 67/84 [00:01<00:00, 36.48batch/s]

Training batches on cuda:  85%|████████▍ | 71/84 [00:02<00:00, 36.84batch/s]

Training batches on cuda:  89%|████████▉ | 75/84 [00:02<00:00, 36.72batch/s]

Training batches on cuda:  94%|█████████▍| 79/84 [00:02<00:00, 36.24batch/s]

Training batches on cuda:  99%|█████████▉| 83/84 [00:02<00:00, 36.73batch/s]

                                                                            
Training epochs on cuda:  90%|█████████ | 9/10 [00:26<00:02,  2.58s/epoch, loss=0.000289, prev_loss=0.000306]2021-09-16:18:51:04,704 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:51:05,820 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:51:20,829 INFO     [evaluator.py:657] Evaluation took 15.01s seconds
2021-09-16:18:51:20,995 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:42<00:00,  7.47s/epoch, loss=0.000289, prev_loss=0.000306]
Training epochs on cuda: 100%|██████████| 10/10 [00:42<00:00,  4.24s/epoch, loss=0.000289, prev_loss=0.000306]
2021-09-16:18:51:20,996 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpcxp85mh1'
2021-09-16:18:51:21,56 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpcxp85mh1' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.61ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.51ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.55ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.57ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.59ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.60ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.61ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:51:35,961 INFO     [evaluator.py:657] Evaluation took 14.88s seconds
[I 2021-09-16 18:51:35,967] Trial 60 finished with value: 0.9272547184634925 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.015616141157863058, 'optimizer.lr': 0.042218799564252465, 'negative_sampler.num_negs_per_pos': 4, 'training.batch_size': 3732}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:51:35,989 WARNING  [api.py:823] No random seed is specified. Setting to 2378242452.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/153 [00:00<00:07, 19.66batch/s]

Training batches on cuda:   5%|▍         | 7/153 [00:00<00:06, 24.02batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:04, 29.38batch/s]

Training batches on cuda:  14%|█▎        | 21/153 [00:00<00:03, 35.57batch/s]

Training batches on cuda:  18%|█▊        | 28/153 [00:00<00:03, 41.65batch/s]

Training batches on cuda:  23%|██▎       | 35/153 [00:00<00:02, 47.11batch/s]

Training batches on cuda:  27%|██▋       | 42/153 [00:00<00:02, 52.05batch/s]

Training batches on cuda:  32%|███▏      | 49/153 [00:00<00:01, 56.14batch/s]

Training batches on cuda:  37%|███▋      | 56/153 [00:00<00:01, 59.16batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:01, 61.27batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:01, 63.54batch/s]

Training batches on cuda:  50%|█████     | 77/153 [00:01<00:01, 64.78batch/s]

Training batches on cuda:  56%|█████▌    | 85/153 [00:01<00:01, 66.23batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:01<00:00, 67.24batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:01<00:00, 67.18batch/s]

Training batches on cuda:  70%|██████▉   | 107/153 [00:01<00:00, 67.94batch/s]

Training batches on cuda:  75%|███████▍  | 114/153 [00:01<00:00, 67.53batch/s]

Training batches on cuda:  79%|███████▉  | 121/153 [00:01<00:00, 67.50batch/s]

Training batches on cuda:  84%|████████▎ | 128/153 [00:01<00:00, 67.64batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:02<00:00, 67.82batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:02<00:00, 66.04batch/s]

Training batches on cuda:  97%|█████████▋| 149/153 [00:02<00:00, 65.45batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000647, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:23,  2.59s/epoch, loss=0.000647, prev_loss=nan]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 26.72batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:04, 31.32batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:03, 35.53batch/s]

Training batches on cuda:  13%|█▎        | 20/153 [00:00<00:03, 38.84batch/s]

Training batches on cuda:  16%|█▌        | 24/153 [00:00<00:03, 39.07batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:02, 41.96batch/s]

Training batches on cuda:  23%|██▎       | 35/153 [00:00<00:02, 43.88batch/s]

Training batches on cuda:  27%|██▋       | 41/153 [00:00<00:02, 46.46batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:00<00:02, 45.31batch/s]

Training batches on cuda:  33%|███▎      | 51/153 [00:01<00:02, 46.30batch/s]

Training batches on cuda:  37%|███▋      | 57/153 [00:01<00:01, 49.46batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:01, 51.41batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:01, 53.21batch/s]

Training batches on cuda:  50%|████▉     | 76/153 [00:01<00:01, 51.60batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 52.22batch/s]

Training batches on cuda:  58%|█████▊    | 88/153 [00:01<00:01, 53.30batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:01<00:01, 54.14batch/s]

Training batches on cuda:  65%|██████▌   | 100/153 [00:01<00:00, 53.31batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:02<00:00, 52.88batch/s]

Training batches on cuda:  74%|███████▍  | 113/153 [00:02<00:00, 54.91batch/s]

Training batches on cuda:  78%|███████▊  | 119/153 [00:02<00:00, 56.14batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:02<00:00, 58.17batch/s]

Training batches on cuda:  87%|████████▋ | 133/153 [00:02<00:00, 60.87batch/s]

Training batches on cuda:  92%|█████████▏| 140/153 [00:02<00:00, 63.15batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:02<00:00, 64.91batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:23,  2.59s/epoch, loss=0.000642, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:21,  2.73s/epoch, loss=0.000642, prev_loss=nan]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 27.84batch/s]

Training batches on cuda:   5%|▌         | 8/153 [00:00<00:04, 31.95batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:03, 36.49batch/s]

Training batches on cuda:  13%|█▎        | 20/153 [00:00<00:03, 40.74batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:02, 44.44batch/s]

Training batches on cuda:  22%|██▏       | 33/153 [00:00<00:02, 48.61batch/s]

Training batches on cuda:  26%|██▌       | 40/153 [00:00<00:02, 51.97batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:00<00:02, 52.37batch/s]

Training batches on cuda:  34%|███▍      | 52/153 [00:01<00:02, 46.14batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:01, 48.35batch/s]

Training batches on cuda:  41%|████      | 63/153 [00:01<00:01, 45.95batch/s]

Training batches on cuda:  45%|████▌     | 69/153 [00:01<00:01, 47.65batch/s]

Training batches on cuda:  50%|████▉     | 76/153 [00:01<00:01, 51.04batch/s]

Training batches on cuda:  54%|█████▍    | 83/153 [00:01<00:01, 53.52batch/s]

Training batches on cuda:  58%|█████▊    | 89/153 [00:01<00:01, 53.19batch/s]

Training batches on cuda:  63%|██████▎   | 96/153 [00:01<00:01, 56.49batch/s]

Training batches on cuda:  67%|██████▋   | 103/153 [00:01<00:00, 58.68batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:02<00:00, 60.20batch/s]

Training batches on cuda:  76%|███████▋  | 117/153 [00:02<00:00, 61.71batch/s]

Training batches on cuda:  81%|████████  | 124/153 [00:02<00:00, 60.98batch/s]

Training batches on cuda:  86%|████████▌ | 131/153 [00:02<00:00, 57.72batch/s]

Training batches on cuda:  90%|████████▉ | 137/153 [00:02<00:00, 57.00batch/s]

Training batches on cuda:  93%|█████████▎| 143/153 [00:02<00:00, 53.65batch/s]

Training batches on cuda:  97%|█████████▋| 149/153 [00:02<00:00, 54.62batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:21,  2.73s/epoch, loss=0.00063, prev_loss=0.000642]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:19,  2.84s/epoch, loss=0.00063, prev_loss=0.000642]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 35.24batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:03, 38.32batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:03, 40.14batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:03, 39.93batch/s]

Training batches on cuda:  16%|█▌        | 24/153 [00:00<00:02, 43.65batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:02, 47.20batch/s]

Training batches on cuda:  24%|██▎       | 36/153 [00:00<00:02, 50.22batch/s]

Training batches on cuda:  27%|██▋       | 41/153 [00:00<00:02, 50.02batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:00<00:02, 49.12batch/s]

Training batches on cuda:  35%|███▍      | 53/153 [00:01<00:01, 53.75batch/s]

Training batches on cuda:  39%|███▊      | 59/153 [00:01<00:01, 54.31batch/s]

Training batches on cuda:  42%|████▏     | 65/153 [00:01<00:01, 51.22batch/s]

Training batches on cuda:  46%|████▋     | 71/153 [00:01<00:01, 53.51batch/s]

Training batches on cuda:  51%|█████     | 78/153 [00:01<00:01, 56.05batch/s]

Training batches on cuda:  55%|█████▍    | 84/153 [00:01<00:01, 56.73batch/s]

Training batches on cuda:  59%|█████▉    | 91/153 [00:01<00:01, 57.31batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:01<00:00, 59.17batch/s]

Training batches on cuda:  69%|██████▊   | 105/153 [00:01<00:00, 59.71batch/s]

Training batches on cuda:  73%|███████▎  | 112/153 [00:02<00:00, 59.86batch/s]

Training batches on cuda:  78%|███████▊  | 119/153 [00:02<00:00, 59.99batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:02<00:00, 58.27batch/s]

Training batches on cuda:  86%|████████▋ | 132/153 [00:02<00:00, 58.25batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:02<00:00, 57.17batch/s]

Training batches on cuda:  94%|█████████▍| 144/153 [00:02<00:00, 56.82batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:02<00:00, 56.35batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:19,  2.84s/epoch, loss=0.000616, prev_loss=0.00063]
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:17,  2.89s/epoch, loss=0.000616, prev_loss=0.00063]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 28.91batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:04, 34.19batch/s]

Training batches on cuda:  10%|█         | 16/153 [00:00<00:03, 39.39batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:03, 42.74batch/s]

Training batches on cuda:  19%|█▉        | 29/153 [00:00<00:02, 46.71batch/s]

Training batches on cuda:  24%|██▎       | 36/153 [00:00<00:02, 50.95batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:00<00:02, 53.93batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:00<00:01, 56.01batch/s]

Training batches on cuda:  37%|███▋      | 56/153 [00:00<00:01, 55.51batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:01, 56.75batch/s]

Training batches on cuda:  44%|████▍     | 68/153 [00:01<00:01, 57.32batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 57.63batch/s]

Training batches on cuda:  53%|█████▎    | 81/153 [00:01<00:01, 59.67batch/s]

Training batches on cuda:  58%|█████▊    | 88/153 [00:01<00:01, 60.94batch/s]

Training batches on cuda:  62%|██████▏   | 95/153 [00:01<00:00, 60.95batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:01<00:00, 61.94batch/s]

Training batches on cuda:  71%|███████   | 109/153 [00:01<00:00, 55.37batch/s]

Training batches on cuda:  76%|███████▌  | 116/153 [00:02<00:00, 58.21batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:02<00:00, 60.86batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:02<00:00, 61.41batch/s]

Training batches on cuda:  90%|████████▉ | 137/153 [00:02<00:00, 61.72batch/s]

Training batches on cuda:  94%|█████████▍| 144/153 [00:02<00:00, 58.07batch/s]

Training batches on cuda:  98%|█████████▊| 150/153 [00:02<00:00, 57.52batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:17,  2.89s/epoch, loss=0.00054, prev_loss=0.000616]
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:14,  2.88s/epoch, loss=0.00054, prev_loss=0.000616]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 29.02batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:04, 34.04batch/s]

Training batches on cuda:  10%|█         | 16/153 [00:00<00:03, 39.48batch/s]

Training batches on cuda:  15%|█▌        | 23/153 [00:00<00:02, 44.69batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:02, 48.23batch/s]

Training batches on cuda:  24%|██▎       | 36/153 [00:00<00:02, 50.55batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:00<00:02, 54.66batch/s]

Training batches on cuda:  33%|███▎      | 51/153 [00:00<00:01, 58.62batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:00<00:01, 60.01batch/s]

Training batches on cuda:  43%|████▎     | 66/153 [00:01<00:01, 62.85batch/s]

Training batches on cuda:  48%|████▊     | 73/153 [00:01<00:01, 64.06batch/s]

Training batches on cuda:  52%|█████▏    | 80/153 [00:01<00:01, 65.33batch/s]

Training batches on cuda:  57%|█████▋    | 87/153 [00:01<00:01, 65.29batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:01<00:00, 66.59batch/s]

Training batches on cuda:  66%|██████▌   | 101/153 [00:01<00:00, 67.48batch/s]

Training batches on cuda:  71%|███████   | 108/153 [00:01<00:00, 67.63batch/s]

Training batches on cuda:  76%|███████▌  | 116/153 [00:01<00:00, 68.48batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:01<00:00, 68.68batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:02<00:00, 68.59batch/s]

Training batches on cuda:  90%|████████▉ | 137/153 [00:02<00:00, 68.58batch/s]

Training batches on cuda:  94%|█████████▍| 144/153 [00:02<00:00, 68.73batch/s]

Training batches on cuda:  99%|█████████▉| 152/153 [00:02<00:00, 69.13batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:14,  2.88s/epoch, loss=0.00039, prev_loss=0.00054] 
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:11,  2.81s/epoch, loss=0.00039, prev_loss=0.00054]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/153 [00:00<00:03, 44.86batch/s]

Training batches on cuda:   8%|▊         | 12/153 [00:00<00:02, 49.23batch/s]

Training batches on cuda:  12%|█▏        | 19/153 [00:00<00:02, 53.13batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:02, 56.01batch/s]

Training batches on cuda:  22%|██▏       | 33/153 [00:00<00:02, 57.67batch/s]

Training batches on cuda:  26%|██▌       | 40/153 [00:00<00:01, 58.74batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:00<00:01, 56.34batch/s]

Training batches on cuda:  34%|███▍      | 52/153 [00:00<00:01, 54.89batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:01, 55.41batch/s]

Training batches on cuda:  42%|████▏     | 64/153 [00:01<00:01, 56.05batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:01, 56.94batch/s]

Training batches on cuda:  50%|█████     | 77/153 [00:01<00:01, 57.57batch/s]

Training batches on cuda:  54%|█████▍    | 83/153 [00:01<00:01, 55.88batch/s]

Training batches on cuda:  58%|█████▊    | 89/153 [00:01<00:01, 53.21batch/s]

Training batches on cuda:  63%|██████▎   | 96/153 [00:01<00:01, 55.56batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:01<00:00, 52.27batch/s]

Training batches on cuda:  71%|███████   | 108/153 [00:01<00:00, 52.96batch/s]

Training batches on cuda:  75%|███████▍  | 114/153 [00:02<00:00, 54.61batch/s]

Training batches on cuda:  78%|███████▊  | 120/153 [00:02<00:00, 55.25batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:02<00:00, 58.77batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:02<00:00, 61.09batch/s]

Training batches on cuda:  92%|█████████▏| 141/153 [00:02<00:00, 59.97batch/s]

Training batches on cuda:  97%|█████████▋| 148/153 [00:02<00:00, 59.36batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:20<00:11,  2.81s/epoch, loss=0.000348, prev_loss=0.00039]
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:08,  2.83s/epoch, loss=0.000348, prev_loss=0.00039]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 26.06batch/s]

Training batches on cuda:   5%|▌         | 8/153 [00:00<00:04, 29.87batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:04, 34.30batch/s]

Training batches on cuda:  13%|█▎        | 20/153 [00:00<00:03, 39.02batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:02, 42.88batch/s]

Training batches on cuda:  21%|██        | 32/153 [00:00<00:02, 46.46batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:00<00:02, 49.47batch/s]

Training batches on cuda:  29%|██▉       | 44/153 [00:00<00:02, 50.89batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:00<00:01, 52.89batch/s]

Training batches on cuda:  37%|███▋      | 56/153 [00:01<00:01, 54.50batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:01, 53.93batch/s]

Training batches on cuda:  44%|████▍     | 68/153 [00:01<00:01, 54.32batch/s]

Training batches on cuda:  49%|████▉     | 75/153 [00:01<00:01, 56.49batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 58.63batch/s]

Training batches on cuda:  58%|█████▊    | 88/153 [00:01<00:01, 58.04batch/s]

Training batches on cuda:  61%|██████▏   | 94/153 [00:01<00:01, 58.38batch/s]

Training batches on cuda:  65%|██████▌   | 100/153 [00:01<00:01, 52.60batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:01<00:00, 51.30batch/s]

Training batches on cuda:  73%|███████▎  | 112/153 [00:02<00:00, 51.38batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:02<00:00, 53.00batch/s]

Training batches on cuda:  82%|████████▏ | 125/153 [00:02<00:00, 55.20batch/s]

Training batches on cuda:  86%|████████▋ | 132/153 [00:02<00:00, 56.94batch/s]

Training batches on cuda:  91%|█████████ | 139/153 [00:02<00:00, 57.32batch/s]

Training batches on cuda:  95%|█████████▍| 145/153 [00:02<00:00, 55.93batch/s]

Training batches on cuda:  99%|█████████▊| 151/153 [00:02<00:00, 52.30batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:23<00:08,  2.83s/epoch, loss=0.0003, prev_loss=0.000348] 
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:05,  2.92s/epoch, loss=0.0003, prev_loss=0.000348]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/153 [00:00<00:08, 17.71batch/s]

Training batches on cuda:   5%|▌         | 8/153 [00:00<00:06, 22.23batch/s]

Training batches on cuda:   9%|▉         | 14/153 [00:00<00:05, 27.28batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:04, 28.40batch/s]

Training batches on cuda:  16%|█▌        | 24/153 [00:00<00:03, 33.13batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:03, 38.19batch/s]

Training batches on cuda:  24%|██▍       | 37/153 [00:00<00:02, 43.01batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:00<00:02, 46.87batch/s]

Training batches on cuda:  32%|███▏      | 49/153 [00:00<00:02, 50.15batch/s]

Training batches on cuda:  36%|███▌      | 55/153 [00:01<00:01, 52.21batch/s]

Training batches on cuda:  40%|███▉      | 61/153 [00:01<00:01, 53.69batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:01, 52.38batch/s]

Training batches on cuda:  48%|████▊     | 73/153 [00:01<00:01, 53.93batch/s]

Training batches on cuda:  52%|█████▏    | 79/153 [00:01<00:01, 54.76batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:01<00:01, 56.61batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:01<00:01, 56.83batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:01<00:00, 59.14batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:01<00:00, 61.18batch/s]

Training batches on cuda:  74%|███████▍  | 113/153 [00:02<00:00, 59.33batch/s]

Training batches on cuda:  78%|███████▊  | 119/153 [00:02<00:00, 58.73batch/s]

Training batches on cuda:  82%|████████▏ | 126/153 [00:02<00:00, 59.21batch/s]

Training batches on cuda:  87%|████████▋ | 133/153 [00:02<00:00, 59.51batch/s]

Training batches on cuda:  92%|█████████▏| 140/153 [00:02<00:00, 59.68batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:02<00:00, 59.26batch/s]

Training batches on cuda:  99%|█████████▉| 152/153 [00:02<00:00, 58.12batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:05,  2.92s/epoch, loss=0.000271, prev_loss=0.0003]
Training epochs on cuda:  90%|█████████ | 9/10 [00:26<00:02,  2.94s/epoch, loss=0.000271, prev_loss=0.0003]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 34.83batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:03, 37.46batch/s]

Training batches on cuda:  10%|█         | 16/153 [00:00<00:03, 42.38batch/s]

Training batches on cuda:  15%|█▌        | 23/153 [00:00<00:02, 46.60batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:02, 50.15batch/s]

Training batches on cuda:  24%|██▎       | 36/153 [00:00<00:02, 52.38batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:00<00:02, 54.91batch/s]

Training batches on cuda:  32%|███▏      | 49/153 [00:00<00:01, 55.59batch/s]

Training batches on cuda:  37%|███▋      | 56/153 [00:01<00:01, 57.36batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:01, 55.09batch/s]

Training batches on cuda:  44%|████▍     | 68/153 [00:01<00:01, 56.10batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 56.01batch/s]

Training batches on cuda:  52%|█████▏    | 80/153 [00:01<00:01, 55.76batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:01<00:01, 55.07batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:01<00:01, 55.64batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:01<00:00, 55.27batch/s]

Training batches on cuda:  68%|██████▊   | 104/153 [00:01<00:00, 53.76batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:02<00:00, 53.11batch/s]

Training batches on cuda:  76%|███████▋  | 117/153 [00:02<00:00, 55.33batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:02<00:00, 55.16batch/s]

Training batches on cuda:  84%|████████▍ | 129/153 [00:02<00:00, 56.30batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:02<00:00, 55.53batch/s]

Training batches on cuda:  92%|█████████▏| 141/153 [00:02<00:00, 56.58batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:02<00:00, 56.80batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:02<00:00, 55.35batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:29<00:02,  2.94s/epoch, loss=0.000261, prev_loss=0.000271]2021-09-16:18:52:05,829 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:52:06,954 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:52:21,980 INFO     [evaluator.py:657] Evaluation took 15.03s seconds
2021-09-16:18:52:22,121 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  7.86s/epoch, loss=0.000261, prev_loss=0.000271]
Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  4.56s/epoch, loss=0.000261, prev_loss=0.000271]
2021-09-16:18:52:22,122 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmptbohw040'
2021-09-16:18:52:22,203 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmptbohw040' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.53ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.62ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:52:37,94 INFO     [evaluator.py:657] Evaluation took 14.86s seconds
[I 2021-09-16 18:52:37,100] Trial 61 finished with value: 0.9563931697510463 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.010710320771885587, 'optimizer.lr': 0.04766384261867445, 'negative_sampler.num_negs_per_pos': 9, 'training.batch_size': 2032}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:52:37,123 WARNING  [api.py:823] No random seed is specified. Setting to 2803253958.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 34.63batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:03, 38.76batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:03, 40.87batch/s]

Training batches on cuda:  14%|█▎        | 21/153 [00:00<00:03, 43.63batch/s]

Training batches on cuda:  18%|█▊        | 27/153 [00:00<00:02, 47.31batch/s]

Training batches on cuda:  22%|██▏       | 33/153 [00:00<00:02, 48.61batch/s]

Training batches on cuda:  25%|██▌       | 39/153 [00:00<00:02, 49.64batch/s]

Training batches on cuda:  29%|██▉       | 44/153 [00:00<00:02, 49.72batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:00<00:02, 50.54batch/s]

Training batches on cuda:  37%|███▋      | 56/153 [00:01<00:01, 50.86batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:01, 52.72batch/s]

Training batches on cuda:  44%|████▍     | 68/153 [00:01<00:01, 50.32batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 48.50batch/s]

Training batches on cuda:  52%|█████▏    | 80/153 [00:01<00:01, 51.31batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:01<00:01, 53.20batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:01<00:01, 54.30batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:01<00:00, 55.97batch/s]

Training batches on cuda:  69%|██████▊   | 105/153 [00:02<00:00, 56.45batch/s]

Training batches on cuda:  73%|███████▎  | 111/153 [00:02<00:00, 56.75batch/s]

Training batches on cuda:  76%|███████▋  | 117/153 [00:02<00:00, 56.39batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:02<00:00, 56.20batch/s]

Training batches on cuda:  84%|████████▍ | 129/153 [00:02<00:00, 54.42batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:02<00:00, 52.55batch/s]

Training batches on cuda:  92%|█████████▏| 141/153 [00:02<00:00, 54.04batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:02<00:00, 55.09batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:02<00:00, 55.00batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000652, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:27,  3.10s/epoch, loss=0.000652, prev_loss=nan]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 36.87batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:03, 40.98batch/s]

Training batches on cuda:  10%|█         | 16/153 [00:00<00:03, 44.53batch/s]

Training batches on cuda:  14%|█▎        | 21/153 [00:00<00:02, 46.03batch/s]

Training batches on cuda:  18%|█▊        | 28/153 [00:00<00:02, 49.71batch/s]

Training batches on cuda:  23%|██▎       | 35/153 [00:00<00:02, 52.49batch/s]

Training batches on cuda:  27%|██▋       | 41/153 [00:00<00:02, 53.20batch/s]

Training batches on cuda:  31%|███       | 47/153 [00:00<00:01, 53.09batch/s]

Training batches on cuda:  35%|███▍      | 53/153 [00:00<00:01, 54.88batch/s]

Training batches on cuda:  39%|███▊      | 59/153 [00:01<00:01, 54.83batch/s]

Training batches on cuda:  42%|████▏     | 65/153 [00:01<00:01, 55.45batch/s]

Training batches on cuda:  46%|████▋     | 71/153 [00:01<00:01, 56.26batch/s]

Training batches on cuda:  50%|█████     | 77/153 [00:01<00:01, 55.50batch/s]

Training batches on cuda:  54%|█████▍    | 83/153 [00:01<00:01, 54.81batch/s]

Training batches on cuda:  58%|█████▊    | 89/153 [00:01<00:01, 55.89batch/s]

Training batches on cuda:  63%|██████▎   | 96/153 [00:01<00:00, 57.19batch/s]

Training batches on cuda:  67%|██████▋   | 103/153 [00:01<00:00, 58.25batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:01<00:00, 59.14batch/s]

Training batches on cuda:  76%|███████▋  | 117/153 [00:02<00:00, 59.53batch/s]

Training batches on cuda:  81%|████████  | 124/153 [00:02<00:00, 59.86batch/s]

Training batches on cuda:  86%|████████▌ | 131/153 [00:02<00:00, 60.14batch/s]

Training batches on cuda:  90%|█████████ | 138/153 [00:02<00:00, 60.27batch/s]

Training batches on cuda:  95%|█████████▍| 145/153 [00:02<00:00, 60.32batch/s]

Training batches on cuda:  99%|█████████▉| 152/153 [00:02<00:00, 60.36batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:27,  3.10s/epoch, loss=0.000649, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:24,  3.06s/epoch, loss=0.000649, prev_loss=nan]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:03, 38.75batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:03, 43.16batch/s]

Training batches on cuda:  11%|█         | 17/153 [00:00<00:02, 47.27batch/s]

Training batches on cuda:  16%|█▌        | 24/153 [00:00<00:02, 51.79batch/s]

Training batches on cuda:  20%|██        | 31/153 [00:00<00:02, 55.88batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:00<00:01, 59.41batch/s]

Training batches on cuda:  29%|██▉       | 45/153 [00:00<00:01, 61.44batch/s]

Training batches on cuda:  34%|███▍      | 52/153 [00:00<00:01, 63.04batch/s]

Training batches on cuda:  39%|███▊      | 59/153 [00:00<00:01, 64.34batch/s]

Training batches on cuda:  43%|████▎     | 66/153 [00:01<00:01, 65.85batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 67.06batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 67.76batch/s]

Training batches on cuda:  58%|█████▊    | 89/153 [00:01<00:00, 68.24batch/s]

Training batches on cuda:  63%|██████▎   | 97/153 [00:01<00:00, 68.84batch/s]

Training batches on cuda:  68%|██████▊   | 104/153 [00:01<00:00, 69.14batch/s]

Training batches on cuda:  73%|███████▎  | 111/153 [00:01<00:00, 69.30batch/s]

Training batches on cuda:  77%|███████▋  | 118/153 [00:01<00:00, 61.75batch/s]

Training batches on cuda:  82%|████████▏ | 125/153 [00:01<00:00, 63.85batch/s]

Training batches on cuda:  86%|████████▋ | 132/153 [00:02<00:00, 65.02batch/s]

Training batches on cuda:  91%|█████████ | 139/153 [00:02<00:00, 66.26batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:02<00:00, 67.04batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:02<00:00, 67.55batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:24,  3.06s/epoch, loss=0.000649, prev_loss=0.000649]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:20,  2.90s/epoch, loss=0.000649, prev_loss=0.000649]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 36.40batch/s]

Training batches on cuda:   7%|▋         | 11/153 [00:00<00:03, 42.07batch/s]

Training batches on cuda:  12%|█▏        | 18/153 [00:00<00:02, 46.29batch/s]

Training batches on cuda:  16%|█▋        | 25/153 [00:00<00:02, 50.38batch/s]

Training batches on cuda:  21%|██        | 32/153 [00:00<00:02, 53.55batch/s]

Training batches on cuda:  25%|██▌       | 39/153 [00:00<00:02, 56.86batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:00<00:01, 59.14batch/s]

Training batches on cuda:  35%|███▍      | 53/153 [00:00<00:01, 59.64batch/s]

Training batches on cuda:  39%|███▊      | 59/153 [00:01<00:01, 56.49batch/s]

Training batches on cuda:  42%|████▏     | 65/153 [00:01<00:01, 56.97batch/s]

Training batches on cuda:  47%|████▋     | 72/153 [00:01<00:01, 58.84batch/s]

Training batches on cuda:  52%|█████▏    | 79/153 [00:01<00:01, 60.61batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:01<00:01, 59.30batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:01<00:01, 58.06batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:01<00:00, 56.74batch/s]

Training batches on cuda:  68%|██████▊   | 104/153 [00:01<00:00, 55.50batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:01<00:00, 54.63batch/s]

Training batches on cuda:  76%|███████▌  | 116/153 [00:02<00:00, 52.76batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:02<00:00, 52.34batch/s]

Training batches on cuda:  84%|████████▎ | 128/153 [00:02<00:00, 51.57batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:02<00:00, 48.53batch/s]

Training batches on cuda:  92%|█████████▏| 140/153 [00:02<00:00, 50.90batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:02<00:00, 53.83batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:02<00:00, 54.21batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:20,  2.90s/epoch, loss=0.000649, prev_loss=0.000649]
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:17,  2.91s/epoch, loss=0.000649, prev_loss=0.000649]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 35.27batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:03, 39.90batch/s]

Training batches on cuda:  11%|█         | 17/153 [00:00<00:03, 44.37batch/s]

Training batches on cuda:  16%|█▌        | 24/153 [00:00<00:02, 48.13batch/s]

Training batches on cuda:  20%|█▉        | 30/153 [00:00<00:02, 51.06batch/s]

Training batches on cuda:  24%|██▍       | 37/153 [00:00<00:02, 53.63batch/s]

Training batches on cuda:  29%|██▉       | 44/153 [00:00<00:01, 55.38batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:00<00:01, 55.92batch/s]

Training batches on cuda:  37%|███▋      | 56/153 [00:00<00:01, 56.95batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:01, 54.41batch/s]

Training batches on cuda:  44%|████▍     | 68/153 [00:01<00:01, 53.93batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 55.56batch/s]

Training batches on cuda:  52%|█████▏    | 80/153 [00:01<00:01, 56.81batch/s]

Training batches on cuda:  57%|█████▋    | 87/153 [00:01<00:01, 57.96batch/s]

Training batches on cuda:  61%|██████    | 93/153 [00:01<00:01, 58.34batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:01<00:00, 58.77batch/s]

Training batches on cuda:  69%|██████▉   | 106/153 [00:01<00:00, 59.21batch/s]

Training batches on cuda:  74%|███████▍  | 113/153 [00:01<00:00, 59.50batch/s]

Training batches on cuda:  78%|███████▊  | 120/153 [00:02<00:00, 59.72batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:02<00:00, 59.89batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:02<00:00, 59.98batch/s]

Training batches on cuda:  92%|█████████▏| 141/153 [00:02<00:00, 59.73batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:02<00:00, 57.43batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:02<00:00, 56.26batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:17,  2.91s/epoch, loss=0.000649, prev_loss=0.000649]
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:14,  2.90s/epoch, loss=0.000649, prev_loss=0.000649]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 36.35batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:03, 40.59batch/s]

Training batches on cuda:  10%|█         | 16/153 [00:00<00:03, 44.46batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:02, 47.88batch/s]

Training batches on cuda:  18%|█▊        | 28/153 [00:00<00:02, 50.22batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:00<00:02, 51.31batch/s]

Training batches on cuda:  27%|██▋       | 41/153 [00:00<00:02, 53.56batch/s]

Training batches on cuda:  31%|███▏      | 48/153 [00:00<00:01, 55.93batch/s]

Training batches on cuda:  36%|███▌      | 55/153 [00:00<00:01, 57.44batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:01, 59.53batch/s]

Training batches on cuda:  45%|████▌     | 69/153 [00:01<00:01, 60.95batch/s]

Training batches on cuda:  50%|████▉     | 76/153 [00:01<00:01, 61.74batch/s]

Training batches on cuda:  54%|█████▍    | 83/153 [00:01<00:01, 61.22batch/s]

Training batches on cuda:  59%|█████▉    | 90/153 [00:01<00:01, 61.22batch/s]

Training batches on cuda:  63%|██████▎   | 97/153 [00:01<00:00, 57.35batch/s]

Training batches on cuda:  67%|██████▋   | 103/153 [00:01<00:00, 53.38batch/s]

Training batches on cuda:  71%|███████   | 109/153 [00:01<00:00, 54.91batch/s]

Training batches on cuda:  75%|███████▌  | 115/153 [00:02<00:00, 55.42batch/s]

Training batches on cuda:  79%|███████▉  | 121/153 [00:02<00:00, 56.48batch/s]

Training batches on cuda:  84%|████████▎ | 128/153 [00:02<00:00, 58.63batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:02<00:00, 59.90batch/s]

Training batches on cuda:  93%|█████████▎| 142/153 [00:02<00:00, 60.04batch/s]

Training batches on cuda:  97%|█████████▋| 149/153 [00:02<00:00, 58.96batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:14,  2.90s/epoch, loss=0.000649, prev_loss=0.000649]
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:11,  2.91s/epoch, loss=0.000649, prev_loss=0.000649]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 34.67batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:03, 37.19batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:03, 41.80batch/s]

Training batches on cuda:  14%|█▎        | 21/153 [00:00<00:02, 45.72batch/s]

Training batches on cuda:  17%|█▋        | 26/153 [00:00<00:02, 46.89batch/s]

Training batches on cuda:  21%|██        | 32/153 [00:00<00:02, 47.70batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:00<00:02, 50.53batch/s]

Training batches on cuda:  28%|██▊       | 43/153 [00:00<00:02, 47.26batch/s]

Training batches on cuda:  32%|███▏      | 49/153 [00:00<00:02, 49.80batch/s]

Training batches on cuda:  36%|███▌      | 55/153 [00:01<00:01, 50.94batch/s]

Training batches on cuda:  40%|███▉      | 61/153 [00:01<00:01, 52.84batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:01, 52.75batch/s]

Training batches on cuda:  48%|████▊     | 73/153 [00:01<00:01, 54.01batch/s]

Training batches on cuda:  52%|█████▏    | 79/153 [00:01<00:01, 55.28batch/s]

Training batches on cuda:  56%|█████▌    | 85/153 [00:01<00:01, 54.24batch/s]

Training batches on cuda:  59%|█████▉    | 91/153 [00:01<00:01, 53.69batch/s]

Training batches on cuda:  63%|██████▎   | 97/153 [00:01<00:01, 54.38batch/s]

Training batches on cuda:  67%|██████▋   | 103/153 [00:01<00:00, 53.31batch/s]

Training batches on cuda:  71%|███████   | 109/153 [00:02<00:00, 52.50batch/s]

Training batches on cuda:  75%|███████▌  | 115/153 [00:02<00:00, 52.08batch/s]

Training batches on cuda:  79%|███████▉  | 121/153 [00:02<00:00, 51.59batch/s]

Training batches on cuda:  83%|████████▎ | 127/153 [00:02<00:00, 51.90batch/s]

Training batches on cuda:  87%|████████▋ | 133/153 [00:02<00:00, 52.27batch/s]

Training batches on cuda:  91%|█████████ | 139/153 [00:02<00:00, 52.13batch/s]

Training batches on cuda:  95%|█████████▍| 145/153 [00:02<00:00, 52.20batch/s]

Training batches on cuda:  99%|█████████▊| 151/153 [00:02<00:00, 53.49batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:20<00:11,  2.91s/epoch, loss=0.000649, prev_loss=0.000649]
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:08,  2.98s/epoch, loss=0.000649, prev_loss=0.000649]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/153 [00:00<00:04, 35.00batch/s]

Training batches on cuda:   7%|▋         | 10/153 [00:00<00:03, 38.84batch/s]

Training batches on cuda:  10%|█         | 16/153 [00:00<00:03, 42.27batch/s]

Training batches on cuda:  14%|█▍        | 22/153 [00:00<00:02, 45.31batch/s]

Training batches on cuda:  18%|█▊        | 28/153 [00:00<00:02, 47.61batch/s]

Training batches on cuda:  22%|██▏       | 34/153 [00:00<00:02, 50.26batch/s]

Training batches on cuda:  26%|██▌       | 40/153 [00:00<00:02, 51.53batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:00<00:02, 52.51batch/s]

Training batches on cuda:  34%|███▍      | 52/153 [00:00<00:01, 54.28batch/s]

Training batches on cuda:  38%|███▊      | 58/153 [00:01<00:01, 54.63batch/s]

Training batches on cuda:  42%|████▏     | 64/153 [00:01<00:01, 50.30batch/s]

Training batches on cuda:  46%|████▌     | 70/153 [00:01<00:01, 52.59batch/s]

Training batches on cuda:  50%|████▉     | 76/153 [00:01<00:01, 53.18batch/s]

Training batches on cuda:  54%|█████▎    | 82/153 [00:01<00:01, 49.45batch/s]

Training batches on cuda:  58%|█████▊    | 88/153 [00:01<00:01, 49.36batch/s]

Training batches on cuda:  61%|██████    | 93/153 [00:01<00:01, 47.11batch/s]

Training batches on cuda:  65%|██████▍   | 99/153 [00:01<00:01, 48.45batch/s]

Training batches on cuda:  69%|██████▊   | 105/153 [00:02<00:00, 51.24batch/s]

Training batches on cuda:  73%|███████▎  | 111/153 [00:02<00:00, 52.34batch/s]

Training batches on cuda:  76%|███████▋  | 117/153 [00:02<00:00, 53.29batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:02<00:00, 54.76batch/s]

Training batches on cuda:  84%|████████▍ | 129/153 [00:02<00:00, 55.17batch/s]

Training batches on cuda:  88%|████████▊ | 135/153 [00:02<00:00, 54.96batch/s]

Training batches on cuda:  92%|█████████▏| 141/153 [00:02<00:00, 56.10batch/s]

Training batches on cuda:  96%|█████████▌| 147/153 [00:02<00:00, 55.95batch/s]

Training batches on cuda: 100%|██████████| 153/153 [00:02<00:00, 54.02batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:23<00:08,  2.98s/epoch, loss=0.000649, prev_loss=0.000649]
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:06,  3.02s/epoch, loss=0.000649, prev_loss=0.000649]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 27.96batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:04, 32.92batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:03, 37.88batch/s]

Training batches on cuda:  14%|█▎        | 21/153 [00:00<00:03, 42.22batch/s]

Training batches on cuda:  18%|█▊        | 27/153 [00:00<00:02, 45.79batch/s]

Training batches on cuda:  22%|██▏       | 33/153 [00:00<00:02, 48.66batch/s]

Training batches on cuda:  25%|██▍       | 38/153 [00:00<00:02, 48.74batch/s]

Training batches on cuda:  29%|██▉       | 44/153 [00:00<00:02, 50.55batch/s]

Training batches on cuda:  33%|███▎      | 50/153 [00:00<00:01, 52.75batch/s]

Training batches on cuda:  37%|███▋      | 56/153 [00:01<00:01, 54.42batch/s]

Training batches on cuda:  41%|████      | 62/153 [00:01<00:01, 55.31batch/s]

Training batches on cuda:  44%|████▍     | 68/153 [00:01<00:01, 53.61batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 51.93batch/s]

Training batches on cuda:  52%|█████▏    | 80/153 [00:01<00:01, 50.65batch/s]

Training batches on cuda:  56%|█████▌    | 86/153 [00:01<00:01, 51.86batch/s]

Training batches on cuda:  60%|██████    | 92/153 [00:01<00:01, 51.93batch/s]

Training batches on cuda:  64%|██████▍   | 98/153 [00:01<00:01, 53.02batch/s]

Training batches on cuda:  68%|██████▊   | 104/153 [00:01<00:00, 54.90batch/s]

Training batches on cuda:  72%|███████▏  | 110/153 [00:02<00:00, 54.91batch/s]

Training batches on cuda:  76%|███████▌  | 116/153 [00:02<00:00, 53.88batch/s]

Training batches on cuda:  80%|███████▉  | 122/153 [00:02<00:00, 53.76batch/s]

Training batches on cuda:  84%|████████▎ | 128/153 [00:02<00:00, 54.13batch/s]

Training batches on cuda:  88%|████████▊ | 134/153 [00:02<00:00, 54.56batch/s]

Training batches on cuda:  92%|█████████▏| 140/153 [00:02<00:00, 55.33batch/s]

Training batches on cuda:  95%|█████████▌| 146/153 [00:02<00:00, 55.90batch/s]

Training batches on cuda:  99%|█████████▉| 152/153 [00:02<00:00, 55.24batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:06,  3.02s/epoch, loss=0.000649, prev_loss=0.000649]
Training epochs on cuda:  90%|█████████ | 9/10 [00:26<00:03,  3.04s/epoch, loss=0.000649, prev_loss=0.000649]

Training batches on cuda:   0%|          | 0/153 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/153 [00:00<00:05, 27.72batch/s]

Training batches on cuda:   6%|▌         | 9/153 [00:00<00:04, 32.67batch/s]

Training batches on cuda:  10%|▉         | 15/153 [00:00<00:03, 37.55batch/s]

Training batches on cuda:  14%|█▎        | 21/153 [00:00<00:03, 41.51batch/s]

Training batches on cuda:  18%|█▊        | 27/153 [00:00<00:02, 45.51batch/s]

Training batches on cuda:  22%|██▏       | 33/153 [00:00<00:02, 48.21batch/s]

Training batches on cuda:  25%|██▌       | 39/153 [00:00<00:02, 49.70batch/s]

Training batches on cuda:  30%|███       | 46/153 [00:00<00:02, 52.71batch/s]

Training batches on cuda:  35%|███▍      | 53/153 [00:00<00:01, 55.12batch/s]

Training batches on cuda:  39%|███▉      | 60/153 [00:01<00:01, 58.20batch/s]

Training batches on cuda:  44%|████▍     | 67/153 [00:01<00:01, 60.48batch/s]

Training batches on cuda:  48%|████▊     | 74/153 [00:01<00:01, 61.62batch/s]

Training batches on cuda:  53%|█████▎    | 81/153 [00:01<00:01, 63.45batch/s]

Training batches on cuda:  58%|█████▊    | 88/153 [00:01<00:01, 64.67batch/s]

Training batches on cuda:  62%|██████▏   | 95/153 [00:01<00:00, 65.03batch/s]

Training batches on cuda:  67%|██████▋   | 102/153 [00:01<00:00, 65.18batch/s]

Training batches on cuda:  71%|███████   | 109/153 [00:01<00:00, 63.65batch/s]

Training batches on cuda:  76%|███████▌  | 116/153 [00:01<00:00, 61.71batch/s]

Training batches on cuda:  80%|████████  | 123/153 [00:02<00:00, 61.78batch/s]

Training batches on cuda:  85%|████████▍ | 130/153 [00:02<00:00, 62.26batch/s]

Training batches on cuda:  90%|████████▉ | 137/153 [00:02<00:00, 62.78batch/s]

Training batches on cuda:  94%|█████████▍| 144/153 [00:02<00:00, 62.41batch/s]

Training batches on cuda:  99%|█████████▊| 151/153 [00:02<00:00, 59.03batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:29<00:03,  3.04s/epoch, loss=0.000649, prev_loss=0.000649]2021-09-16:18:53:07,241 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:53:08,388 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:53:23,445 INFO     [evaluator.py:657] Evaluation took 15.06s seconds
2021-09-16:18:53:23,587 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  7.89s/epoch, loss=0.000649, prev_loss=0.000649]
Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  4.59s/epoch, loss=0.000649, prev_loss=0.000649]
2021-09-16:18:53:23,588 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpq6v99qyr'
2021-09-16:18:53:23,673 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpq6v99qyr' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.35ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.60ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.62ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.60ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.61ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.62ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:53:38,586 INFO     [evaluator.py:657] Evaluation took 14.87s seconds
[I 2021-09-16 18:53:38,592] Trial 62 finished with value: 0.0006822307133957928 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.09106441331410131, 'optimizer.lr': 0.04798541076509851, 'negative_sampler.num_negs_per_pos': 10, 'training.batch_size': 2032}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:53:38,614 WARNING  [api.py:823] No random seed is specified. Setting to 2988583922.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/191 [00:00<00:10, 18.81batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:08, 22.32batch/s]

Training batches on cuda:   6%|▌         | 11/191 [00:00<00:06, 25.88batch/s]

Training batches on cuda:   8%|▊         | 16/191 [00:00<00:05, 29.30batch/s]

Training batches on cuda:  11%|█         | 21/191 [00:00<00:05, 31.97batch/s]

Training batches on cuda:  13%|█▎        | 25/191 [00:00<00:04, 33.79batch/s]

Training batches on cuda:  16%|█▌        | 30/191 [00:00<00:04, 35.58batch/s]

Training batches on cuda:  18%|█▊        | 34/191 [00:00<00:04, 36.15batch/s]

Training batches on cuda:  20%|██        | 39/191 [00:01<00:04, 37.15batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:01<00:03, 37.93batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:03, 38.23batch/s]

Training batches on cuda:  27%|██▋       | 51/191 [00:01<00:03, 38.63batch/s]

Training batches on cuda:  29%|██▉       | 55/191 [00:01<00:03, 38.78batch/s]

Training batches on cuda:  31%|███       | 59/191 [00:01<00:03, 38.84batch/s]

Training batches on cuda:  34%|███▎      | 64/191 [00:01<00:03, 39.31batch/s]

Training batches on cuda:  36%|███▌      | 69/191 [00:01<00:03, 40.39batch/s]

Training batches on cuda:  39%|███▊      | 74/191 [00:01<00:02, 39.77batch/s]

Training batches on cuda:  41%|████      | 78/191 [00:02<00:02, 39.15batch/s]

Training batches on cuda:  43%|████▎     | 83/191 [00:02<00:02, 39.32batch/s]

Training batches on cuda:  46%|████▌     | 87/191 [00:02<00:02, 39.45batch/s]

Training batches on cuda:  48%|████▊     | 92/191 [00:02<00:02, 40.27batch/s]

Training batches on cuda:  51%|█████     | 97/191 [00:02<00:02, 39.51batch/s]

Training batches on cuda:  53%|█████▎    | 101/191 [00:02<00:02, 37.41batch/s]

Training batches on cuda:  55%|█████▍    | 105/191 [00:02<00:02, 36.88batch/s]

Training batches on cuda:  57%|█████▋    | 109/191 [00:02<00:02, 36.60batch/s]

Training batches on cuda:  60%|█████▉    | 114/191 [00:02<00:02, 37.95batch/s]

Training batches on cuda:  62%|██████▏   | 118/191 [00:03<00:01, 38.26batch/s]

Training batches on cuda:  64%|██████▍   | 122/191 [00:03<00:01, 37.02batch/s]

Training batches on cuda:  66%|██████▌   | 126/191 [00:03<00:01, 36.98batch/s]

Training batches on cuda:  69%|██████▊   | 131/191 [00:03<00:01, 38.50batch/s]

Training batches on cuda:  71%|███████   | 135/191 [00:03<00:01, 38.15batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 38.75batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 39.69batch/s]

Training batches on cuda:  78%|███████▊  | 149/191 [00:03<00:01, 39.33batch/s]

Training batches on cuda:  81%|████████  | 154/191 [00:03<00:00, 39.68batch/s]

Training batches on cuda:  83%|████████▎ | 159/191 [00:04<00:00, 40.58batch/s]

Training batches on cuda:  86%|████████▌ | 164/191 [00:04<00:00, 41.34batch/s]

Training batches on cuda:  88%|████████▊ | 169/191 [00:04<00:00, 41.15batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:04<00:00, 41.10batch/s]

Training batches on cuda:  94%|█████████▎| 179/191 [00:04<00:00, 41.02batch/s]

Training batches on cuda:  96%|█████████▋| 184/191 [00:04<00:00, 40.85batch/s]

Training batches on cuda:  99%|█████████▉| 189/191 [00:04<00:00, 40.48batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.000811, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:47,  5.23s/epoch, loss=0.000811, prev_loss=nan]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:07, 25.06batch/s]

Training batches on cuda:   4%|▍         | 8/191 [00:00<00:06, 28.31batch/s]

Training batches on cuda:   6%|▋         | 12/191 [00:00<00:05, 30.50batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:05, 33.02batch/s]

Training batches on cuda:  11%|█         | 21/191 [00:00<00:04, 34.70batch/s]

Training batches on cuda:  14%|█▎        | 26/191 [00:00<00:04, 36.51batch/s]

Training batches on cuda:  16%|█▌        | 30/191 [00:00<00:04, 36.82batch/s]

Training batches on cuda:  18%|█▊        | 34/191 [00:00<00:04, 37.14batch/s]

Training batches on cuda:  20%|█▉        | 38/191 [00:01<00:04, 36.37batch/s]

Training batches on cuda:  22%|██▏       | 42/191 [00:01<00:04, 36.31batch/s]

Training batches on cuda:  24%|██▍       | 46/191 [00:01<00:03, 36.38batch/s]

Training batches on cuda:  26%|██▌       | 50/191 [00:01<00:03, 35.68batch/s]

Training batches on cuda:  28%|██▊       | 54/191 [00:01<00:03, 36.22batch/s]

Training batches on cuda:  30%|███       | 58/191 [00:01<00:03, 34.47batch/s]

Training batches on cuda:  33%|███▎      | 63/191 [00:01<00:03, 36.27batch/s]

Training batches on cuda:  36%|███▌      | 68/191 [00:01<00:03, 37.36batch/s]

Training batches on cuda:  38%|███▊      | 73/191 [00:01<00:03, 38.32batch/s]

Training batches on cuda:  41%|████      | 78/191 [00:02<00:02, 38.60batch/s]

Training batches on cuda:  43%|████▎     | 82/191 [00:02<00:02, 38.70batch/s]

Training batches on cuda:  45%|████▌     | 86/191 [00:02<00:02, 37.58batch/s]

Training batches on cuda:  47%|████▋     | 90/191 [00:02<00:02, 37.04batch/s]

Training batches on cuda:  49%|████▉     | 94/191 [00:02<00:02, 36.72batch/s]

Training batches on cuda:  51%|█████▏    | 98/191 [00:02<00:02, 35.86batch/s]

Training batches on cuda:  53%|█████▎    | 102/191 [00:02<00:02, 35.30batch/s]

Training batches on cuda:  56%|█████▌    | 107/191 [00:02<00:02, 37.48batch/s]

Training batches on cuda:  59%|█████▊    | 112/191 [00:03<00:02, 38.43batch/s]

Training batches on cuda:  61%|██████▏   | 117/191 [00:03<00:01, 39.01batch/s]

Training batches on cuda:  63%|██████▎   | 121/191 [00:03<00:01, 38.80batch/s]

Training batches on cuda:  66%|██████▌   | 126/191 [00:03<00:01, 39.22batch/s]

Training batches on cuda:  69%|██████▊   | 131/191 [00:03<00:01, 39.47batch/s]

Training batches on cuda:  71%|███████   | 136/191 [00:03<00:01, 39.83batch/s]

Training batches on cuda:  74%|███████▍  | 141/191 [00:03<00:01, 40.22batch/s]

Training batches on cuda:  76%|███████▋  | 146/191 [00:03<00:01, 39.70batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:01, 39.78batch/s]

Training batches on cuda:  81%|████████  | 154/191 [00:04<00:00, 39.10batch/s]

Training batches on cuda:  83%|████████▎ | 159/191 [00:04<00:00, 39.12batch/s]

Training batches on cuda:  85%|████████▌ | 163/191 [00:04<00:00, 39.34batch/s]

Training batches on cuda:  88%|████████▊ | 168/191 [00:04<00:00, 39.42batch/s]

Training batches on cuda:  91%|█████████ | 173/191 [00:04<00:00, 39.39batch/s]

Training batches on cuda:  93%|█████████▎| 177/191 [00:04<00:00, 38.85batch/s]

Training batches on cuda:  95%|█████████▍| 181/191 [00:04<00:00, 38.46batch/s]

Training batches on cuda:  97%|█████████▋| 185/191 [00:04<00:00, 37.96batch/s]

Training batches on cuda:  99%|█████████▉| 189/191 [00:04<00:00, 38.29batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:10<00:47,  5.23s/epoch, loss=0.000809, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:10<00:41,  5.24s/epoch, loss=0.000809, prev_loss=nan]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/191 [00:00<00:23,  8.13batch/s]

Training batches on cuda:   3%|▎         | 5/191 [00:00<00:17, 10.68batch/s]

Training batches on cuda:   5%|▌         | 10/191 [00:00<00:13, 13.72batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:10, 17.17batch/s]

Training batches on cuda:  10%|█         | 20/191 [00:00<00:08, 20.89batch/s]

Training batches on cuda:  13%|█▎        | 25/191 [00:00<00:06, 24.56batch/s]

Training batches on cuda:  16%|█▌        | 30/191 [00:00<00:05, 27.86batch/s]

Training batches on cuda:  18%|█▊        | 35/191 [00:00<00:05, 31.07batch/s]

Training batches on cuda:  20%|██        | 39/191 [00:01<00:04, 33.23batch/s]

Training batches on cuda:  23%|██▎       | 44/191 [00:01<00:04, 35.09batch/s]

Training batches on cuda:  25%|██▌       | 48/191 [00:01<00:03, 36.22batch/s]

Training batches on cuda:  27%|██▋       | 52/191 [00:01<00:03, 37.13batch/s]

Training batches on cuda:  30%|██▉       | 57/191 [00:01<00:03, 38.18batch/s]

Training batches on cuda:  32%|███▏      | 62/191 [00:01<00:03, 39.13batch/s]

Training batches on cuda:  35%|███▌      | 67/191 [00:01<00:03, 39.21batch/s]

Training batches on cuda:  38%|███▊      | 72/191 [00:01<00:02, 40.35batch/s]

Training batches on cuda:  40%|████      | 77/191 [00:01<00:02, 40.44batch/s]

Training batches on cuda:  43%|████▎     | 82/191 [00:02<00:02, 40.34batch/s]

Training batches on cuda:  46%|████▌     | 87/191 [00:02<00:02, 39.21batch/s]

Training batches on cuda:  48%|████▊     | 92/191 [00:02<00:02, 39.87batch/s]

Training batches on cuda:  51%|█████     | 97/191 [00:02<00:02, 40.76batch/s]

Training batches on cuda:  53%|█████▎    | 102/191 [00:02<00:02, 40.38batch/s]

Training batches on cuda:  56%|█████▌    | 107/191 [00:02<00:02, 40.52batch/s]

Training batches on cuda:  59%|█████▊    | 112/191 [00:02<00:02, 39.03batch/s]

Training batches on cuda:  61%|██████▏   | 117/191 [00:02<00:01, 39.70batch/s]

Training batches on cuda:  63%|██████▎   | 121/191 [00:03<00:01, 38.39batch/s]

Training batches on cuda:  65%|██████▌   | 125/191 [00:03<00:01, 38.70batch/s]

Training batches on cuda:  68%|██████▊   | 129/191 [00:03<00:01, 38.20batch/s]

Training batches on cuda:  70%|██████▉   | 133/191 [00:03<00:01, 38.18batch/s]

Training batches on cuda:  72%|███████▏  | 137/191 [00:03<00:01, 37.83batch/s]

Training batches on cuda:  74%|███████▍  | 141/191 [00:03<00:01, 36.92batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 37.69batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:01, 38.67batch/s]

Training batches on cuda:  81%|████████  | 154/191 [00:03<00:00, 38.79batch/s]

Training batches on cuda:  83%|████████▎ | 159/191 [00:04<00:00, 39.05batch/s]

Training batches on cuda:  86%|████████▌ | 164/191 [00:04<00:00, 39.43batch/s]

Training batches on cuda:  88%|████████▊ | 169/191 [00:04<00:00, 40.02batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:04<00:00, 40.00batch/s]

Training batches on cuda:  94%|█████████▎| 179/191 [00:04<00:00, 40.43batch/s]

Training batches on cuda:  96%|█████████▋| 184/191 [00:04<00:00, 40.78batch/s]

Training batches on cuda:  99%|█████████▉| 189/191 [00:04<00:00, 41.28batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:15<00:41,  5.24s/epoch, loss=0.00081, prev_loss=0.000809]
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:36,  5.21s/epoch, loss=0.00081, prev_loss=0.000809]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:06, 27.16batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:06, 30.03batch/s]

Training batches on cuda:   6%|▋         | 12/191 [00:00<00:05, 32.77batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:05, 34.75batch/s]

Training batches on cuda:  12%|█▏        | 22/191 [00:00<00:04, 36.28batch/s]

Training batches on cuda:  14%|█▍        | 27/191 [00:00<00:04, 37.66batch/s]

Training batches on cuda:  16%|█▌        | 31/191 [00:00<00:04, 37.47batch/s]

Training batches on cuda:  19%|█▉        | 36/191 [00:00<00:04, 38.69batch/s]

Training batches on cuda:  21%|██▏       | 41/191 [00:01<00:03, 39.53batch/s]

Training batches on cuda:  24%|██▍       | 46/191 [00:01<00:03, 40.23batch/s]

Training batches on cuda:  27%|██▋       | 51/191 [00:01<00:03, 40.42batch/s]

Training batches on cuda:  29%|██▉       | 56/191 [00:01<00:03, 40.36batch/s]

Training batches on cuda:  32%|███▏      | 61/191 [00:01<00:03, 40.13batch/s]

Training batches on cuda:  34%|███▍      | 65/191 [00:01<00:03, 40.03batch/s]

Training batches on cuda:  36%|███▌      | 69/191 [00:01<00:03, 38.21batch/s]

Training batches on cuda:  38%|███▊      | 73/191 [00:01<00:03, 37.74batch/s]

Training batches on cuda:  41%|████      | 78/191 [00:01<00:02, 39.17batch/s]

Training batches on cuda:  43%|████▎     | 83/191 [00:02<00:02, 40.41batch/s]

Training batches on cuda:  46%|████▌     | 88/191 [00:02<00:02, 41.35batch/s]

Training batches on cuda:  49%|████▊     | 93/191 [00:02<00:02, 41.98batch/s]

Training batches on cuda:  51%|█████▏    | 98/191 [00:02<00:02, 42.38batch/s]

Training batches on cuda:  54%|█████▍    | 103/191 [00:02<00:02, 42.87batch/s]

Training batches on cuda:  57%|█████▋    | 108/191 [00:02<00:01, 43.05batch/s]

Training batches on cuda:  59%|█████▉    | 113/191 [00:02<00:01, 43.04batch/s]

Training batches on cuda:  62%|██████▏   | 118/191 [00:02<00:01, 43.02batch/s]

Training batches on cuda:  64%|██████▍   | 123/191 [00:03<00:01, 43.10batch/s]

Training batches on cuda:  67%|██████▋   | 128/191 [00:03<00:01, 43.13batch/s]

Training batches on cuda:  70%|██████▉   | 133/191 [00:03<00:01, 43.44batch/s]

Training batches on cuda:  72%|███████▏  | 138/191 [00:03<00:01, 43.50batch/s]

Training batches on cuda:  75%|███████▍  | 143/191 [00:03<00:01, 43.49batch/s]

Training batches on cuda:  77%|███████▋  | 148/191 [00:03<00:00, 43.35batch/s]

Training batches on cuda:  80%|████████  | 153/191 [00:03<00:00, 43.23batch/s]

Training batches on cuda:  83%|████████▎ | 158/191 [00:03<00:00, 43.10batch/s]

Training batches on cuda:  85%|████████▌ | 163/191 [00:03<00:00, 43.22batch/s]

Training batches on cuda:  88%|████████▊ | 168/191 [00:04<00:00, 43.50batch/s]

Training batches on cuda:  91%|█████████ | 173/191 [00:04<00:00, 43.53batch/s]

Training batches on cuda:  93%|█████████▎| 178/191 [00:04<00:00, 43.62batch/s]

Training batches on cuda:  96%|█████████▌| 183/191 [00:04<00:00, 43.21batch/s]

Training batches on cuda:  98%|█████████▊| 188/191 [00:04<00:00, 43.12batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:20<00:36,  5.21s/epoch, loss=0.00081, prev_loss=0.00081] 
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:30,  5.08s/epoch, loss=0.00081, prev_loss=0.00081]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/191 [00:00<00:11, 16.72batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:09, 20.07batch/s]

Training batches on cuda:   5%|▌         | 10/191 [00:00<00:07, 23.57batch/s]

Training batches on cuda:   7%|▋         | 14/191 [00:00<00:06, 26.78batch/s]

Training batches on cuda:  10%|▉         | 19/191 [00:00<00:05, 29.70batch/s]

Training batches on cuda:  13%|█▎        | 24/191 [00:00<00:05, 32.52batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:04, 34.22batch/s]

Training batches on cuda:  17%|█▋        | 32/191 [00:00<00:04, 35.74batch/s]

Training batches on cuda:  19%|█▉        | 37/191 [00:01<00:04, 36.99batch/s]

Training batches on cuda:  21%|██▏       | 41/191 [00:01<00:04, 36.94batch/s]

Training batches on cuda:  24%|██▍       | 46/191 [00:01<00:03, 37.97batch/s]

Training batches on cuda:  27%|██▋       | 51/191 [00:01<00:03, 38.77batch/s]

Training batches on cuda:  29%|██▉       | 55/191 [00:01<00:03, 38.45batch/s]

Training batches on cuda:  31%|███       | 59/191 [00:01<00:03, 38.69batch/s]

Training batches on cuda:  33%|███▎      | 63/191 [00:01<00:03, 38.84batch/s]

Training batches on cuda:  35%|███▌      | 67/191 [00:01<00:03, 38.56batch/s]

Training batches on cuda:  37%|███▋      | 71/191 [00:01<00:03, 38.33batch/s]

Training batches on cuda:  39%|███▉      | 75/191 [00:01<00:03, 38.44batch/s]

Training batches on cuda:  41%|████▏     | 79/191 [00:02<00:02, 38.82batch/s]

Training batches on cuda:  43%|████▎     | 83/191 [00:02<00:02, 39.04batch/s]

Training batches on cuda:  46%|████▌     | 87/191 [00:02<00:02, 39.11batch/s]

Training batches on cuda:  48%|████▊     | 91/191 [00:02<00:02, 39.02batch/s]

Training batches on cuda:  50%|█████     | 96/191 [00:02<00:02, 39.82batch/s]

Training batches on cuda:  53%|█████▎    | 101/191 [00:02<00:02, 40.02batch/s]

Training batches on cuda:  55%|█████▌    | 106/191 [00:02<00:02, 40.17batch/s]

Training batches on cuda:  58%|█████▊    | 111/191 [00:02<00:01, 40.23batch/s]

Training batches on cuda:  61%|██████    | 116/191 [00:03<00:01, 38.89batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:03<00:01, 38.91batch/s]

Training batches on cuda:  65%|██████▌   | 125/191 [00:03<00:01, 38.76batch/s]

Training batches on cuda:  68%|██████▊   | 129/191 [00:03<00:01, 38.47batch/s]

Training batches on cuda:  70%|██████▉   | 133/191 [00:03<00:01, 38.61batch/s]

Training batches on cuda:  72%|███████▏  | 138/191 [00:03<00:01, 38.92batch/s]

Training batches on cuda:  75%|███████▍  | 143/191 [00:03<00:01, 39.42batch/s]

Training batches on cuda:  77%|███████▋  | 147/191 [00:03<00:01, 38.75batch/s]

Training batches on cuda:  79%|███████▉  | 151/191 [00:03<00:01, 38.14batch/s]

Training batches on cuda:  82%|████████▏ | 156/191 [00:04<00:00, 38.86batch/s]

Training batches on cuda:  84%|████████▍ | 161/191 [00:04<00:00, 39.25batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:04<00:00, 39.34batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:04<00:00, 39.51batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:04<00:00, 39.35batch/s]

Training batches on cuda:  93%|█████████▎| 178/191 [00:04<00:00, 38.84batch/s]

Training batches on cuda:  95%|█████████▌| 182/191 [00:04<00:00, 39.06batch/s]

Training batches on cuda:  97%|█████████▋| 186/191 [00:04<00:00, 39.31batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:04<00:00, 39.13batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:25<00:30,  5.08s/epoch, loss=0.00081, prev_loss=0.00081]
Training epochs on cuda:  50%|█████     | 5/10 [00:25<00:25,  5.12s/epoch, loss=0.00081, prev_loss=0.00081]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/191 [00:00<00:10, 17.40batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:08, 20.80batch/s]

Training batches on cuda:   6%|▌         | 11/191 [00:00<00:07, 24.32batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:06, 27.37batch/s]

Training batches on cuda:  10%|█         | 20/191 [00:00<00:05, 30.38batch/s]

Training batches on cuda:  13%|█▎        | 25/191 [00:00<00:05, 32.88batch/s]

Training batches on cuda:  16%|█▌        | 30/191 [00:00<00:04, 34.84batch/s]

Training batches on cuda:  18%|█▊        | 34/191 [00:00<00:04, 36.19batch/s]

Training batches on cuda:  20%|██        | 39/191 [00:01<00:04, 37.45batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:01<00:03, 37.98batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:04, 34.77batch/s]

Training batches on cuda:  27%|██▋       | 51/191 [00:01<00:03, 36.07batch/s]

Training batches on cuda:  29%|██▉       | 55/191 [00:01<00:03, 36.91batch/s]

Training batches on cuda:  31%|███▏      | 60/191 [00:01<00:03, 38.18batch/s]

Training batches on cuda:  34%|███▍      | 65/191 [00:01<00:03, 39.07batch/s]

Training batches on cuda:  37%|███▋      | 70/191 [00:01<00:03, 39.85batch/s]

Training batches on cuda:  39%|███▉      | 75/191 [00:01<00:02, 40.00batch/s]

Training batches on cuda:  42%|████▏     | 80/191 [00:02<00:02, 38.74batch/s]

Training batches on cuda:  45%|████▍     | 85/191 [00:02<00:02, 39.43batch/s]

Training batches on cuda:  47%|████▋     | 89/191 [00:02<00:02, 38.99batch/s]

Training batches on cuda:  49%|████▉     | 94/191 [00:02<00:02, 39.88batch/s]

Training batches on cuda:  52%|█████▏    | 99/191 [00:02<00:02, 40.28batch/s]

Training batches on cuda:  54%|█████▍    | 104/191 [00:02<00:02, 40.44batch/s]

Training batches on cuda:  57%|█████▋    | 109/191 [00:02<00:02, 40.21batch/s]

Training batches on cuda:  60%|█████▉    | 114/191 [00:02<00:01, 40.06batch/s]

Training batches on cuda:  62%|██████▏   | 119/191 [00:03<00:01, 39.96batch/s]

Training batches on cuda:  64%|██████▍   | 123/191 [00:03<00:01, 39.87batch/s]

Training batches on cuda:  66%|██████▋   | 127/191 [00:03<00:01, 39.67batch/s]

Training batches on cuda:  69%|██████▊   | 131/191 [00:03<00:01, 39.55batch/s]

Training batches on cuda:  71%|███████   | 135/191 [00:03<00:01, 39.45batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 39.60batch/s]

Training batches on cuda:  75%|███████▌  | 144/191 [00:03<00:01, 38.95batch/s]

Training batches on cuda:  78%|███████▊  | 149/191 [00:03<00:01, 39.41batch/s]

Training batches on cuda:  80%|████████  | 153/191 [00:03<00:00, 39.43batch/s]

Training batches on cuda:  82%|████████▏ | 157/191 [00:04<00:00, 38.84batch/s]

Training batches on cuda:  84%|████████▍ | 161/191 [00:04<00:00, 38.79batch/s]

Training batches on cuda:  87%|████████▋ | 166/191 [00:04<00:00, 39.01batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:04<00:00, 38.22batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:04<00:00, 36.95batch/s]

Training batches on cuda:  93%|█████████▎| 178/191 [00:04<00:00, 36.75batch/s]

Training batches on cuda:  96%|█████████▌| 183/191 [00:04<00:00, 37.92batch/s]

Training batches on cuda:  98%|█████████▊| 188/191 [00:04<00:00, 38.86batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:25,  5.12s/epoch, loss=0.00081, prev_loss=0.00081]
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:20,  5.13s/epoch, loss=0.00081, prev_loss=0.00081]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/191 [00:00<00:20,  9.34batch/s]

Training batches on cuda:   3%|▎         | 5/191 [00:00<00:15, 11.93batch/s]

Training batches on cuda:   5%|▍         | 9/191 [00:00<00:12, 14.95batch/s]

Training batches on cuda:   7%|▋         | 14/191 [00:00<00:09, 18.40batch/s]

Training batches on cuda:   9%|▉         | 18/191 [00:00<00:07, 21.93batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:06, 25.44batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:05, 28.54batch/s]

Training batches on cuda:  17%|█▋        | 32/191 [00:00<00:05, 30.84batch/s]

Training batches on cuda:  19%|█▉        | 36/191 [00:01<00:04, 32.92batch/s]

Training batches on cuda:  21%|██        | 40/191 [00:01<00:04, 32.22batch/s]

Training batches on cuda:  24%|██▎       | 45/191 [00:01<00:04, 34.65batch/s]

Training batches on cuda:  26%|██▌       | 50/191 [00:01<00:03, 36.58batch/s]

Training batches on cuda:  29%|██▉       | 55/191 [00:01<00:03, 37.99batch/s]

Training batches on cuda:  31%|███▏      | 60/191 [00:01<00:03, 39.51batch/s]

Training batches on cuda:  34%|███▍      | 65/191 [00:01<00:03, 40.60batch/s]

Training batches on cuda:  37%|███▋      | 70/191 [00:01<00:02, 41.06batch/s]

Training batches on cuda:  39%|███▉      | 75/191 [00:01<00:02, 41.34batch/s]

Training batches on cuda:  42%|████▏     | 80/191 [00:02<00:02, 41.39batch/s]

Training batches on cuda:  45%|████▍     | 85/191 [00:02<00:02, 41.23batch/s]

Training batches on cuda:  47%|████▋     | 90/191 [00:02<00:02, 40.83batch/s]

Training batches on cuda:  50%|████▉     | 95/191 [00:02<00:02, 40.36batch/s]

Training batches on cuda:  52%|█████▏    | 100/191 [00:02<00:02, 40.81batch/s]

Training batches on cuda:  55%|█████▍    | 105/191 [00:02<00:02, 40.77batch/s]

Training batches on cuda:  58%|█████▊    | 110/191 [00:02<00:01, 40.73batch/s]

Training batches on cuda:  60%|██████    | 115/191 [00:02<00:01, 41.05batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:03<00:01, 41.84batch/s]

Training batches on cuda:  65%|██████▌   | 125/191 [00:03<00:01, 42.44batch/s]

Training batches on cuda:  68%|██████▊   | 130/191 [00:03<00:01, 42.34batch/s]

Training batches on cuda:  71%|███████   | 135/191 [00:03<00:01, 41.91batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 41.30batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 41.17batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:01, 40.33batch/s]

Training batches on cuda:  81%|████████  | 155/191 [00:03<00:00, 40.77batch/s]

Training batches on cuda:  84%|████████▍ | 160/191 [00:04<00:00, 40.99batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:04<00:00, 41.36batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:04<00:00, 41.28batch/s]

Training batches on cuda:  92%|█████████▏| 175/191 [00:04<00:00, 40.45batch/s]

Training batches on cuda:  94%|█████████▍| 180/191 [00:04<00:00, 40.23batch/s]

Training batches on cuda:  97%|█████████▋| 185/191 [00:04<00:00, 39.66batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:04<00:00, 40.24batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:35<00:20,  5.13s/epoch, loss=0.00081, prev_loss=0.00081]
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:15,  5.10s/epoch, loss=0.00081, prev_loss=0.00081]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/191 [00:00<00:10, 18.71batch/s]

Training batches on cuda:   3%|▎         | 6/191 [00:00<00:08, 22.19batch/s]

Training batches on cuda:   6%|▌         | 11/191 [00:00<00:06, 25.79batch/s]

Training batches on cuda:   8%|▊         | 15/191 [00:00<00:06, 28.25batch/s]

Training batches on cuda:  10%|▉         | 19/191 [00:00<00:05, 30.70batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:05, 32.82batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:04, 34.78batch/s]

Training batches on cuda:  17%|█▋        | 32/191 [00:00<00:04, 35.57batch/s]

Training batches on cuda:  19%|█▉        | 37/191 [00:01<00:04, 36.98batch/s]

Training batches on cuda:  21%|██▏       | 41/191 [00:01<00:04, 36.56batch/s]

Training batches on cuda:  24%|██▎       | 45/191 [00:01<00:03, 36.51batch/s]

Training batches on cuda:  26%|██▌       | 49/191 [00:01<00:03, 37.31batch/s]

Training batches on cuda:  28%|██▊       | 53/191 [00:01<00:03, 37.68batch/s]

Training batches on cuda:  30%|██▉       | 57/191 [00:01<00:03, 38.24batch/s]

Training batches on cuda:  32%|███▏      | 61/191 [00:01<00:03, 38.18batch/s]

Training batches on cuda:  35%|███▍      | 66/191 [00:01<00:03, 38.65batch/s]

Training batches on cuda:  37%|███▋      | 70/191 [00:01<00:03, 38.18batch/s]

Training batches on cuda:  39%|███▉      | 75/191 [00:01<00:02, 38.70batch/s]

Training batches on cuda:  42%|████▏     | 80/191 [00:02<00:02, 38.99batch/s]

Training batches on cuda:  45%|████▍     | 85/191 [00:02<00:02, 39.12batch/s]

Training batches on cuda:  47%|████▋     | 90/191 [00:02<00:02, 39.37batch/s]

Training batches on cuda:  49%|████▉     | 94/191 [00:02<00:02, 39.23batch/s]

Training batches on cuda:  51%|█████▏    | 98/191 [00:02<00:02, 38.87batch/s]

Training batches on cuda:  53%|█████▎    | 102/191 [00:02<00:02, 39.07batch/s]

Training batches on cuda:  56%|█████▌    | 107/191 [00:02<00:02, 39.40batch/s]

Training batches on cuda:  59%|█████▊    | 112/191 [00:02<00:01, 39.89batch/s]

Training batches on cuda:  61%|██████    | 116/191 [00:03<00:01, 39.58batch/s]

Training batches on cuda:  63%|██████▎   | 120/191 [00:03<00:01, 37.75batch/s]

Training batches on cuda:  65%|██████▍   | 124/191 [00:03<00:01, 36.02batch/s]

Training batches on cuda:  67%|██████▋   | 128/191 [00:03<00:01, 36.62batch/s]

Training batches on cuda:  69%|██████▉   | 132/191 [00:03<00:01, 36.55batch/s]

Training batches on cuda:  71%|███████   | 136/191 [00:03<00:01, 36.94batch/s]

Training batches on cuda:  74%|███████▍  | 141/191 [00:03<00:01, 38.19batch/s]

Training batches on cuda:  76%|███████▌  | 145/191 [00:03<00:01, 37.61batch/s]

Training batches on cuda:  78%|███████▊  | 149/191 [00:03<00:01, 37.37batch/s]

Training batches on cuda:  80%|████████  | 153/191 [00:04<00:01, 37.23batch/s]

Training batches on cuda:  82%|████████▏ | 157/191 [00:04<00:00, 36.39batch/s]

Training batches on cuda:  84%|████████▍ | 161/191 [00:04<00:00, 36.76batch/s]

Training batches on cuda:  86%|████████▋ | 165/191 [00:04<00:00, 36.99batch/s]

Training batches on cuda:  88%|████████▊ | 169/191 [00:04<00:00, 37.54batch/s]

Training batches on cuda:  91%|█████████ | 173/191 [00:04<00:00, 37.80batch/s]

Training batches on cuda:  93%|█████████▎| 177/191 [00:04<00:00, 37.44batch/s]

Training batches on cuda:  95%|█████████▍| 181/191 [00:04<00:00, 38.12batch/s]

Training batches on cuda:  97%|█████████▋| 185/191 [00:04<00:00, 38.30batch/s]

Training batches on cuda:  99%|█████████▉| 189/191 [00:04<00:00, 38.70batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:41<00:15,  5.10s/epoch, loss=0.00081, prev_loss=0.00081]
Training epochs on cuda:  80%|████████  | 8/10 [00:41<00:10,  5.17s/epoch, loss=0.00081, prev_loss=0.00081]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:06, 27.11batch/s]

Training batches on cuda:   4%|▍         | 8/191 [00:00<00:06, 30.26batch/s]

Training batches on cuda:   7%|▋         | 13/191 [00:00<00:05, 33.05batch/s]

Training batches on cuda:   9%|▉         | 18/191 [00:00<00:04, 35.62batch/s]

Training batches on cuda:  12%|█▏        | 23/191 [00:00<00:04, 37.61batch/s]

Training batches on cuda:  15%|█▍        | 28/191 [00:00<00:04, 39.13batch/s]

Training batches on cuda:  17%|█▋        | 32/191 [00:00<00:04, 36.92batch/s]

Training batches on cuda:  19%|█▉        | 37/191 [00:00<00:04, 38.39batch/s]

Training batches on cuda:  22%|██▏       | 42/191 [00:01<00:03, 39.58batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:03, 40.43batch/s]

Training batches on cuda:  27%|██▋       | 52/191 [00:01<00:03, 40.89batch/s]

Training batches on cuda:  30%|██▉       | 57/191 [00:01<00:03, 41.40batch/s]

Training batches on cuda:  32%|███▏      | 62/191 [00:01<00:03, 41.89batch/s]

Training batches on cuda:  35%|███▌      | 67/191 [00:01<00:02, 41.55batch/s]

Training batches on cuda:  38%|███▊      | 72/191 [00:01<00:02, 41.58batch/s]

Training batches on cuda:  40%|████      | 77/191 [00:01<00:02, 41.34batch/s]

Training batches on cuda:  43%|████▎     | 82/191 [00:02<00:02, 40.58batch/s]

Training batches on cuda:  46%|████▌     | 87/191 [00:02<00:02, 40.79batch/s]

Training batches on cuda:  48%|████▊     | 92/191 [00:02<00:02, 40.76batch/s]

Training batches on cuda:  51%|█████     | 97/191 [00:02<00:02, 40.59batch/s]

Training batches on cuda:  53%|█████▎    | 102/191 [00:02<00:02, 40.88batch/s]

Training batches on cuda:  56%|█████▌    | 107/191 [00:02<00:02, 40.82batch/s]

Training batches on cuda:  59%|█████▊    | 112/191 [00:02<00:01, 39.91batch/s]

Training batches on cuda:  61%|██████    | 116/191 [00:02<00:01, 39.68batch/s]

Training batches on cuda:  63%|██████▎   | 121/191 [00:02<00:01, 40.39batch/s]

Training batches on cuda:  66%|██████▌   | 126/191 [00:03<00:01, 40.85batch/s]

Training batches on cuda:  69%|██████▊   | 131/191 [00:03<00:01, 40.50batch/s]

Training batches on cuda:  71%|███████   | 136/191 [00:03<00:01, 39.94batch/s]

Training batches on cuda:  73%|███████▎  | 140/191 [00:03<00:01, 39.55batch/s]

Training batches on cuda:  75%|███████▌  | 144/191 [00:03<00:01, 39.43batch/s]

Training batches on cuda:  78%|███████▊  | 149/191 [00:03<00:01, 39.99batch/s]

Training batches on cuda:  81%|████████  | 154/191 [00:03<00:00, 39.48batch/s]

Training batches on cuda:  83%|████████▎ | 158/191 [00:03<00:00, 38.96batch/s]

Training batches on cuda:  85%|████████▍ | 162/191 [00:04<00:00, 37.86batch/s]

Training batches on cuda:  87%|████████▋ | 166/191 [00:04<00:00, 36.14batch/s]

Training batches on cuda:  89%|████████▉ | 170/191 [00:04<00:00, 36.06batch/s]

Training batches on cuda:  91%|█████████ | 174/191 [00:04<00:00, 36.22batch/s]

Training batches on cuda:  93%|█████████▎| 178/191 [00:04<00:00, 36.14batch/s]

Training batches on cuda:  95%|█████████▌| 182/191 [00:04<00:00, 36.17batch/s]

Training batches on cuda:  97%|█████████▋| 186/191 [00:04<00:00, 36.18batch/s]

Training batches on cuda:  99%|█████████▉| 190/191 [00:04<00:00, 35.54batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:46<00:10,  5.17s/epoch, loss=0.00081, prev_loss=0.00081]
Training epochs on cuda:  90%|█████████ | 9/10 [00:46<00:05,  5.14s/epoch, loss=0.00081, prev_loss=0.00081]

Training batches on cuda:   0%|          | 0/191 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/191 [00:00<00:07, 26.40batch/s]

Training batches on cuda:   4%|▎         | 7/191 [00:00<00:06, 27.32batch/s]

Training batches on cuda:   6%|▋         | 12/191 [00:00<00:05, 30.37batch/s]

Training batches on cuda:   9%|▉         | 17/191 [00:00<00:05, 33.19batch/s]

Training batches on cuda:  12%|█▏        | 22/191 [00:00<00:04, 35.01batch/s]

Training batches on cuda:  14%|█▎        | 26/191 [00:00<00:04, 36.08batch/s]

Training batches on cuda:  16%|█▌        | 30/191 [00:00<00:04, 34.78batch/s]

Training batches on cuda:  18%|█▊        | 34/191 [00:00<00:04, 36.06batch/s]

Training batches on cuda:  20%|█▉        | 38/191 [00:01<00:04, 36.76batch/s]

Training batches on cuda:  23%|██▎       | 43/191 [00:01<00:03, 37.62batch/s]

Training batches on cuda:  25%|██▍       | 47/191 [00:01<00:03, 38.14batch/s]

Training batches on cuda:  27%|██▋       | 51/191 [00:01<00:03, 38.53batch/s]

Training batches on cuda:  29%|██▉       | 56/191 [00:01<00:03, 39.20batch/s]

Training batches on cuda:  31%|███▏      | 60/191 [00:01<00:03, 38.93batch/s]

Training batches on cuda:  34%|███▎      | 64/191 [00:01<00:03, 38.79batch/s]

Training batches on cuda:  36%|███▌      | 68/191 [00:01<00:03, 39.14batch/s]

Training batches on cuda:  38%|███▊      | 73/191 [00:01<00:02, 39.63batch/s]

Training batches on cuda:  40%|████      | 77/191 [00:02<00:02, 39.54batch/s]

Training batches on cuda:  42%|████▏     | 81/191 [00:02<00:02, 39.55batch/s]

Training batches on cuda:  45%|████▍     | 85/191 [00:02<00:02, 39.64batch/s]

Training batches on cuda:  47%|████▋     | 89/191 [00:02<00:02, 38.31batch/s]

Training batches on cuda:  49%|████▊     | 93/191 [00:02<00:02, 38.73batch/s]

Training batches on cuda:  51%|█████     | 97/191 [00:02<00:02, 37.05batch/s]

Training batches on cuda:  53%|█████▎    | 102/191 [00:02<00:02, 37.49batch/s]

Training batches on cuda:  55%|█████▌    | 106/191 [00:02<00:02, 37.36batch/s]

Training batches on cuda:  58%|█████▊    | 111/191 [00:02<00:02, 38.06batch/s]

Training batches on cuda:  60%|██████    | 115/191 [00:03<00:02, 36.91batch/s]

Training batches on cuda:  62%|██████▏   | 119/191 [00:03<00:01, 36.28batch/s]

Training batches on cuda:  64%|██████▍   | 123/191 [00:03<00:01, 36.32batch/s]

Training batches on cuda:  67%|██████▋   | 128/191 [00:03<00:01, 37.58batch/s]

Training batches on cuda:  70%|██████▉   | 133/191 [00:03<00:01, 38.20batch/s]

Training batches on cuda:  72%|███████▏  | 138/191 [00:03<00:01, 38.95batch/s]

Training batches on cuda:  74%|███████▍  | 142/191 [00:03<00:01, 38.92batch/s]

Training batches on cuda:  76%|███████▋  | 146/191 [00:03<00:01, 38.63batch/s]

Training batches on cuda:  79%|███████▊  | 150/191 [00:03<00:01, 37.97batch/s]

Training batches on cuda:  81%|████████  | 154/191 [00:04<00:00, 37.63batch/s]

Training batches on cuda:  83%|████████▎ | 159/191 [00:04<00:00, 38.84batch/s]

Training batches on cuda:  86%|████████▌ | 164/191 [00:04<00:00, 39.45batch/s]

Training batches on cuda:  88%|████████▊ | 168/191 [00:04<00:00, 39.41batch/s]

Training batches on cuda:  90%|█████████ | 172/191 [00:04<00:00, 38.46batch/s]

Training batches on cuda:  92%|█████████▏| 176/191 [00:04<00:00, 37.75batch/s]

Training batches on cuda:  94%|█████████▍| 180/191 [00:04<00:00, 37.17batch/s]

Training batches on cuda:  96%|█████████▋| 184/191 [00:04<00:00, 37.07batch/s]

Training batches on cuda:  98%|█████████▊| 188/191 [00:04<00:00, 34.35batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:51<00:05,  5.14s/epoch, loss=0.000808, prev_loss=0.00081]2021-09-16:18:54:30,834 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:54:32,104 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:54:47,199 INFO     [evaluator.py:657] Evaluation took 15.09s seconds
2021-09-16:18:54:47,395 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00, 10.20s/epoch, loss=0.000808, prev_loss=0.00081]
Training epochs on cuda: 100%|██████████| 10/10 [01:08<00:00,  6.82s/epoch, loss=0.000808, prev_loss=0.00081]
2021-09-16:18:54:47,396 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp0gsdus6b'
2021-09-16:18:54:47,456 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp0gsdus6b' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.36ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:14, 2.43ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.49ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:13, 2.53ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.56ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.58ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.60ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:11, 2.59ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.60ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.60ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.62ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.62ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.61ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.62ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.62ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.62ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.62ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.62ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.56ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.58ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.59ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.60ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.59ktriple/s]
2021-09-16:18:55:02,464 INFO     [evaluator.py:657] Evaluation took 14.99s seconds
[I 2021-09-16 18:55:02,469] Trial 63 finished with value: 0.07176131577911948 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.04480188208670142, 'optimizer.lr': 0.06107450429319164, 'negative_sampler.num_negs_per_pos': 36, 'training.batch_size': 1632}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:55:02,494 WARNING  [api.py:823] No random seed is specified. Setting to 2746782089.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/134 [00:00<00:03, 33.40batch/s]

Training batches on cuda:   7%|▋         | 10/134 [00:00<00:03, 37.66batch/s]

Training batches on cuda:  12%|█▏        | 16/134 [00:00<00:02, 42.05batch/s]

Training batches on cuda:  16%|█▋        | 22/134 [00:00<00:02, 45.67batch/s]

Training batches on cuda:  21%|██        | 28/134 [00:00<00:02, 48.50batch/s]

Training batches on cuda:  25%|██▌       | 34/134 [00:00<00:01, 51.02batch/s]

Training batches on cuda:  30%|██▉       | 40/134 [00:00<00:01, 52.59batch/s]

Training batches on cuda:  34%|███▍      | 46/134 [00:00<00:01, 54.03batch/s]

Training batches on cuda:  39%|███▉      | 52/134 [00:00<00:01, 54.41batch/s]

Training batches on cuda:  43%|████▎     | 58/134 [00:01<00:01, 54.41batch/s]

Training batches on cuda:  48%|████▊     | 64/134 [00:01<00:01, 55.97batch/s]

Training batches on cuda:  52%|█████▏    | 70/134 [00:01<00:01, 56.98batch/s]

Training batches on cuda:  57%|█████▋    | 76/134 [00:01<00:01, 57.67batch/s]

Training batches on cuda:  61%|██████    | 82/134 [00:01<00:00, 56.66batch/s]

Training batches on cuda:  66%|██████▌   | 88/134 [00:01<00:00, 56.84batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:01<00:00, 56.64batch/s]

Training batches on cuda:  75%|███████▍  | 100/134 [00:01<00:00, 56.15batch/s]

Training batches on cuda:  79%|███████▉  | 106/134 [00:01<00:00, 56.45batch/s]

Training batches on cuda:  84%|████████▎ | 112/134 [00:02<00:00, 56.46batch/s]

Training batches on cuda:  88%|████████▊ | 118/134 [00:02<00:00, 56.50batch/s]

Training batches on cuda:  93%|█████████▎| 124/134 [00:02<00:00, 56.34batch/s]

Training batches on cuda:  97%|█████████▋| 130/134 [00:02<00:00, 56.68batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000574, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:23,  2.60s/epoch, loss=0.000574, prev_loss=nan]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/134 [00:00<00:07, 16.98batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:06, 21.03batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:00<00:04, 25.80batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:00<00:03, 30.91batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:00<00:03, 35.63batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:00<00:02, 40.08batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:00<00:02, 43.60batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:00<00:01, 45.57batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:00<00:01, 46.77batch/s]

Training batches on cuda:  40%|████      | 54/134 [00:01<00:01, 47.64batch/s]

Training batches on cuda:  45%|████▍     | 60/134 [00:01<00:01, 49.58batch/s]

Training batches on cuda:  49%|████▉     | 66/134 [00:01<00:01, 50.58batch/s]

Training batches on cuda:  54%|█████▎    | 72/134 [00:01<00:01, 51.82batch/s]

Training batches on cuda:  58%|█████▊    | 78/134 [00:01<00:01, 52.27batch/s]

Training batches on cuda:  63%|██████▎   | 84/134 [00:01<00:00, 52.93batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:01<00:00, 55.99batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:01<00:00, 56.46batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:01<00:00, 56.51batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:02<00:00, 57.25batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:02<00:00, 57.58batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:02<00:00, 56.93batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:02<00:00, 57.67batch/s]

Training batches on cuda: 100%|██████████| 134/134 [00:02<00:00, 59.42batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:23,  2.60s/epoch, loss=0.000569, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:21,  2.63s/epoch, loss=0.000569, prev_loss=nan]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/134 [00:00<00:03, 35.96batch/s]

Training batches on cuda:   7%|▋         | 10/134 [00:00<00:03, 40.50batch/s]

Training batches on cuda:  12%|█▏        | 16/134 [00:00<00:02, 44.72batch/s]

Training batches on cuda:  16%|█▋        | 22/134 [00:00<00:02, 47.95batch/s]

Training batches on cuda:  21%|██        | 28/134 [00:00<00:02, 50.46batch/s]

Training batches on cuda:  25%|██▌       | 34/134 [00:00<00:01, 52.33batch/s]

Training batches on cuda:  30%|██▉       | 40/134 [00:00<00:01, 53.34batch/s]

Training batches on cuda:  34%|███▍      | 46/134 [00:00<00:01, 54.13batch/s]

Training batches on cuda:  39%|███▉      | 52/134 [00:00<00:01, 54.40batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:01<00:01, 56.47batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:01<00:01, 57.48batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:01<00:01, 55.28batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:01<00:01, 56.61batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:01<00:00, 55.03batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:01<00:00, 53.64batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:01<00:00, 52.96batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:01<00:00, 52.66batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:01<00:00, 54.27batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:02<00:00, 54.78batch/s]

Training batches on cuda:  90%|████████▉ | 120/134 [00:02<00:00, 56.84batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:02<00:00, 55.95batch/s]

Training batches on cuda:  99%|█████████▊| 132/134 [00:02<00:00, 52.61batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:21,  2.63s/epoch, loss=0.000569, prev_loss=0.000569]
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:18,  2.63s/epoch, loss=0.000569, prev_loss=0.000569]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:04, 26.74batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:03, 31.54batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:00<00:03, 35.46batch/s]

Training batches on cuda:  15%|█▍        | 20/134 [00:00<00:02, 38.75batch/s]

Training batches on cuda:  19%|█▉        | 26/134 [00:00<00:02, 41.93batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:00<00:02, 42.59batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:00<00:02, 45.11batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:00<00:01, 47.60batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:01<00:01, 49.72batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:01<00:01, 50.43batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:01<00:01, 52.03batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:01<00:01, 52.58batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:01<00:01, 53.23batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:01<00:01, 51.06batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:01<00:00, 51.34batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:01<00:00, 52.35batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:01<00:00, 53.44batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:02<00:00, 54.36batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:02<00:00, 54.97batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:02<00:00, 55.67batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:02<00:00, 56.04batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:02<00:00, 55.64batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:02<00:00, 56.61batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:18,  2.63s/epoch, loss=0.000569, prev_loss=0.000569]
Training epochs on cuda:  40%|████      | 4/10 [00:10<00:16,  2.69s/epoch, loss=0.000569, prev_loss=0.000569]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:05, 25.75batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:04, 30.76batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:00<00:03, 35.94batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:00<00:02, 40.40batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:00<00:02, 44.19batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:00<00:02, 45.62batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:00<00:01, 47.68batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:00<00:01, 50.36batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:00<00:01, 51.25batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:01<00:01, 52.66batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:01<00:01, 54.35batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:01<00:01, 54.91batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:01<00:01, 55.61batch/s]

Training batches on cuda:  61%|██████    | 82/134 [00:01<00:00, 57.26batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:01<00:00, 59.09batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:01<00:00, 58.96batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:01<00:00, 59.19batch/s]

Training batches on cuda:  81%|████████  | 108/134 [00:01<00:00, 59.49batch/s]

Training batches on cuda:  85%|████████▌ | 114/134 [00:02<00:00, 58.54batch/s]

Training batches on cuda:  90%|████████▉ | 120/134 [00:02<00:00, 58.65batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:02<00:00, 52.13batch/s]

Training batches on cuda:  99%|█████████▊| 132/134 [00:02<00:00, 50.86batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:13<00:16,  2.69s/epoch, loss=0.000569, prev_loss=0.000569]
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:13,  2.69s/epoch, loss=0.000569, prev_loss=0.000569]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:02, 44.49batch/s]

Training batches on cuda:   9%|▉         | 12/134 [00:00<00:02, 48.89batch/s]

Training batches on cuda:  13%|█▎        | 18/134 [00:00<00:02, 51.18batch/s]

Training batches on cuda:  18%|█▊        | 24/134 [00:00<00:02, 51.18batch/s]

Training batches on cuda:  22%|██▏       | 30/134 [00:00<00:02, 51.18batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:00<00:01, 49.74batch/s]

Training batches on cuda:  30%|██▉       | 40/134 [00:00<00:01, 49.37batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:00<00:01, 48.39batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:00<00:01, 49.91batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:01<00:01, 52.31batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:01<00:01, 53.66batch/s]

Training batches on cuda:  52%|█████▏    | 70/134 [00:01<00:01, 56.26batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:01<00:00, 58.91batch/s]

Training batches on cuda:  63%|██████▎   | 84/134 [00:01<00:00, 60.64batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:01<00:00, 61.98batch/s]

Training batches on cuda:  73%|███████▎  | 98/134 [00:01<00:00, 62.93batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:01<00:00, 64.17batch/s]

Training batches on cuda:  84%|████████▎ | 112/134 [00:01<00:00, 65.01batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:02<00:00, 65.87batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:02<00:00, 66.33batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:02<00:00, 65.60batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:13,  2.69s/epoch, loss=0.000569, prev_loss=0.000569]
Training epochs on cuda:  60%|██████    | 6/10 [00:15<00:10,  2.62s/epoch, loss=0.000569, prev_loss=0.000569]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/134 [00:00<00:03, 34.44batch/s]

Training batches on cuda:   7%|▋         | 10/134 [00:00<00:03, 39.42batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:00<00:02, 44.79batch/s]

Training batches on cuda:  18%|█▊        | 24/134 [00:00<00:02, 49.55batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:00<00:01, 53.34batch/s]

Training batches on cuda:  28%|██▊       | 38/134 [00:00<00:01, 56.36batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:00<00:01, 59.12batch/s]

Training batches on cuda:  39%|███▉      | 52/134 [00:00<00:01, 60.92batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:00<00:01, 62.45batch/s]

Training batches on cuda:  49%|████▉     | 66/134 [00:01<00:01, 62.39batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:01<00:00, 61.57batch/s]

Training batches on cuda:  60%|█████▉    | 80/134 [00:01<00:00, 61.05batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:01<00:00, 60.69batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:01<00:00, 60.56batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:01<00:00, 60.67batch/s]

Training batches on cuda:  81%|████████  | 108/134 [00:01<00:00, 60.39batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:01<00:00, 58.82batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:02<00:00, 57.09batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:02<00:00, 57.65batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:02<00:00, 56.13batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:10,  2.62s/epoch, loss=0.000569, prev_loss=0.000569]
Training epochs on cuda:  70%|███████   | 7/10 [00:18<00:07,  2.57s/epoch, loss=0.000569, prev_loss=0.000569]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/134 [00:00<00:03, 34.35batch/s]

Training batches on cuda:   7%|▋         | 10/134 [00:00<00:03, 38.49batch/s]

Training batches on cuda:  12%|█▏        | 16/134 [00:00<00:02, 42.92batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:00<00:02, 44.37batch/s]

Training batches on cuda:  21%|██        | 28/134 [00:00<00:02, 48.96batch/s]

Training batches on cuda:  25%|██▌       | 34/134 [00:00<00:02, 49.65batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:00<00:01, 48.98batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:00<00:01, 51.47batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:00<00:01, 52.98batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:01<00:01, 53.13batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:01<00:01, 53.56batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:01<00:01, 54.01batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:01<00:01, 54.72batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:01<00:00, 53.74batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:01<00:00, 54.28batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:01<00:00, 55.10batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:01<00:00, 53.68batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:01<00:00, 51.62batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:02<00:00, 53.61batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:02<00:00, 53.37batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:02<00:00, 54.30batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:02<00:00, 55.46batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:21<00:07,  2.57s/epoch, loss=0.000569, prev_loss=0.000569]
Training epochs on cuda:  80%|████████  | 8/10 [00:21<00:05,  2.62s/epoch, loss=0.000569, prev_loss=0.000569]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:14,  9.10batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:10, 11.98batch/s]

Training batches on cuda:   9%|▉         | 12/134 [00:00<00:07, 15.59batch/s]

Training batches on cuda:  13%|█▎        | 18/134 [00:00<00:05, 19.75batch/s]

Training batches on cuda:  18%|█▊        | 24/134 [00:00<00:04, 24.29batch/s]

Training batches on cuda:  22%|██▏       | 30/134 [00:00<00:03, 29.15batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:00<00:02, 35.05batch/s]

Training batches on cuda:  33%|███▎      | 44/134 [00:00<00:02, 40.32batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:01<00:01, 44.86batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:01<00:01, 48.44batch/s]

Training batches on cuda:  48%|████▊     | 64/134 [00:01<00:01, 51.44batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:01<00:01, 54.53batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:01<00:01, 55.98batch/s]

Training batches on cuda:  63%|██████▎   | 84/134 [00:01<00:00, 58.47batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:01<00:00, 60.54batch/s]

Training batches on cuda:  73%|███████▎  | 98/134 [00:01<00:00, 62.01batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:01<00:00, 63.40batch/s]

Training batches on cuda:  84%|████████▎ | 112/134 [00:01<00:00, 63.58batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:02<00:00, 65.21batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:02<00:00, 66.30batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:02<00:00, 66.92batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:05,  2.62s/epoch, loss=0.000569, prev_loss=0.000569]
Training epochs on cuda:  90%|█████████ | 9/10 [00:23<00:02,  2.59s/epoch, loss=0.000569, prev_loss=0.000569]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/134 [00:00<00:03, 37.68batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:00<00:02, 42.80batch/s]

Training batches on cuda:  13%|█▎        | 18/134 [00:00<00:02, 47.01batch/s]

Training batches on cuda:  18%|█▊        | 24/134 [00:00<00:02, 50.25batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:00<00:01, 53.08batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:00<00:01, 54.75batch/s]

Training batches on cuda:  33%|███▎      | 44/134 [00:00<00:01, 55.75batch/s]

Training batches on cuda:  37%|███▋      | 50/134 [00:00<00:01, 56.96batch/s]

Training batches on cuda:  42%|████▏     | 56/134 [00:00<00:01, 55.68batch/s]

Training batches on cuda:  46%|████▋     | 62/134 [00:01<00:01, 55.64batch/s]

Training batches on cuda:  51%|█████     | 68/134 [00:01<00:01, 55.47batch/s]

Training batches on cuda:  55%|█████▌    | 74/134 [00:01<00:01, 55.16batch/s]

Training batches on cuda:  60%|█████▉    | 80/134 [00:01<00:00, 55.84batch/s]

Training batches on cuda:  64%|██████▍   | 86/134 [00:01<00:00, 56.27batch/s]

Training batches on cuda:  69%|██████▊   | 92/134 [00:01<00:00, 54.55batch/s]

Training batches on cuda:  73%|███████▎  | 98/134 [00:01<00:00, 55.31batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:01<00:00, 57.11batch/s]

Training batches on cuda:  84%|████████▎ | 112/134 [00:01<00:00, 58.64batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:02<00:00, 59.62batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:02<00:00, 60.28batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:02<00:00, 60.70batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:26<00:02,  2.59s/epoch, loss=0.000569, prev_loss=0.000569]2021-09-16:18:55:29,202 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:55:30,513 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:55:45,599 INFO     [evaluator.py:657] Evaluation took 15.08s seconds
2021-09-16:18:55:45,729 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:42<00:00,  7.52s/epoch, loss=0.000569, prev_loss=0.000569]
Training epochs on cuda: 100%|██████████| 10/10 [00:42<00:00,  4.26s/epoch, loss=0.000569, prev_loss=0.000569]
2021-09-16:18:55:45,729 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmppf8gv8lj'
2021-09-16:18:55:45,818 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmppf8gv8lj' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.53ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.60ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.61ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.62ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:18:56:00,697 INFO     [evaluator.py:657] Evaluation took 14.85s seconds
[I 2021-09-16 18:56:00,702] Trial 64 finished with value: 0.007276044347863331 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.1219787944751227, 'optimizer.lr': 0.0537234829937943, 'negative_sampler.num_negs_per_pos': 6, 'training.batch_size': 2332}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:56:00,725 WARNING  [api.py:823] No random seed is specified. Setting to 596383326.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:07, 18.88batch/s]

Training batches on cuda:   4%|▍         | 6/146 [00:00<00:06, 21.60batch/s]

Training batches on cuda:   7%|▋         | 10/146 [00:00<00:05, 24.05batch/s]

Training batches on cuda:  10%|▉         | 14/146 [00:00<00:05, 26.24batch/s]

Training batches on cuda:  12%|█▏        | 17/146 [00:00<00:05, 24.55batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:04, 26.76batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:04, 28.45batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:03, 29.62batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:03, 30.50batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:01<00:03, 31.40batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:01<00:03, 31.70batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:03, 32.33batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:02, 32.44batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:02, 32.60batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:02, 32.90batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:01<00:02, 33.11batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:02<00:02, 32.94batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:02<00:02, 33.23batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 32.78batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:02<00:02, 32.93batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:02<00:01, 33.13batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:02<00:01, 33.19batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 33.25batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:02<00:01, 33.42batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:03<00:01, 33.57batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:03<00:01, 33.44batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 33.39batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:03<00:01, 33.47batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:03<00:00, 33.59batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:03<00:00, 33.65batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:03<00:00, 33.58batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:03<00:00, 33.54batch/s]

Training batches on cuda:  88%|████████▊ | 129/146 [00:03<00:00, 33.65batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:04<00:00, 33.46batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 33.31batch/s]

Training batches on cuda:  97%|█████████▋| 141/146 [00:04<00:00, 33.41batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 33.44batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000617, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:42,  4.77s/epoch, loss=0.000617, prev_loss=nan]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/146 [00:00<00:15,  9.43batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:12, 11.62batch/s]

Training batches on cuda:   5%|▍         | 7/146 [00:00<00:09, 14.23batch/s]

Training batches on cuda:   8%|▊         | 11/146 [00:00<00:07, 17.02batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:06, 19.66batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:05, 22.00batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:05, 21.97batch/s]

Training batches on cuda:  18%|█▊        | 26/146 [00:00<00:04, 24.04batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:01<00:04, 25.77batch/s]

Training batches on cuda:  23%|██▎       | 34/146 [00:01<00:04, 27.17batch/s]

Training batches on cuda:  26%|██▌       | 38/146 [00:01<00:03, 28.29batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:01<00:03, 29.03batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:03, 29.27batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:03, 29.64batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:03, 29.66batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:01<00:03, 29.59batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:02<00:02, 30.12batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:02<00:02, 30.44batch/s]

Training batches on cuda:  47%|████▋     | 68/146 [00:02<00:02, 30.03batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:02<00:02, 30.16batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:02<00:02, 29.65batch/s]

Training batches on cuda:  54%|█████▍    | 79/146 [00:02<00:02, 29.36batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:02<00:02, 29.78batch/s]

Training batches on cuda:  60%|█████▉    | 87/146 [00:02<00:01, 30.27batch/s]

Training batches on cuda:  62%|██████▏   | 91/146 [00:03<00:01, 30.48batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:03<00:01, 30.75batch/s]

Training batches on cuda:  68%|██████▊   | 99/146 [00:03<00:01, 30.92batch/s]

Training batches on cuda:  71%|███████   | 103/146 [00:03<00:01, 30.94batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:03<00:01, 31.09batch/s]

Training batches on cuda:  76%|███████▌  | 111/146 [00:03<00:01, 30.86batch/s]

Training batches on cuda:  79%|███████▉  | 115/146 [00:03<00:01, 30.89batch/s]

Training batches on cuda:  82%|████████▏ | 119/146 [00:04<00:00, 30.69batch/s]

Training batches on cuda:  84%|████████▍ | 123/146 [00:04<00:00, 30.52batch/s]

Training batches on cuda:  87%|████████▋ | 127/146 [00:04<00:00, 30.21batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:04<00:00, 30.33batch/s]

Training batches on cuda:  92%|█████████▏| 135/146 [00:04<00:00, 29.86batch/s]

Training batches on cuda:  95%|█████████▌| 139/146 [00:04<00:00, 29.90batch/s]

Training batches on cuda:  98%|█████████▊| 143/146 [00:04<00:00, 29.96batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:09<00:42,  4.77s/epoch, loss=0.000611, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:39,  4.88s/epoch, loss=0.000611, prev_loss=nan]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:07, 18.63batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:06, 20.95batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:05, 23.24batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:05, 25.26batch/s]

Training batches on cuda:  12%|█▏        | 17/146 [00:00<00:04, 26.72batch/s]

Training batches on cuda:  14%|█▎        | 20/146 [00:00<00:05, 25.20batch/s]

Training batches on cuda:  16%|█▋        | 24/146 [00:00<00:04, 26.70batch/s]

Training batches on cuda:  18%|█▊        | 27/146 [00:00<00:04, 26.71batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:01<00:04, 27.58batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:04, 28.23batch/s]

Training batches on cuda:  25%|██▍       | 36/146 [00:01<00:03, 28.57batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:01<00:03, 29.24batch/s]

Training batches on cuda:  29%|██▉       | 43/146 [00:01<00:03, 29.39batch/s]

Training batches on cuda:  32%|███▏      | 46/146 [00:01<00:03, 29.17batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:03, 28.86batch/s]

Training batches on cuda:  36%|███▌      | 52/146 [00:01<00:03, 29.13batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:01<00:03, 29.67batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:02<00:02, 29.94batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:02<00:02, 30.35batch/s]

Training batches on cuda:  47%|████▋     | 68/146 [00:02<00:02, 30.66batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:02<00:02, 30.84batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:02<00:02, 30.59batch/s]

Training batches on cuda:  55%|█████▍    | 80/146 [00:02<00:02, 30.84batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:02<00:02, 30.99batch/s]

Training batches on cuda:  60%|██████    | 88/146 [00:02<00:01, 31.20batch/s]

Training batches on cuda:  63%|██████▎   | 92/146 [00:03<00:01, 30.94batch/s]

Training batches on cuda:  66%|██████▌   | 96/146 [00:03<00:01, 30.89batch/s]

Training batches on cuda:  68%|██████▊   | 100/146 [00:03<00:01, 31.05batch/s]

Training batches on cuda:  71%|███████   | 104/146 [00:03<00:01, 31.09batch/s]

Training batches on cuda:  74%|███████▍  | 108/146 [00:03<00:01, 31.15batch/s]

Training batches on cuda:  77%|███████▋  | 112/146 [00:03<00:01, 31.25batch/s]

Training batches on cuda:  79%|███████▉  | 116/146 [00:03<00:00, 31.23batch/s]

Training batches on cuda:  82%|████████▏ | 120/146 [00:04<00:00, 31.14batch/s]

Training batches on cuda:  85%|████████▍ | 124/146 [00:04<00:00, 31.12batch/s]

Training batches on cuda:  88%|████████▊ | 128/146 [00:04<00:00, 30.83batch/s]

Training batches on cuda:  90%|█████████ | 132/146 [00:04<00:00, 30.22batch/s]

Training batches on cuda:  93%|█████████▎| 136/146 [00:04<00:00, 30.21batch/s]

Training batches on cuda:  96%|█████████▌| 140/146 [00:04<00:00, 30.50batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:04<00:00, 30.57batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:15<00:39,  4.88s/epoch, loss=0.000576, prev_loss=0.000611]
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:34,  4.96s/epoch, loss=0.000576, prev_loss=0.000611]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/146 [00:00<00:18,  7.98batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:13, 10.23batch/s]

Training batches on cuda:   5%|▌         | 8/146 [00:00<00:10, 12.76batch/s]

Training batches on cuda:   8%|▊         | 11/146 [00:00<00:08, 15.22batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:08, 16.20batch/s]

Training batches on cuda:  12%|█▏        | 17/146 [00:00<00:06, 18.81batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:05, 21.28batch/s]

Training batches on cuda:  16%|█▋        | 24/146 [00:00<00:05, 22.53batch/s]

Training batches on cuda:  18%|█▊        | 27/146 [00:01<00:04, 23.83batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:01<00:04, 24.92batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:04, 26.05batch/s]

Training batches on cuda:  25%|██▍       | 36/146 [00:01<00:04, 26.81batch/s]

Training batches on cuda:  27%|██▋       | 39/146 [00:01<00:03, 27.10batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:01<00:03, 27.62batch/s]

Training batches on cuda:  32%|███▏      | 46/146 [00:01<00:03, 28.26batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:03, 26.97batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:03, 27.74batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:02<00:03, 27.77batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:02<00:03, 28.65batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:02<00:02, 29.25batch/s]

Training batches on cuda:  46%|████▌     | 67/146 [00:02<00:02, 28.89batch/s]

Training batches on cuda:  48%|████▊     | 70/146 [00:02<00:02, 28.92batch/s]

Training batches on cuda:  51%|█████     | 74/146 [00:02<00:02, 30.12batch/s]

Training batches on cuda:  53%|█████▎    | 78/146 [00:02<00:02, 31.04batch/s]

Training batches on cuda:  56%|█████▌    | 82/146 [00:02<00:02, 31.73batch/s]

Training batches on cuda:  59%|█████▉    | 86/146 [00:03<00:01, 32.04batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:03<00:01, 32.13batch/s]

Training batches on cuda:  64%|██████▍   | 94/146 [00:03<00:01, 32.27batch/s]

Training batches on cuda:  67%|██████▋   | 98/146 [00:03<00:01, 32.62batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:03<00:01, 32.64batch/s]

Training batches on cuda:  73%|███████▎  | 106/146 [00:03<00:01, 32.69batch/s]

Training batches on cuda:  75%|███████▌  | 110/146 [00:03<00:01, 32.82batch/s]

Training batches on cuda:  78%|███████▊  | 114/146 [00:03<00:00, 33.04batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:04<00:00, 33.00batch/s]

Training batches on cuda:  84%|████████▎ | 122/146 [00:04<00:00, 32.84batch/s]

Training batches on cuda:  86%|████████▋ | 126/146 [00:04<00:00, 33.04batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:04<00:00, 32.98batch/s]

Training batches on cuda:  92%|█████████▏| 134/146 [00:04<00:00, 32.79batch/s]

Training batches on cuda:  95%|█████████▍| 138/146 [00:04<00:00, 32.61batch/s]

Training batches on cuda:  97%|█████████▋| 142/146 [00:04<00:00, 32.63batch/s]

Training batches on cuda: 100%|██████████| 146/146 [00:04<00:00, 33.04batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:20<00:34,  4.96s/epoch, loss=0.000477, prev_loss=0.000576]
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:30,  5.01s/epoch, loss=0.000477, prev_loss=0.000576]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/146 [00:00<00:06, 23.70batch/s]

Training batches on cuda:   5%|▍         | 7/146 [00:00<00:05, 25.72batch/s]

Training batches on cuda:   8%|▊         | 11/146 [00:00<00:04, 27.54batch/s]

Training batches on cuda:  10%|▉         | 14/146 [00:00<00:04, 27.77batch/s]

Training batches on cuda:  12%|█▏        | 18/146 [00:00<00:04, 28.80batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:04, 29.62batch/s]

Training batches on cuda:  18%|█▊        | 26/146 [00:00<00:03, 30.24batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:03, 30.02batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:03, 30.78batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:01<00:03, 31.30batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:01<00:03, 31.31batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:03, 30.72batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:03, 30.96batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:03, 30.83batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:02, 31.35batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:01<00:02, 31.10batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:02<00:02, 31.03batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:02<00:02, 30.45batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 30.04batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:02<00:02, 30.00batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:02<00:02, 30.42batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:02<00:02, 30.30batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 30.38batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:03<00:01, 29.85batch/s]

Training batches on cuda:  66%|██████▌   | 96/146 [00:03<00:01, 29.63batch/s]

Training batches on cuda:  68%|██████▊   | 99/146 [00:03<00:01, 29.01batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:03<00:01, 28.81batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 28.91batch/s]

Training batches on cuda:  74%|███████▍  | 108/146 [00:03<00:01, 29.08batch/s]

Training batches on cuda:  76%|███████▌  | 111/146 [00:03<00:01, 29.02batch/s]

Training batches on cuda:  79%|███████▉  | 115/146 [00:03<00:01, 29.52batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:03<00:00, 29.52batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:04<00:00, 28.64batch/s]

Training batches on cuda:  85%|████████▍ | 124/146 [00:04<00:00, 28.96batch/s]

Training batches on cuda:  87%|████████▋ | 127/146 [00:04<00:00, 28.92batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:04<00:00, 28.50batch/s]

Training batches on cuda:  92%|█████████▏| 134/146 [00:04<00:00, 29.19batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 28.69batch/s]

Training batches on cuda:  97%|█████████▋| 141/146 [00:04<00:00, 29.27batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 29.62batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:25<00:30,  5.01s/epoch, loss=0.000383, prev_loss=0.000477]
Training epochs on cuda:  50%|█████     | 5/10 [00:25<00:25,  5.04s/epoch, loss=0.000383, prev_loss=0.000477]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:08, 16.61batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:07, 19.11batch/s]

Training batches on cuda:   5%|▌         | 8/146 [00:00<00:06, 20.58batch/s]

Training batches on cuda:   8%|▊         | 12/146 [00:00<00:05, 22.62batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:05, 24.16batch/s]

Training batches on cuda:  12%|█▏        | 18/146 [00:00<00:05, 24.52batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:04, 25.77batch/s]

Training batches on cuda:  18%|█▊        | 26/146 [00:00<00:04, 27.07batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:01<00:04, 28.32batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:04, 28.13batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:01<00:03, 28.61batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:01<00:03, 28.14batch/s]

Training batches on cuda:  30%|███       | 44/146 [00:01<00:03, 29.08batch/s]

Training batches on cuda:  32%|███▏      | 47/146 [00:01<00:03, 29.34batch/s]

Training batches on cuda:  34%|███▍      | 50/146 [00:01<00:03, 28.44batch/s]

Training batches on cuda:  37%|███▋      | 54/146 [00:01<00:03, 29.37batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:02<00:03, 29.13batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:02<00:02, 29.35batch/s]

Training batches on cuda:  43%|████▎     | 63/146 [00:02<00:02, 28.91batch/s]

Training batches on cuda:  45%|████▌     | 66/146 [00:02<00:02, 28.78batch/s]

Training batches on cuda:  48%|████▊     | 70/146 [00:02<00:02, 28.90batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 27.59batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:02<00:02, 27.27batch/s]

Training batches on cuda:  54%|█████▍    | 79/146 [00:02<00:02, 28.03batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:02<00:02, 28.12batch/s]

Training batches on cuda:  59%|█████▉    | 86/146 [00:03<00:02, 26.62batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:03<00:02, 27.15batch/s]

Training batches on cuda:  63%|██████▎   | 92/146 [00:03<00:01, 27.54batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:03<00:01, 27.93batch/s]

Training batches on cuda:  67%|██████▋   | 98/146 [00:03<00:01, 28.41batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:03<00:01, 28.55batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 29.30batch/s]

Training batches on cuda:  74%|███████▍  | 108/146 [00:03<00:01, 29.08batch/s]

Training batches on cuda:  76%|███████▌  | 111/146 [00:03<00:01, 28.42batch/s]

Training batches on cuda:  79%|███████▉  | 115/146 [00:04<00:01, 28.89batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:04<00:01, 27.24batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:04<00:00, 26.73batch/s]

Training batches on cuda:  85%|████████▍ | 124/146 [00:04<00:00, 26.00batch/s]

Training batches on cuda:  87%|████████▋ | 127/146 [00:04<00:00, 26.84batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:04<00:00, 26.49batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:04<00:00, 26.97batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 28.13batch/s]

Training batches on cuda:  96%|█████████▌| 140/146 [00:04<00:00, 28.25batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:05<00:00, 29.15batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:25,  5.04s/epoch, loss=0.000312, prev_loss=0.000383]
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:20,  5.16s/epoch, loss=0.000312, prev_loss=0.000383]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:09, 15.47batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:07, 17.73batch/s]

Training batches on cuda:   5%|▌         | 8/146 [00:00<00:06, 19.99batch/s]

Training batches on cuda:   8%|▊         | 12/146 [00:00<00:05, 22.61batch/s]

Training batches on cuda:  11%|█         | 16/146 [00:00<00:05, 24.82batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:04, 26.08batch/s]

Training batches on cuda:  16%|█▌        | 23/146 [00:00<00:04, 27.31batch/s]

Training batches on cuda:  18%|█▊        | 27/146 [00:00<00:04, 28.11batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:01<00:04, 28.55batch/s]

Training batches on cuda:  23%|██▎       | 34/146 [00:01<00:03, 29.21batch/s]

Training batches on cuda:  26%|██▌       | 38/146 [00:01<00:03, 29.44batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:01<00:03, 29.55batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:03, 30.15batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:03, 30.60batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:03, 30.78batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:02, 30.90batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:02<00:02, 30.94batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:02<00:02, 30.82batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:02<00:02, 30.31batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 29.51batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:02<00:02, 30.12batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:02<00:02, 30.41batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:02<00:01, 30.61batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 30.88batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:03<00:01, 30.86batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:03<00:01, 30.95batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:03<00:01, 31.06batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 30.21batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:03<00:01, 30.63batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:03<00:01, 30.97batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:03<00:00, 30.90batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:04<00:00, 31.03batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:04<00:00, 29.98batch/s]

Training batches on cuda:  88%|████████▊ | 129/146 [00:04<00:00, 30.50batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:04<00:00, 30.75batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 31.07batch/s]

Training batches on cuda:  97%|█████████▋| 141/146 [00:04<00:00, 31.19batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 31.06batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:35<00:20,  5.16s/epoch, loss=0.000272, prev_loss=0.000312]
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:15,  5.16s/epoch, loss=0.000272, prev_loss=0.000312]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:08, 17.55batch/s]

Training batches on cuda:   4%|▍         | 6/146 [00:00<00:06, 20.14batch/s]

Training batches on cuda:   7%|▋         | 10/146 [00:00<00:05, 22.87batch/s]

Training batches on cuda:  10%|▉         | 14/146 [00:00<00:05, 24.99batch/s]

Training batches on cuda:  12%|█▏        | 18/146 [00:00<00:04, 26.42batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:04, 27.83batch/s]

Training batches on cuda:  18%|█▊        | 26/146 [00:00<00:04, 28.78batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:01<00:03, 29.61batch/s]

Training batches on cuda:  23%|██▎       | 34/146 [00:01<00:03, 29.92batch/s]

Training batches on cuda:  26%|██▌       | 38/146 [00:01<00:03, 30.22batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:01<00:03, 29.71batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:03, 28.96batch/s]

Training batches on cuda:  33%|███▎      | 48/146 [00:01<00:03, 28.69batch/s]

Training batches on cuda:  35%|███▍      | 51/146 [00:01<00:03, 28.94batch/s]

Training batches on cuda:  37%|███▋      | 54/146 [00:01<00:03, 29.12batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:03, 29.22batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:02<00:02, 30.10batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:02<00:02, 30.95batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:02<00:02, 31.63batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 32.17batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:02<00:02, 32.71batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:02<00:01, 32.66batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:02<00:01, 32.47batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 32.18batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:03<00:01, 32.02batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:03<00:01, 31.98batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:03<00:01, 31.87batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 32.02batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:03<00:01, 32.01batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:03<00:01, 31.91batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:03<00:00, 31.92batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:03<00:00, 31.82batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:04<00:00, 32.06batch/s]

Training batches on cuda:  88%|████████▊ | 129/146 [00:04<00:00, 31.91batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:04<00:00, 31.78batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 31.75batch/s]

Training batches on cuda:  97%|█████████▋| 141/146 [00:04<00:00, 31.80batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 32.26batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:40<00:15,  5.16s/epoch, loss=0.000254, prev_loss=0.000272]
Training epochs on cuda:  80%|████████  | 8/10 [00:40<00:10,  5.08s/epoch, loss=0.000254, prev_loss=0.000272]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:08, 17.42batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:07, 19.87batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:06, 22.46batch/s]

Training batches on cuda:   8%|▊         | 12/146 [00:00<00:05, 24.05batch/s]

Training batches on cuda:  11%|█         | 16/146 [00:00<00:05, 25.86batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:04, 26.84batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:04, 27.10batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:04, 27.85batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:01<00:04, 28.83batch/s]

Training batches on cuda:  22%|██▏       | 32/146 [00:01<00:03, 29.13batch/s]

Training batches on cuda:  25%|██▍       | 36/146 [00:01<00:03, 29.95batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:01<00:03, 30.09batch/s]

Training batches on cuda:  30%|███       | 44/146 [00:01<00:03, 30.52batch/s]

Training batches on cuda:  33%|███▎      | 48/146 [00:01<00:03, 30.76batch/s]

Training batches on cuda:  36%|███▌      | 52/146 [00:01<00:03, 31.00batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:01<00:02, 31.12batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:02<00:02, 31.08batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:02<00:02, 31.06batch/s]

Training batches on cuda:  47%|████▋     | 68/146 [00:02<00:02, 30.49batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:02<00:02, 29.61batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:02<00:02, 29.71batch/s]

Training batches on cuda:  55%|█████▍    | 80/146 [00:02<00:02, 29.55batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:02<00:02, 29.73batch/s]

Training batches on cuda:  60%|█████▉    | 87/146 [00:02<00:02, 29.48batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:03<00:01, 29.49batch/s]

Training batches on cuda:  64%|██████▍   | 94/146 [00:03<00:01, 29.93batch/s]

Training batches on cuda:  67%|██████▋   | 98/146 [00:03<00:01, 30.05batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:03<00:01, 30.38batch/s]

Training batches on cuda:  73%|███████▎  | 106/146 [00:03<00:01, 30.41batch/s]

Training batches on cuda:  75%|███████▌  | 110/146 [00:03<00:01, 29.01batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:03<00:01, 29.00batch/s]

Training batches on cuda:  79%|███████▉  | 116/146 [00:03<00:01, 28.58batch/s]

Training batches on cuda:  82%|████████▏ | 119/146 [00:04<00:00, 28.93batch/s]

Training batches on cuda:  84%|████████▍ | 123/146 [00:04<00:00, 29.37batch/s]

Training batches on cuda:  87%|████████▋ | 127/146 [00:04<00:00, 29.72batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:04<00:00, 29.93batch/s]

Training batches on cuda:  92%|█████████▏| 134/146 [00:04<00:00, 29.36batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 28.55batch/s]

Training batches on cuda:  96%|█████████▌| 140/146 [00:04<00:00, 28.53batch/s]

Training batches on cuda:  98%|█████████▊| 143/146 [00:04<00:00, 28.23batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:46<00:10,  5.08s/epoch, loss=0.000244, prev_loss=0.000254]
Training epochs on cuda:  90%|█████████ | 9/10 [00:46<00:05,  5.14s/epoch, loss=0.000244, prev_loss=0.000254]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:07, 18.79batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:06, 21.04batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:05, 23.28batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:05, 25.03batch/s]

Training batches on cuda:  11%|█         | 16/146 [00:00<00:04, 26.05batch/s]

Training batches on cuda:  14%|█▎        | 20/146 [00:00<00:04, 27.34batch/s]

Training batches on cuda:  16%|█▋        | 24/146 [00:00<00:04, 28.43batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:04, 29.03batch/s]

Training batches on cuda:  22%|██▏       | 32/146 [00:01<00:03, 29.59batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:01<00:03, 28.79batch/s]

Training batches on cuda:  27%|██▋       | 39/146 [00:01<00:03, 29.75batch/s]

Training batches on cuda:  29%|██▉       | 43/146 [00:01<00:03, 30.11batch/s]

Training batches on cuda:  32%|███▏      | 47/146 [00:01<00:03, 30.38batch/s]

Training batches on cuda:  35%|███▍      | 51/146 [00:01<00:03, 29.60batch/s]

Training batches on cuda:  38%|███▊      | 55/146 [00:01<00:03, 30.02batch/s]

Training batches on cuda:  40%|████      | 59/146 [00:01<00:02, 30.30batch/s]

Training batches on cuda:  43%|████▎     | 63/146 [00:02<00:02, 30.47batch/s]

Training batches on cuda:  46%|████▌     | 67/146 [00:02<00:02, 30.67batch/s]

Training batches on cuda:  49%|████▊     | 71/146 [00:02<00:02, 30.82batch/s]

Training batches on cuda:  51%|█████▏    | 75/146 [00:02<00:02, 30.63batch/s]

Training batches on cuda:  54%|█████▍    | 79/146 [00:02<00:02, 30.71batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:02<00:02, 29.66batch/s]

Training batches on cuda:  59%|█████▉    | 86/146 [00:02<00:02, 29.71batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 29.75batch/s]

Training batches on cuda:  63%|██████▎   | 92/146 [00:03<00:01, 29.80batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:03<00:01, 29.78batch/s]

Training batches on cuda:  67%|██████▋   | 98/146 [00:03<00:01, 29.62batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:03<00:01, 29.52batch/s]

Training batches on cuda:  71%|███████   | 104/146 [00:03<00:01, 28.53batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:03<00:01, 28.59batch/s]

Training batches on cuda:  76%|███████▌  | 111/146 [00:03<00:01, 29.14batch/s]

Training batches on cuda:  79%|███████▉  | 115/146 [00:03<00:01, 29.47batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:03<00:00, 29.62batch/s]

Training batches on cuda:  84%|████████▎ | 122/146 [00:04<00:00, 30.04batch/s]

Training batches on cuda:  86%|████████▋ | 126/146 [00:04<00:00, 29.75batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:04<00:00, 30.11batch/s]

Training batches on cuda:  92%|█████████▏| 134/146 [00:04<00:00, 30.15batch/s]

Training batches on cuda:  95%|█████████▍| 138/146 [00:04<00:00, 30.14batch/s]

Training batches on cuda:  97%|█████████▋| 142/146 [00:04<00:00, 29.66batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 28.55batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:51<00:05,  5.14s/epoch, loss=0.000238, prev_loss=0.000244]2021-09-16:18:56:52,450 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:56:53,678 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:57:08,748 INFO     [evaluator.py:657] Evaluation took 15.07s seconds
2021-09-16:18:57:08,897 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 10.08s/epoch, loss=0.000238, prev_loss=0.000244]
Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00,  6.77s/epoch, loss=0.000238, prev_loss=0.000244]
2021-09-16:18:57:08,898 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmplffo9t_1'
2021-09-16:18:57:08,965 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmplffo9t_1' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.41ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   4%|▍         | 1.54k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   5%|▌         | 2.05k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.56k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   8%|▊         | 3.07k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▉         | 3.58k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.60ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.60ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.60ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.57ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.59ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.60ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:18:57:23,881 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
[I 2021-09-16 18:57:23,886] Trial 65 finished with value: 0.9618004330916888 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.011414150442787738, 'optimizer.lr': 0.043645984782633464, 'negative_sampler.num_negs_per_pos': 31, 'training.batch_size': 2132}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:57:23,912 WARNING  [api.py:823] No random seed is specified. Setting to 1624461701.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/217 [00:00<00:07, 26.79batch/s]

Training batches on cuda:   4%|▍         | 9/217 [00:00<00:06, 31.18batch/s]

Training batches on cuda:   7%|▋         | 15/217 [00:00<00:05, 35.46batch/s]

Training batches on cuda:  10%|▉         | 21/217 [00:00<00:04, 39.59batch/s]

Training batches on cuda:  12%|█▏        | 27/217 [00:00<00:04, 43.09batch/s]

Training batches on cuda:  15%|█▌        | 33/217 [00:00<00:03, 46.24batch/s]

Training batches on cuda:  18%|█▊        | 39/217 [00:00<00:03, 48.45batch/s]

Training batches on cuda:  21%|██        | 45/217 [00:00<00:03, 50.38batch/s]

Training batches on cuda:  24%|██▎       | 51/217 [00:01<00:03, 51.60batch/s]

Training batches on cuda:  26%|██▋       | 57/217 [00:01<00:03, 52.45batch/s]

Training batches on cuda:  29%|██▉       | 63/217 [00:01<00:02, 53.39batch/s]

Training batches on cuda:  32%|███▏      | 69/217 [00:01<00:02, 53.02batch/s]

Training batches on cuda:  35%|███▍      | 75/217 [00:01<00:02, 51.11batch/s]

Training batches on cuda:  37%|███▋      | 81/217 [00:01<00:02, 52.92batch/s]

Training batches on cuda:  40%|████      | 87/217 [00:01<00:02, 53.74batch/s]

Training batches on cuda:  43%|████▎     | 93/217 [00:01<00:02, 54.74batch/s]

Training batches on cuda:  46%|████▌     | 99/217 [00:01<00:02, 55.54batch/s]

Training batches on cuda:  48%|████▊     | 105/217 [00:01<00:01, 56.79batch/s]

Training batches on cuda:  51%|█████     | 111/217 [00:02<00:01, 57.60batch/s]

Training batches on cuda:  54%|█████▍    | 117/217 [00:02<00:01, 56.94batch/s]

Training batches on cuda:  57%|█████▋    | 123/217 [00:02<00:01, 56.01batch/s]

Training batches on cuda:  59%|█████▉    | 129/217 [00:02<00:01, 55.03batch/s]

Training batches on cuda:  62%|██████▏   | 135/217 [00:02<00:01, 55.53batch/s]

Training batches on cuda:  65%|██████▍   | 141/217 [00:02<00:01, 55.40batch/s]

Training batches on cuda:  68%|██████▊   | 147/217 [00:02<00:01, 55.18batch/s]

Training batches on cuda:  71%|███████   | 153/217 [00:02<00:01, 55.74batch/s]

Training batches on cuda:  73%|███████▎  | 159/217 [00:02<00:01, 55.25batch/s]

Training batches on cuda:  76%|███████▌  | 165/217 [00:03<00:00, 52.81batch/s]

Training batches on cuda:  79%|███████▉  | 171/217 [00:03<00:00, 51.76batch/s]

Training batches on cuda:  82%|████████▏ | 177/217 [00:03<00:00, 52.30batch/s]

Training batches on cuda:  84%|████████▍ | 183/217 [00:03<00:00, 53.72batch/s]

Training batches on cuda:  87%|████████▋ | 189/217 [00:03<00:00, 54.59batch/s]

Training batches on cuda:  90%|████████▉ | 195/217 [00:03<00:00, 54.14batch/s]

Training batches on cuda:  93%|█████████▎| 201/217 [00:03<00:00, 54.74batch/s]

Training batches on cuda:  95%|█████████▌| 207/217 [00:03<00:00, 53.53batch/s]

Training batches on cuda:  98%|█████████▊| 213/217 [00:03<00:00, 52.80batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000923, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:38,  4.33s/epoch, loss=0.000923, prev_loss=nan]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/217 [00:00<00:07, 28.48batch/s]

Training batches on cuda:   4%|▍         | 9/217 [00:00<00:06, 33.46batch/s]

Training batches on cuda:   7%|▋         | 16/217 [00:00<00:05, 38.64batch/s]

Training batches on cuda:  10%|█         | 22/217 [00:00<00:04, 42.48batch/s]

Training batches on cuda:  13%|█▎        | 28/217 [00:00<00:04, 45.38batch/s]

Training batches on cuda:  16%|█▌        | 34/217 [00:00<00:03, 48.11batch/s]

Training batches on cuda:  18%|█▊        | 40/217 [00:00<00:03, 50.42batch/s]

Training batches on cuda:  21%|██        | 46/217 [00:00<00:03, 50.80batch/s]

Training batches on cuda:  24%|██▍       | 52/217 [00:00<00:03, 50.42batch/s]

Training batches on cuda:  27%|██▋       | 58/217 [00:01<00:03, 50.53batch/s]

Training batches on cuda:  29%|██▉       | 64/217 [00:01<00:03, 50.41batch/s]

Training batches on cuda:  32%|███▏      | 70/217 [00:01<00:02, 50.77batch/s]

Training batches on cuda:  35%|███▌      | 76/217 [00:01<00:02, 50.78batch/s]

Training batches on cuda:  38%|███▊      | 82/217 [00:01<00:02, 49.81batch/s]

Training batches on cuda:  41%|████      | 88/217 [00:01<00:02, 49.94batch/s]

Training batches on cuda:  43%|████▎     | 94/217 [00:01<00:02, 49.90batch/s]

Training batches on cuda:  46%|████▌     | 100/217 [00:01<00:02, 50.10batch/s]

Training batches on cuda:  49%|████▉     | 106/217 [00:02<00:02, 50.19batch/s]

Training batches on cuda:  52%|█████▏    | 112/217 [00:02<00:02, 50.97batch/s]

Training batches on cuda:  54%|█████▍    | 118/217 [00:02<00:01, 51.72batch/s]

Training batches on cuda:  57%|█████▋    | 124/217 [00:02<00:01, 52.45batch/s]

Training batches on cuda:  60%|█████▉    | 130/217 [00:02<00:01, 52.73batch/s]

Training batches on cuda:  63%|██████▎   | 137/217 [00:02<00:01, 54.96batch/s]

Training batches on cuda:  66%|██████▌   | 143/217 [00:02<00:01, 56.12batch/s]

Training batches on cuda:  69%|██████▊   | 149/217 [00:02<00:01, 56.50batch/s]

Training batches on cuda:  72%|███████▏  | 156/217 [00:02<00:01, 57.66batch/s]

Training batches on cuda:  75%|███████▌  | 163/217 [00:03<00:00, 58.54batch/s]

Training batches on cuda:  78%|███████▊  | 169/217 [00:03<00:00, 57.60batch/s]

Training batches on cuda:  81%|████████  | 175/217 [00:03<00:00, 54.46batch/s]

Training batches on cuda:  83%|████████▎ | 181/217 [00:03<00:00, 54.05batch/s]

Training batches on cuda:  86%|████████▌ | 187/217 [00:03<00:00, 54.87batch/s]

Training batches on cuda:  89%|████████▉ | 193/217 [00:03<00:00, 53.90batch/s]

Training batches on cuda:  92%|█████████▏| 199/217 [00:03<00:00, 51.94batch/s]

Training batches on cuda:  94%|█████████▍| 205/217 [00:03<00:00, 53.87batch/s]

Training batches on cuda:  97%|█████████▋| 211/217 [00:03<00:00, 53.96batch/s]

Training batches on cuda: 100%|██████████| 217/217 [00:04<00:00, 54.98batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:38,  4.33s/epoch, loss=0.000919, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:34,  4.34s/epoch, loss=0.000919, prev_loss=nan]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/217 [00:00<00:08, 26.71batch/s]

Training batches on cuda:   4%|▎         | 8/217 [00:00<00:06, 30.61batch/s]

Training batches on cuda:   6%|▋         | 14/217 [00:00<00:05, 34.64batch/s]

Training batches on cuda:   9%|▉         | 19/217 [00:00<00:05, 38.03batch/s]

Training batches on cuda:  11%|█         | 24/217 [00:00<00:04, 40.74batch/s]

Training batches on cuda:  14%|█▍        | 30/217 [00:00<00:04, 43.27batch/s]

Training batches on cuda:  16%|█▌        | 35/217 [00:00<00:04, 45.04batch/s]

Training batches on cuda:  18%|█▊        | 40/217 [00:00<00:03, 46.21batch/s]

Training batches on cuda:  21%|██        | 46/217 [00:00<00:03, 48.05batch/s]

Training batches on cuda:  24%|██▍       | 52/217 [00:01<00:03, 48.71batch/s]

Training batches on cuda:  27%|██▋       | 58/217 [00:01<00:03, 49.24batch/s]

Training batches on cuda:  29%|██▉       | 63/217 [00:01<00:03, 49.08batch/s]

Training batches on cuda:  31%|███▏      | 68/217 [00:01<00:03, 49.22batch/s]

Training batches on cuda:  34%|███▍      | 74/217 [00:01<00:02, 50.84batch/s]

Training batches on cuda:  37%|███▋      | 80/217 [00:01<00:02, 51.71batch/s]

Training batches on cuda:  40%|███▉      | 86/217 [00:01<00:02, 50.51batch/s]

Training batches on cuda:  42%|████▏     | 92/217 [00:01<00:02, 50.74batch/s]

Training batches on cuda:  45%|████▌     | 98/217 [00:01<00:02, 51.58batch/s]

Training batches on cuda:  48%|████▊     | 104/217 [00:02<00:02, 52.43batch/s]

Training batches on cuda:  51%|█████     | 110/217 [00:02<00:02, 52.64batch/s]

Training batches on cuda:  53%|█████▎    | 116/217 [00:02<00:01, 53.39batch/s]

Training batches on cuda:  56%|█████▌    | 122/217 [00:02<00:01, 53.58batch/s]

Training batches on cuda:  59%|█████▉    | 128/217 [00:02<00:01, 54.36batch/s]

Training batches on cuda:  62%|██████▏   | 134/217 [00:02<00:01, 54.69batch/s]

Training batches on cuda:  65%|██████▍   | 140/217 [00:02<00:01, 54.78batch/s]

Training batches on cuda:  67%|██████▋   | 146/217 [00:02<00:01, 54.90batch/s]

Training batches on cuda:  70%|███████   | 152/217 [00:02<00:01, 55.56batch/s]

Training batches on cuda:  73%|███████▎  | 158/217 [00:03<00:01, 54.84batch/s]

Training batches on cuda:  76%|███████▌  | 164/217 [00:03<00:00, 54.63batch/s]

Training batches on cuda:  78%|███████▊  | 170/217 [00:03<00:00, 54.77batch/s]

Training batches on cuda:  81%|████████  | 176/217 [00:03<00:00, 54.79batch/s]

Training batches on cuda:  84%|████████▍ | 182/217 [00:03<00:00, 54.94batch/s]

Training batches on cuda:  87%|████████▋ | 188/217 [00:03<00:00, 54.06batch/s]

Training batches on cuda:  89%|████████▉ | 194/217 [00:03<00:00, 54.09batch/s]

Training batches on cuda:  92%|█████████▏| 200/217 [00:03<00:00, 53.95batch/s]

Training batches on cuda:  95%|█████████▍| 206/217 [00:03<00:00, 54.53batch/s]

Training batches on cuda:  98%|█████████▊| 212/217 [00:04<00:00, 55.31batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:34,  4.34s/epoch, loss=0.000919, prev_loss=0.000919]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:30,  4.36s/epoch, loss=0.000919, prev_loss=0.000919]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/217 [00:00<00:05, 36.98batch/s]

Training batches on cuda:   5%|▍         | 10/217 [00:00<00:05, 41.25batch/s]

Training batches on cuda:   7%|▋         | 16/217 [00:00<00:04, 45.28batch/s]

Training batches on cuda:  10%|█         | 22/217 [00:00<00:04, 48.72batch/s]

Training batches on cuda:  13%|█▎        | 28/217 [00:00<00:03, 50.84batch/s]

Training batches on cuda:  16%|█▌        | 34/217 [00:00<00:03, 52.53batch/s]

Training batches on cuda:  18%|█▊        | 40/217 [00:00<00:03, 53.92batch/s]

Training batches on cuda:  21%|██        | 46/217 [00:00<00:03, 54.67batch/s]

Training batches on cuda:  24%|██▍       | 52/217 [00:00<00:02, 55.78batch/s]

Training batches on cuda:  27%|██▋       | 58/217 [00:01<00:02, 55.76batch/s]

Training batches on cuda:  29%|██▉       | 64/217 [00:01<00:02, 55.99batch/s]

Training batches on cuda:  32%|███▏      | 70/217 [00:01<00:02, 56.18batch/s]

Training batches on cuda:  35%|███▌      | 76/217 [00:01<00:02, 56.98batch/s]

Training batches on cuda:  38%|███▊      | 82/217 [00:01<00:02, 57.63batch/s]

Training batches on cuda:  41%|████      | 88/217 [00:01<00:02, 58.13batch/s]

Training batches on cuda:  43%|████▎     | 94/217 [00:01<00:02, 58.48batch/s]

Training batches on cuda:  46%|████▌     | 100/217 [00:01<00:02, 58.37batch/s]

Training batches on cuda:  49%|████▉     | 106/217 [00:01<00:01, 58.30batch/s]

Training batches on cuda:  52%|█████▏    | 112/217 [00:01<00:01, 58.38batch/s]

Training batches on cuda:  54%|█████▍    | 118/217 [00:02<00:01, 58.42batch/s]

Training batches on cuda:  57%|█████▋    | 124/217 [00:02<00:01, 58.51batch/s]

Training batches on cuda:  60%|█████▉    | 130/217 [00:02<00:01, 58.76batch/s]

Training batches on cuda:  63%|██████▎   | 136/217 [00:02<00:01, 58.83batch/s]

Training batches on cuda:  66%|██████▌   | 143/217 [00:02<00:01, 59.46batch/s]

Training batches on cuda:  69%|██████▊   | 149/217 [00:02<00:01, 59.47batch/s]

Training batches on cuda:  71%|███████▏  | 155/217 [00:02<00:01, 59.37batch/s]

Training batches on cuda:  75%|███████▍  | 162/217 [00:02<00:00, 59.83batch/s]

Training batches on cuda:  78%|███████▊  | 169/217 [00:02<00:00, 60.22batch/s]

Training batches on cuda:  81%|████████  | 176/217 [00:03<00:00, 61.09batch/s]

Training batches on cuda:  84%|████████▍ | 183/217 [00:03<00:00, 61.64batch/s]

Training batches on cuda:  88%|████████▊ | 190/217 [00:03<00:00, 61.75batch/s]

Training batches on cuda:  91%|█████████ | 197/217 [00:03<00:00, 61.40batch/s]

Training batches on cuda:  94%|█████████▍| 204/217 [00:03<00:00, 60.77batch/s]

Training batches on cuda:  97%|█████████▋| 211/217 [00:03<00:00, 60.56batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:30,  4.36s/epoch, loss=0.000919, prev_loss=0.000919]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:25,  4.24s/epoch, loss=0.000919, prev_loss=0.000919]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/217 [00:00<00:05, 38.22batch/s]

Training batches on cuda:   5%|▍         | 10/217 [00:00<00:04, 42.71batch/s]

Training batches on cuda:   7%|▋         | 16/217 [00:00<00:04, 46.46batch/s]

Training batches on cuda:  11%|█         | 23/217 [00:00<00:03, 50.11batch/s]

Training batches on cuda:  14%|█▍        | 30/217 [00:00<00:03, 53.08batch/s]

Training batches on cuda:  17%|█▋        | 37/217 [00:00<00:03, 55.60batch/s]

Training batches on cuda:  20%|██        | 44/217 [00:00<00:03, 56.83batch/s]

Training batches on cuda:  23%|██▎       | 50/217 [00:00<00:02, 57.45batch/s]

Training batches on cuda:  26%|██▌       | 56/217 [00:00<00:02, 57.99batch/s]

Training batches on cuda:  29%|██▊       | 62/217 [00:01<00:02, 52.20batch/s]

Training batches on cuda:  32%|███▏      | 69/217 [00:01<00:02, 54.53batch/s]

Training batches on cuda:  35%|███▍      | 75/217 [00:01<00:02, 55.76batch/s]

Training batches on cuda:  37%|███▋      | 81/217 [00:01<00:02, 55.90batch/s]

Training batches on cuda:  40%|████      | 87/217 [00:01<00:02, 54.87batch/s]

Training batches on cuda:  43%|████▎     | 93/217 [00:01<00:02, 54.42batch/s]

Training batches on cuda:  46%|████▌     | 99/217 [00:01<00:02, 53.40batch/s]

Training batches on cuda:  48%|████▊     | 105/217 [00:01<00:02, 52.73batch/s]

Training batches on cuda:  51%|█████     | 111/217 [00:02<00:02, 52.61batch/s]

Training batches on cuda:  54%|█████▍    | 118/217 [00:02<00:01, 54.36batch/s]

Training batches on cuda:  57%|█████▋    | 124/217 [00:02<00:01, 51.43batch/s]

Training batches on cuda:  60%|█████▉    | 130/217 [00:02<00:01, 52.01batch/s]

Training batches on cuda:  63%|██████▎   | 136/217 [00:02<00:01, 54.10batch/s]

Training batches on cuda:  65%|██████▌   | 142/217 [00:02<00:01, 53.85batch/s]

Training batches on cuda:  68%|██████▊   | 148/217 [00:02<00:01, 54.52batch/s]

Training batches on cuda:  71%|███████   | 154/217 [00:02<00:01, 54.76batch/s]

Training batches on cuda:  74%|███████▎  | 160/217 [00:02<00:01, 54.63batch/s]

Training batches on cuda:  76%|███████▋  | 166/217 [00:03<00:00, 53.60batch/s]

Training batches on cuda:  79%|███████▉  | 172/217 [00:03<00:00, 53.50batch/s]

Training batches on cuda:  82%|████████▏ | 178/217 [00:03<00:00, 52.70batch/s]

Training batches on cuda:  85%|████████▍ | 184/217 [00:03<00:00, 51.98batch/s]

Training batches on cuda:  88%|████████▊ | 190/217 [00:03<00:00, 51.35batch/s]

Training batches on cuda:  90%|█████████ | 196/217 [00:03<00:00, 52.22batch/s]

Training batches on cuda:  93%|█████████▎| 202/217 [00:03<00:00, 51.06batch/s]

Training batches on cuda:  96%|█████████▌| 208/217 [00:03<00:00, 51.62batch/s]

Training batches on cuda:  99%|█████████▊| 214/217 [00:03<00:00, 51.34batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:21<00:25,  4.24s/epoch, loss=0.000919, prev_loss=0.000919]
Training epochs on cuda:  50%|█████     | 5/10 [00:21<00:21,  4.25s/epoch, loss=0.000919, prev_loss=0.000919]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/217 [00:00<00:07, 29.16batch/s]

Training batches on cuda:   4%|▎         | 8/217 [00:00<00:06, 32.95batch/s]

Training batches on cuda:   6%|▋         | 14/217 [00:00<00:05, 36.86batch/s]

Training batches on cuda:   9%|▉         | 20/217 [00:00<00:04, 40.21batch/s]

Training batches on cuda:  12%|█▏        | 26/217 [00:00<00:04, 42.85batch/s]

Training batches on cuda:  14%|█▍        | 31/217 [00:00<00:04, 43.77batch/s]

Training batches on cuda:  17%|█▋        | 37/217 [00:00<00:03, 45.60batch/s]

Training batches on cuda:  19%|█▉        | 42/217 [00:00<00:03, 43.75batch/s]

Training batches on cuda:  22%|██▏       | 47/217 [00:01<00:03, 44.48batch/s]

Training batches on cuda:  24%|██▍       | 53/217 [00:01<00:03, 46.25batch/s]

Training batches on cuda:  27%|██▋       | 59/217 [00:01<00:03, 47.62batch/s]

Training batches on cuda:  30%|██▉       | 65/217 [00:01<00:03, 48.15batch/s]

Training batches on cuda:  33%|███▎      | 71/217 [00:01<00:02, 50.17batch/s]

Training batches on cuda:  35%|███▌      | 77/217 [00:01<00:02, 51.28batch/s]

Training batches on cuda:  38%|███▊      | 83/217 [00:01<00:02, 51.95batch/s]

Training batches on cuda:  41%|████      | 89/217 [00:01<00:02, 46.99batch/s]

Training batches on cuda:  44%|████▍     | 95/217 [00:01<00:02, 48.00batch/s]

Training batches on cuda:  47%|████▋     | 101/217 [00:02<00:02, 49.79batch/s]

Training batches on cuda:  49%|████▉     | 107/217 [00:02<00:02, 51.11batch/s]

Training batches on cuda:  52%|█████▏    | 113/217 [00:02<00:02, 51.58batch/s]

Training batches on cuda:  55%|█████▍    | 119/217 [00:02<00:01, 51.58batch/s]

Training batches on cuda:  58%|█████▊    | 125/217 [00:02<00:01, 51.65batch/s]

Training batches on cuda:  60%|██████    | 131/217 [00:02<00:01, 52.26batch/s]

Training batches on cuda:  63%|██████▎   | 137/217 [00:02<00:01, 52.44batch/s]

Training batches on cuda:  66%|██████▌   | 143/217 [00:02<00:01, 52.48batch/s]

Training batches on cuda:  69%|██████▊   | 149/217 [00:03<00:01, 52.90batch/s]

Training batches on cuda:  71%|███████▏  | 155/217 [00:03<00:01, 53.03batch/s]

Training batches on cuda:  74%|███████▍  | 161/217 [00:03<00:01, 52.88batch/s]

Training batches on cuda:  77%|███████▋  | 167/217 [00:03<00:00, 52.81batch/s]

Training batches on cuda:  80%|███████▉  | 173/217 [00:03<00:00, 52.42batch/s]

Training batches on cuda:  82%|████████▏ | 179/217 [00:03<00:00, 51.71batch/s]

Training batches on cuda:  85%|████████▌ | 185/217 [00:03<00:00, 52.89batch/s]

Training batches on cuda:  88%|████████▊ | 191/217 [00:03<00:00, 48.91batch/s]

Training batches on cuda:  90%|█████████ | 196/217 [00:03<00:00, 49.08batch/s]

Training batches on cuda:  93%|█████████▎| 201/217 [00:04<00:00, 48.99batch/s]

Training batches on cuda:  95%|█████████▍| 206/217 [00:04<00:00, 49.15batch/s]

Training batches on cuda:  97%|█████████▋| 211/217 [00:04<00:00, 47.90batch/s]

Training batches on cuda: 100%|█████████▉| 216/217 [00:04<00:00, 46.47batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:26<00:21,  4.25s/epoch, loss=0.000919, prev_loss=0.000919]
Training epochs on cuda:  60%|██████    | 6/10 [00:26<00:17,  4.38s/epoch, loss=0.000919, prev_loss=0.000919]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/217 [00:00<00:07, 27.57batch/s]

Training batches on cuda:   4%|▎         | 8/217 [00:00<00:06, 31.80batch/s]

Training batches on cuda:   6%|▋         | 14/217 [00:00<00:05, 35.61batch/s]

Training batches on cuda:   9%|▉         | 19/217 [00:00<00:05, 38.93batch/s]

Training batches on cuda:  12%|█▏        | 25/217 [00:00<00:04, 42.48batch/s]

Training batches on cuda:  14%|█▍        | 31/217 [00:00<00:04, 44.66batch/s]

Training batches on cuda:  17%|█▋        | 36/217 [00:00<00:04, 44.68batch/s]

Training batches on cuda:  19%|█▉        | 42/217 [00:00<00:03, 46.71batch/s]

Training batches on cuda:  22%|██▏       | 48/217 [00:01<00:03, 48.26batch/s]

Training batches on cuda:  25%|██▍       | 54/217 [00:01<00:03, 49.83batch/s]

Training batches on cuda:  28%|██▊       | 60/217 [00:01<00:03, 50.58batch/s]

Training batches on cuda:  30%|███       | 66/217 [00:01<00:02, 51.14batch/s]

Training batches on cuda:  33%|███▎      | 72/217 [00:01<00:02, 52.30batch/s]

Training batches on cuda:  36%|███▌      | 78/217 [00:01<00:02, 53.57batch/s]

Training batches on cuda:  39%|███▊      | 84/217 [00:01<00:02, 53.89batch/s]

Training batches on cuda:  41%|████▏     | 90/217 [00:01<00:02, 54.97batch/s]

Training batches on cuda:  44%|████▍     | 96/217 [00:01<00:02, 54.46batch/s]

Training batches on cuda:  47%|████▋     | 102/217 [00:01<00:02, 54.07batch/s]

Training batches on cuda:  50%|████▉     | 108/217 [00:02<00:02, 53.24batch/s]

Training batches on cuda:  53%|█████▎    | 114/217 [00:02<00:01, 52.50batch/s]

Training batches on cuda:  55%|█████▌    | 120/217 [00:02<00:01, 51.82batch/s]

Training batches on cuda:  58%|█████▊    | 126/217 [00:02<00:01, 51.38batch/s]

Training batches on cuda:  61%|██████    | 132/217 [00:02<00:01, 52.93batch/s]

Training batches on cuda:  64%|██████▎   | 138/217 [00:02<00:01, 54.13batch/s]

Training batches on cuda:  66%|██████▋   | 144/217 [00:02<00:01, 54.90batch/s]

Training batches on cuda:  69%|██████▉   | 150/217 [00:02<00:01, 55.05batch/s]

Training batches on cuda:  72%|███████▏  | 156/217 [00:02<00:01, 56.00batch/s]

Training batches on cuda:  75%|███████▍  | 162/217 [00:03<00:00, 56.37batch/s]

Training batches on cuda:  77%|███████▋  | 168/217 [00:03<00:00, 55.86batch/s]

Training batches on cuda:  80%|████████  | 174/217 [00:03<00:00, 53.14batch/s]

Training batches on cuda:  83%|████████▎ | 180/217 [00:03<00:00, 53.62batch/s]

Training batches on cuda:  86%|████████▌ | 186/217 [00:03<00:00, 52.96batch/s]

Training batches on cuda:  88%|████████▊ | 192/217 [00:03<00:00, 52.72batch/s]

Training batches on cuda:  91%|█████████ | 198/217 [00:03<00:00, 51.53batch/s]

Training batches on cuda:  94%|█████████▍| 204/217 [00:03<00:00, 50.90batch/s]

Training batches on cuda:  97%|█████████▋| 210/217 [00:04<00:00, 49.76batch/s]

Training batches on cuda: 100%|█████████▉| 216/217 [00:04<00:00, 50.22batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:17,  4.38s/epoch, loss=0.000918, prev_loss=0.000919]
Training epochs on cuda:  70%|███████   | 7/10 [00:30<00:13,  4.39s/epoch, loss=0.000918, prev_loss=0.000919]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/217 [00:00<00:08, 26.52batch/s]

Training batches on cuda:   4%|▎         | 8/217 [00:00<00:06, 30.77batch/s]

Training batches on cuda:   6%|▌         | 13/217 [00:00<00:06, 33.94batch/s]

Training batches on cuda:   9%|▉         | 19/217 [00:00<00:05, 38.68batch/s]

Training batches on cuda:  12%|█▏        | 25/217 [00:00<00:04, 42.52batch/s]

Training batches on cuda:  14%|█▍        | 31/217 [00:00<00:04, 45.08batch/s]

Training batches on cuda:  17%|█▋        | 37/217 [00:00<00:03, 47.38batch/s]

Training batches on cuda:  20%|█▉        | 43/217 [00:00<00:03, 48.84batch/s]

Training batches on cuda:  23%|██▎       | 49/217 [00:00<00:03, 49.34batch/s]

Training batches on cuda:  25%|██▌       | 55/217 [00:01<00:03, 50.61batch/s]

Training batches on cuda:  28%|██▊       | 61/217 [00:01<00:03, 49.36batch/s]

Training batches on cuda:  30%|███       | 66/217 [00:01<00:03, 48.54batch/s]

Training batches on cuda:  33%|███▎      | 72/217 [00:01<00:02, 49.45batch/s]

Training batches on cuda:  36%|███▌      | 78/217 [00:01<00:02, 50.60batch/s]

Training batches on cuda:  39%|███▊      | 84/217 [00:01<00:02, 49.38batch/s]

Training batches on cuda:  41%|████▏     | 90/217 [00:01<00:02, 49.99batch/s]

Training batches on cuda:  44%|████▍     | 96/217 [00:01<00:02, 50.51batch/s]

Training batches on cuda:  47%|████▋     | 102/217 [00:02<00:02, 50.56batch/s]

Training batches on cuda:  50%|████▉     | 108/217 [00:02<00:02, 50.48batch/s]

Training batches on cuda:  53%|█████▎    | 114/217 [00:02<00:02, 50.33batch/s]

Training batches on cuda:  55%|█████▌    | 120/217 [00:02<00:01, 51.98batch/s]

Training batches on cuda:  58%|█████▊    | 126/217 [00:02<00:01, 52.79batch/s]

Training batches on cuda:  61%|██████    | 132/217 [00:02<00:01, 51.65batch/s]

Training batches on cuda:  64%|██████▎   | 138/217 [00:02<00:01, 53.76batch/s]

Training batches on cuda:  66%|██████▋   | 144/217 [00:02<00:01, 49.01batch/s]

Training batches on cuda:  69%|██████▉   | 150/217 [00:02<00:01, 50.12batch/s]

Training batches on cuda:  72%|███████▏  | 156/217 [00:03<00:01, 49.41batch/s]

Training batches on cuda:  75%|███████▍  | 162/217 [00:03<00:01, 50.85batch/s]

Training batches on cuda:  77%|███████▋  | 168/217 [00:03<00:00, 50.54batch/s]

Training batches on cuda:  80%|████████  | 174/217 [00:03<00:00, 50.98batch/s]

Training batches on cuda:  83%|████████▎ | 180/217 [00:03<00:00, 51.37batch/s]

Training batches on cuda:  86%|████████▌ | 186/217 [00:03<00:00, 51.92batch/s]

Training batches on cuda:  88%|████████▊ | 192/217 [00:03<00:00, 52.69batch/s]

Training batches on cuda:  91%|█████████ | 198/217 [00:03<00:00, 52.79batch/s]

Training batches on cuda:  94%|█████████▍| 204/217 [00:04<00:00, 53.12batch/s]

Training batches on cuda:  97%|█████████▋| 210/217 [00:04<00:00, 53.44batch/s]

Training batches on cuda: 100%|█████████▉| 216/217 [00:04<00:00, 54.70batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:34<00:13,  4.39s/epoch, loss=0.000918, prev_loss=0.000918]
Training epochs on cuda:  80%|████████  | 8/10 [00:34<00:08,  4.44s/epoch, loss=0.000918, prev_loss=0.000918]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/217 [00:00<00:06, 35.19batch/s]

Training batches on cuda:   4%|▍         | 9/217 [00:00<00:05, 37.89batch/s]

Training batches on cuda:   7%|▋         | 15/217 [00:00<00:04, 40.94batch/s]

Training batches on cuda:  10%|▉         | 21/217 [00:00<00:04, 44.64batch/s]

Training batches on cuda:  12%|█▏        | 27/217 [00:00<00:03, 48.33batch/s]

Training batches on cuda:  15%|█▌        | 33/217 [00:00<00:03, 50.71batch/s]

Training batches on cuda:  18%|█▊        | 39/217 [00:00<00:03, 52.11batch/s]

Training batches on cuda:  21%|██        | 45/217 [00:00<00:03, 53.87batch/s]

Training batches on cuda:  24%|██▎       | 51/217 [00:00<00:03, 54.10batch/s]

Training batches on cuda:  26%|██▋       | 57/217 [00:01<00:02, 54.09batch/s]

Training batches on cuda:  29%|██▉       | 63/217 [00:01<00:02, 54.21batch/s]

Training batches on cuda:  32%|███▏      | 69/217 [00:01<00:02, 53.92batch/s]

Training batches on cuda:  35%|███▍      | 75/217 [00:01<00:02, 54.06batch/s]

Training batches on cuda:  37%|███▋      | 81/217 [00:01<00:02, 54.51batch/s]

Training batches on cuda:  40%|████      | 87/217 [00:01<00:02, 53.73batch/s]

Training batches on cuda:  43%|████▎     | 93/217 [00:01<00:02, 54.02batch/s]

Training batches on cuda:  46%|████▌     | 99/217 [00:01<00:02, 53.82batch/s]

Training batches on cuda:  48%|████▊     | 105/217 [00:01<00:02, 52.59batch/s]

Training batches on cuda:  51%|█████     | 111/217 [00:02<00:02, 52.70batch/s]

Training batches on cuda:  54%|█████▍    | 117/217 [00:02<00:01, 53.06batch/s]

Training batches on cuda:  57%|█████▋    | 123/217 [00:02<00:01, 53.48batch/s]

Training batches on cuda:  59%|█████▉    | 129/217 [00:02<00:01, 54.14batch/s]

Training batches on cuda:  62%|██████▏   | 135/217 [00:02<00:01, 54.45batch/s]

Training batches on cuda:  65%|██████▍   | 141/217 [00:02<00:01, 55.57batch/s]

Training batches on cuda:  68%|██████▊   | 147/217 [00:02<00:01, 56.06batch/s]

Training batches on cuda:  71%|███████   | 153/217 [00:02<00:01, 53.06batch/s]

Training batches on cuda:  73%|███████▎  | 159/217 [00:02<00:01, 52.63batch/s]

Training batches on cuda:  76%|███████▌  | 165/217 [00:03<00:00, 53.34batch/s]

Training batches on cuda:  79%|███████▉  | 171/217 [00:03<00:00, 52.66batch/s]

Training batches on cuda:  82%|████████▏ | 177/217 [00:03<00:00, 49.18batch/s]

Training batches on cuda:  84%|████████▍ | 183/217 [00:03<00:00, 50.42batch/s]

Training batches on cuda:  87%|████████▋ | 189/217 [00:03<00:00, 50.85batch/s]

Training batches on cuda:  90%|████████▉ | 195/217 [00:03<00:00, 50.87batch/s]

Training batches on cuda:  93%|█████████▎| 201/217 [00:03<00:00, 51.08batch/s]

Training batches on cuda:  95%|█████████▌| 207/217 [00:03<00:00, 51.93batch/s]

Training batches on cuda:  98%|█████████▊| 213/217 [00:04<00:00, 52.45batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:39<00:08,  4.44s/epoch, loss=0.000916, prev_loss=0.000918]
Training epochs on cuda:  90%|█████████ | 9/10 [00:39<00:04,  4.40s/epoch, loss=0.000916, prev_loss=0.000918]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/217 [00:00<00:10, 19.88batch/s]

Training batches on cuda:   4%|▎         | 8/217 [00:00<00:08, 24.42batch/s]

Training batches on cuda:   6%|▋         | 14/217 [00:00<00:06, 29.05batch/s]

Training batches on cuda:   9%|▉         | 20/217 [00:00<00:05, 33.53batch/s]

Training batches on cuda:  12%|█▏        | 26/217 [00:00<00:05, 38.11batch/s]

Training batches on cuda:  15%|█▍        | 32/217 [00:00<00:04, 42.65batch/s]

Training batches on cuda:  18%|█▊        | 38/217 [00:00<00:03, 46.38batch/s]

Training batches on cuda:  20%|██        | 44/217 [00:00<00:03, 49.56batch/s]

Training batches on cuda:  23%|██▎       | 50/217 [00:00<00:03, 52.03batch/s]

Training batches on cuda:  26%|██▌       | 56/217 [00:01<00:02, 53.91batch/s]

Training batches on cuda:  29%|██▊       | 62/217 [00:01<00:02, 55.46batch/s]

Training batches on cuda:  31%|███▏      | 68/217 [00:01<00:02, 56.66batch/s]

Training batches on cuda:  34%|███▍      | 74/217 [00:01<00:02, 57.48batch/s]

Training batches on cuda:  37%|███▋      | 80/217 [00:01<00:02, 57.74batch/s]

Training batches on cuda:  40%|███▉      | 86/217 [00:01<00:02, 57.97batch/s]

Training batches on cuda:  42%|████▏     | 92/217 [00:01<00:02, 58.49batch/s]

Training batches on cuda:  45%|████▌     | 98/217 [00:01<00:02, 58.38batch/s]

Training batches on cuda:  48%|████▊     | 104/217 [00:01<00:01, 58.28batch/s]

Training batches on cuda:  51%|█████     | 110/217 [00:01<00:01, 58.54batch/s]

Training batches on cuda:  53%|█████▎    | 116/217 [00:02<00:01, 58.69batch/s]

Training batches on cuda:  56%|█████▌    | 122/217 [00:02<00:01, 58.84batch/s]

Training batches on cuda:  59%|█████▉    | 128/217 [00:02<00:01, 58.66batch/s]

Training batches on cuda:  62%|██████▏   | 134/217 [00:02<00:01, 58.57batch/s]

Training batches on cuda:  65%|██████▍   | 140/217 [00:02<00:01, 58.79batch/s]

Training batches on cuda:  67%|██████▋   | 146/217 [00:02<00:01, 58.94batch/s]

Training batches on cuda:  70%|███████   | 152/217 [00:02<00:01, 58.15batch/s]

Training batches on cuda:  73%|███████▎  | 158/217 [00:02<00:01, 58.58batch/s]

Training batches on cuda:  76%|███████▌  | 164/217 [00:02<00:00, 58.83batch/s]

Training batches on cuda:  78%|███████▊  | 170/217 [00:03<00:00, 58.80batch/s]

Training batches on cuda:  81%|████████  | 176/217 [00:03<00:00, 58.16batch/s]

Training batches on cuda:  84%|████████▍ | 182/217 [00:03<00:00, 57.66batch/s]

Training batches on cuda:  87%|████████▋ | 188/217 [00:03<00:00, 58.01batch/s]

Training batches on cuda:  89%|████████▉ | 194/217 [00:03<00:00, 55.21batch/s]

Training batches on cuda:  92%|█████████▏| 200/217 [00:03<00:00, 56.25batch/s]

Training batches on cuda:  95%|█████████▍| 206/217 [00:03<00:00, 56.87batch/s]

Training batches on cuda:  98%|█████████▊| 212/217 [00:03<00:00, 57.18batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:43<00:04,  4.40s/epoch, loss=0.000906, prev_loss=0.000916]2021-09-16:18:58:07,807 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:18:58:08,899 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:18:58:23,963 INFO     [evaluator.py:657] Evaluation took 15.06s seconds
2021-09-16:18:58:24,99 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00,  9.19s/epoch, loss=0.000906, prev_loss=0.000916]
Training epochs on cuda: 100%|██████████| 10/10 [00:59<00:00,  5.96s/epoch, loss=0.000906, prev_loss=0.000916]
2021-09-16:18:58:24,100 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp5s7jcf8q'
2021-09-16:18:58:24,180 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp5s7jcf8q' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:18, 2.03ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:17, 2.18ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:16, 2.30ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.39ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.46ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.51ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.54ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.59ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.60ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.62ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.62ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.62ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.62ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.57ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.59ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.60ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.60ktriple/s]
2021-09-16:18:58:39,145 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
[I 2021-09-16 18:58:39,151] Trial 66 finished with value: 0.2231952602560262 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.0891707751572982, 'optimizer.lr': 0.01865272667623853, 'negative_sampler.num_negs_per_pos': 30, 'training.batch_size': 1432}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:18:58:39,175 WARNING  [api.py:823] No random seed is specified. Setting to 2398313523.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 12/9708 [00:00<01:23, 115.72batch/s]

Training batches on cuda:   0%|          | 36/9708 [00:00<01:11, 136.22batch/s]

Training batches on cuda:   1%|          | 59/9708 [00:00<01:02, 154.57batch/s]

Training batches on cuda:   1%|          | 81/9708 [00:00<00:56, 169.61batch/s]

Training batches on cuda:   1%|          | 101/9708 [00:00<00:54, 177.49batch/s]

Training batches on cuda:   1%|▏         | 123/9708 [00:00<00:50, 187.99batch/s]

Training batches on cuda:   1%|▏         | 145/9708 [00:00<00:48, 196.24batch/s]

Training batches on cuda:   2%|▏         | 167/9708 [00:00<00:47, 200.17batch/s]

Training batches on cuda:   2%|▏         | 187/9708 [00:00<00:48, 198.23batch/s]

Training batches on cuda:   2%|▏         | 208/9708 [00:01<00:47, 199.89batch/s]

Training batches on cuda:   2%|▏         | 229/9708 [00:01<00:46, 202.42batch/s]

Training batches on cuda:   3%|▎         | 250/9708 [00:01<00:47, 201.19batch/s]

Training batches on cuda:   3%|▎         | 272/9708 [00:01<00:46, 204.82batch/s]

Training batches on cuda:   3%|▎         | 294/9708 [00:01<00:45, 207.31batch/s]

Training batches on cuda:   3%|▎         | 316/9708 [00:01<00:44, 208.84batch/s]

Training batches on cuda:   3%|▎         | 338/9708 [00:01<00:44, 211.03batch/s]

Training batches on cuda:   4%|▎         | 360/9708 [00:01<00:44, 210.71batch/s]

Training batches on cuda:   4%|▍         | 382/9708 [00:01<00:44, 209.32batch/s]

Training batches on cuda:   4%|▍         | 404/9708 [00:01<00:44, 210.22batch/s]

Training batches on cuda:   4%|▍         | 426/9708 [00:02<00:44, 209.65batch/s]

Training batches on cuda:   5%|▍         | 447/9708 [00:02<00:44, 206.55batch/s]

Training batches on cuda:   5%|▍         | 468/9708 [00:02<00:44, 206.69batch/s]

Training batches on cuda:   5%|▌         | 489/9708 [00:02<00:45, 203.53batch/s]

Training batches on cuda:   5%|▌         | 510/9708 [00:02<00:45, 204.33batch/s]

Training batches on cuda:   5%|▌         | 531/9708 [00:02<00:44, 205.19batch/s]

Training batches on cuda:   6%|▌         | 553/9708 [00:02<00:44, 206.70batch/s]

Training batches on cuda:   6%|▌         | 575/9708 [00:02<00:43, 208.02batch/s]

Training batches on cuda:   6%|▌         | 596/9708 [00:02<00:44, 203.47batch/s]

Training batches on cuda:   6%|▋         | 617/9708 [00:03<00:45, 199.88batch/s]

Training batches on cuda:   7%|▋         | 638/9708 [00:03<00:45, 201.11batch/s]

Training batches on cuda:   7%|▋         | 659/9708 [00:03<00:45, 199.37batch/s]

Training batches on cuda:   7%|▋         | 681/9708 [00:03<00:44, 203.45batch/s]

Training batches on cuda:   7%|▋         | 704/9708 [00:03<00:43, 209.32batch/s]

Training batches on cuda:   7%|▋         | 726/9708 [00:03<00:42, 209.97batch/s]

Training batches on cuda:   8%|▊         | 748/9708 [00:03<00:42, 210.51batch/s]

Training batches on cuda:   8%|▊         | 772/9708 [00:03<00:41, 216.65batch/s]

Training batches on cuda:   8%|▊         | 794/9708 [00:03<00:41, 212.69batch/s]

Training batches on cuda:   8%|▊         | 816/9708 [00:03<00:43, 206.34batch/s]

Training batches on cuda:   9%|▊         | 837/9708 [00:04<00:43, 206.18batch/s]

Training batches on cuda:   9%|▉         | 858/9708 [00:04<00:43, 205.10batch/s]

Training batches on cuda:   9%|▉         | 879/9708 [00:04<00:43, 202.15batch/s]

Training batches on cuda:   9%|▉         | 901/9708 [00:04<00:43, 204.69batch/s]

Training batches on cuda:   9%|▉         | 922/9708 [00:04<00:43, 202.47batch/s]

Training batches on cuda:  10%|▉         | 944/9708 [00:04<00:42, 205.80batch/s]

Training batches on cuda:  10%|▉         | 965/9708 [00:04<00:42, 206.74batch/s]

Training batches on cuda:  10%|█         | 986/9708 [00:04<00:42, 203.69batch/s]

Training batches on cuda:  10%|█         | 1007/9708 [00:04<00:44, 197.63batch/s]

Training batches on cuda:  11%|█         | 1027/9708 [00:05<00:45, 192.58batch/s]

Training batches on cuda:  11%|█         | 1048/9708 [00:05<00:44, 196.36batch/s]

Training batches on cuda:  11%|█         | 1069/9708 [00:05<00:43, 199.18batch/s]

Training batches on cuda:  11%|█         | 1090/9708 [00:05<00:43, 198.87batch/s]

Training batches on cuda:  11%|█▏        | 1111/9708 [00:05<00:42, 200.58batch/s]

Training batches on cuda:  12%|█▏        | 1132/9708 [00:05<00:42, 201.55batch/s]

Training batches on cuda:  12%|█▏        | 1155/9708 [00:05<00:41, 206.44batch/s]

Training batches on cuda:  12%|█▏        | 1179/9708 [00:05<00:39, 215.10batch/s]

Training batches on cuda:  12%|█▏        | 1204/9708 [00:05<00:38, 222.54batch/s]

Training batches on cuda:  13%|█▎        | 1227/9708 [00:05<00:38, 221.36batch/s]

Training batches on cuda:  13%|█▎        | 1250/9708 [00:06<00:38, 216.99batch/s]

Training batches on cuda:  13%|█▎        | 1272/9708 [00:06<00:40, 208.30batch/s]

Training batches on cuda:  13%|█▎        | 1296/9708 [00:06<00:38, 215.71batch/s]

Training batches on cuda:  14%|█▎        | 1319/9708 [00:06<00:38, 218.71batch/s]

Training batches on cuda:  14%|█▍        | 1342/9708 [00:06<00:38, 219.93batch/s]

Training batches on cuda:  14%|█▍        | 1366/9708 [00:06<00:37, 224.60batch/s]

Training batches on cuda:  14%|█▍        | 1389/9708 [00:06<00:36, 225.77batch/s]

Training batches on cuda:  15%|█▍        | 1412/9708 [00:06<00:36, 226.48batch/s]

Training batches on cuda:  15%|█▍        | 1435/9708 [00:06<00:38, 215.20batch/s]

Training batches on cuda:  15%|█▌        | 1457/9708 [00:07<00:39, 209.63batch/s]

Training batches on cuda:  15%|█▌        | 1479/9708 [00:07<00:38, 211.02batch/s]

Training batches on cuda:  15%|█▌        | 1501/9708 [00:07<00:38, 211.73batch/s]

Training batches on cuda:  16%|█▌        | 1524/9708 [00:07<00:38, 214.35batch/s]

Training batches on cuda:  16%|█▌        | 1548/9708 [00:07<00:37, 218.91batch/s]

Training batches on cuda:  16%|█▌        | 1570/9708 [00:07<00:37, 215.89batch/s]

Training batches on cuda:  16%|█▋        | 1592/9708 [00:07<00:37, 213.88batch/s]

Training batches on cuda:  17%|█▋        | 1614/9708 [00:07<00:38, 212.86batch/s]

Training batches on cuda:  17%|█▋        | 1636/9708 [00:07<00:38, 208.63batch/s]

Training batches on cuda:  17%|█▋        | 1657/9708 [00:07<00:41, 193.12batch/s]

Training batches on cuda:  17%|█▋        | 1678/9708 [00:08<00:40, 195.91batch/s]

Training batches on cuda:  17%|█▋        | 1698/9708 [00:08<00:42, 190.20batch/s]

Training batches on cuda:  18%|█▊        | 1719/9708 [00:08<00:40, 195.65batch/s]

Training batches on cuda:  18%|█▊        | 1740/9708 [00:08<00:39, 199.41batch/s]

Training batches on cuda:  18%|█▊        | 1762/9708 [00:08<00:39, 203.24batch/s]

Training batches on cuda:  18%|█▊        | 1784/9708 [00:08<00:38, 206.52batch/s]

Training batches on cuda:  19%|█▊        | 1806/9708 [00:08<00:37, 208.86batch/s]

Training batches on cuda:  19%|█▉        | 1827/9708 [00:08<00:37, 209.03batch/s]

Training batches on cuda:  19%|█▉        | 1848/9708 [00:08<00:43, 182.61batch/s]

Training batches on cuda:  19%|█▉        | 1869/9708 [00:09<00:41, 187.58batch/s]

Training batches on cuda:  19%|█▉        | 1889/9708 [00:09<00:41, 190.14batch/s]

Training batches on cuda:  20%|█▉        | 1909/9708 [00:09<00:40, 192.77batch/s]

Training batches on cuda:  20%|█▉        | 1931/9708 [00:09<00:38, 199.70batch/s]

Training batches on cuda:  20%|██        | 1952/9708 [00:09<00:38, 199.37batch/s]

Training batches on cuda:  20%|██        | 1973/9708 [00:09<00:38, 202.27batch/s]

Training batches on cuda:  21%|██        | 1994/9708 [00:09<00:37, 203.46batch/s]

Training batches on cuda:  21%|██        | 2016/9708 [00:09<00:37, 205.53batch/s]

Training batches on cuda:  21%|██        | 2037/9708 [00:09<00:37, 206.71batch/s]

Training batches on cuda:  21%|██        | 2059/9708 [00:09<00:36, 208.51batch/s]

Training batches on cuda:  21%|██▏       | 2082/9708 [00:10<00:35, 212.79batch/s]

Training batches on cuda:  22%|██▏       | 2105/9708 [00:10<00:34, 217.54batch/s]

Training batches on cuda:  22%|██▏       | 2127/9708 [00:10<00:35, 214.45batch/s]

Training batches on cuda:  22%|██▏       | 2149/9708 [00:10<00:35, 213.47batch/s]

Training batches on cuda:  22%|██▏       | 2171/9708 [00:10<00:36, 208.23batch/s]

Training batches on cuda:  23%|██▎       | 2193/9708 [00:10<00:35, 210.16batch/s]

Training batches on cuda:  23%|██▎       | 2215/9708 [00:10<00:35, 209.12batch/s]

Training batches on cuda:  23%|██▎       | 2236/9708 [00:10<00:35, 208.60batch/s]

Training batches on cuda:  23%|██▎       | 2257/9708 [00:10<00:36, 206.64batch/s]

Training batches on cuda:  23%|██▎       | 2278/9708 [00:11<00:35, 207.46batch/s]

Training batches on cuda:  24%|██▎       | 2299/9708 [00:11<00:35, 205.81batch/s]

Training batches on cuda:  24%|██▍       | 2320/9708 [00:11<00:36, 201.81batch/s]

Training batches on cuda:  24%|██▍       | 2342/9708 [00:11<00:36, 204.40batch/s]

Training batches on cuda:  24%|██▍       | 2365/9708 [00:11<00:35, 208.97batch/s]

Training batches on cuda:  25%|██▍       | 2386/9708 [00:11<00:36, 198.27batch/s]

Training batches on cuda:  25%|██▍       | 2406/9708 [00:11<00:38, 191.35batch/s]

Training batches on cuda:  25%|██▍       | 2426/9708 [00:11<00:38, 190.16batch/s]

Training batches on cuda:  25%|██▌       | 2447/9708 [00:11<00:37, 193.32batch/s]

Training batches on cuda:  25%|██▌       | 2470/9708 [00:11<00:35, 202.67batch/s]

Training batches on cuda:  26%|██▌       | 2494/9708 [00:12<00:34, 212.08batch/s]

Training batches on cuda:  26%|██▌       | 2517/9708 [00:12<00:33, 216.65batch/s]

Training batches on cuda:  26%|██▌       | 2540/9708 [00:12<00:32, 219.28batch/s]

Training batches on cuda:  26%|██▋       | 2563/9708 [00:12<00:32, 218.13batch/s]

Training batches on cuda:  27%|██▋       | 2585/9708 [00:12<00:32, 216.32batch/s]

Training batches on cuda:  27%|██▋       | 2607/9708 [00:12<00:33, 210.06batch/s]

Training batches on cuda:  27%|██▋       | 2629/9708 [00:12<00:34, 203.38batch/s]

Training batches on cuda:  27%|██▋       | 2651/9708 [00:12<00:34, 207.42batch/s]

Training batches on cuda:  28%|██▊       | 2673/9708 [00:12<00:33, 210.38batch/s]

Training batches on cuda:  28%|██▊       | 2695/9708 [00:13<00:34, 204.57batch/s]

Training batches on cuda:  28%|██▊       | 2716/9708 [00:13<00:34, 202.27batch/s]

Training batches on cuda:  28%|██▊       | 2737/9708 [00:13<00:34, 202.94batch/s]

Training batches on cuda:  28%|██▊       | 2758/9708 [00:13<00:33, 204.74batch/s]

Training batches on cuda:  29%|██▊       | 2779/9708 [00:13<00:35, 193.80batch/s]

Training batches on cuda:  29%|██▉       | 2799/9708 [00:13<00:36, 189.90batch/s]

Training batches on cuda:  29%|██▉       | 2819/9708 [00:13<00:36, 188.21batch/s]

Training batches on cuda:  29%|██▉       | 2838/9708 [00:13<00:37, 184.55batch/s]

Training batches on cuda:  29%|██▉       | 2860/9708 [00:13<00:35, 192.27batch/s]

Training batches on cuda:  30%|██▉       | 2880/9708 [00:14<00:36, 185.07batch/s]

Training batches on cuda:  30%|██▉       | 2899/9708 [00:14<00:37, 182.44batch/s]

Training batches on cuda:  30%|███       | 2918/9708 [00:14<00:36, 183.94batch/s]

Training batches on cuda:  30%|███       | 2937/9708 [00:14<00:36, 183.97batch/s]

Training batches on cuda:  30%|███       | 2959/9708 [00:14<00:35, 192.16batch/s]

Training batches on cuda:  31%|███       | 2979/9708 [00:14<00:34, 193.78batch/s]

Training batches on cuda:  31%|███       | 3000/9708 [00:14<00:33, 197.58batch/s]

Training batches on cuda:  31%|███       | 3021/9708 [00:14<00:33, 199.66batch/s]

Training batches on cuda:  31%|███▏      | 3042/9708 [00:14<00:33, 198.49batch/s]

Training batches on cuda:  32%|███▏      | 3062/9708 [00:14<00:33, 196.79batch/s]

Training batches on cuda:  32%|███▏      | 3082/9708 [00:15<00:33, 197.38batch/s]

Training batches on cuda:  32%|███▏      | 3104/9708 [00:15<00:32, 201.35batch/s]

Training batches on cuda:  32%|███▏      | 3126/9708 [00:15<00:32, 205.49batch/s]

Training batches on cuda:  32%|███▏      | 3149/9708 [00:15<00:31, 209.63batch/s]

Training batches on cuda:  33%|███▎      | 3173/9708 [00:15<00:30, 217.13batch/s]

Training batches on cuda:  33%|███▎      | 3195/9708 [00:15<00:30, 215.99batch/s]

Training batches on cuda:  33%|███▎      | 3217/9708 [00:15<00:30, 215.49batch/s]

Training batches on cuda:  33%|███▎      | 3239/9708 [00:15<00:30, 214.03batch/s]

Training batches on cuda:  34%|███▎      | 3261/9708 [00:15<00:30, 212.81batch/s]

Training batches on cuda:  34%|███▍      | 3283/9708 [00:15<00:30, 213.34batch/s]

Training batches on cuda:  34%|███▍      | 3305/9708 [00:16<00:29, 213.54batch/s]

Training batches on cuda:  34%|███▍      | 3330/9708 [00:16<00:28, 222.68batch/s]

Training batches on cuda:  35%|███▍      | 3357/9708 [00:16<00:27, 233.98batch/s]

Training batches on cuda:  35%|███▍      | 3384/9708 [00:16<00:26, 242.52batch/s]

Training batches on cuda:  35%|███▌      | 3409/9708 [00:16<00:25, 242.45batch/s]

Training batches on cuda:  35%|███▌      | 3434/9708 [00:16<00:27, 227.89batch/s]

Training batches on cuda:  36%|███▌      | 3458/9708 [00:16<00:28, 217.43batch/s]

Training batches on cuda:  36%|███▌      | 3481/9708 [00:16<00:29, 209.22batch/s]

Training batches on cuda:  36%|███▌      | 3503/9708 [00:16<00:30, 202.80batch/s]

Training batches on cuda:  36%|███▋      | 3524/9708 [00:17<00:31, 196.72batch/s]

Training batches on cuda:  37%|███▋      | 3545/9708 [00:17<00:31, 198.66batch/s]

Training batches on cuda:  37%|███▋      | 3566/9708 [00:17<00:31, 193.06batch/s]

Training batches on cuda:  37%|███▋      | 3586/9708 [00:17<00:31, 191.76batch/s]

Training batches on cuda:  37%|███▋      | 3606/9708 [00:17<00:31, 193.26batch/s]

Training batches on cuda:  37%|███▋      | 3626/9708 [00:17<00:31, 193.42batch/s]

Training batches on cuda:  38%|███▊      | 3647/9708 [00:17<00:30, 196.16batch/s]

Training batches on cuda:  38%|███▊      | 3668/9708 [00:17<00:30, 198.96batch/s]

Training batches on cuda:  38%|███▊      | 3689/9708 [00:17<00:30, 200.57batch/s]

Training batches on cuda:  38%|███▊      | 3710/9708 [00:18<00:30, 198.76batch/s]

Training batches on cuda:  38%|███▊      | 3731/9708 [00:18<00:29, 200.51batch/s]

Training batches on cuda:  39%|███▊      | 3752/9708 [00:18<00:29, 202.13batch/s]

Training batches on cuda:  39%|███▉      | 3773/9708 [00:18<00:29, 201.60batch/s]

Training batches on cuda:  39%|███▉      | 3794/9708 [00:18<00:30, 195.19batch/s]

Training batches on cuda:  39%|███▉      | 3818/9708 [00:18<00:28, 204.53batch/s]

Training batches on cuda:  40%|███▉      | 3839/9708 [00:18<00:28, 203.43batch/s]

Training batches on cuda:  40%|███▉      | 3860/9708 [00:18<00:28, 204.96batch/s]

Training batches on cuda:  40%|███▉      | 3881/9708 [00:18<00:29, 198.08batch/s]

Training batches on cuda:  40%|████      | 3901/9708 [00:19<00:29, 194.68batch/s]

Training batches on cuda:  40%|████      | 3921/9708 [00:19<00:32, 175.89batch/s]

Training batches on cuda:  41%|████      | 3939/9708 [00:19<00:32, 176.15batch/s]

Training batches on cuda:  41%|████      | 3957/9708 [00:19<00:32, 177.27batch/s]

Training batches on cuda:  41%|████      | 3976/9708 [00:19<00:32, 178.32batch/s]

Training batches on cuda:  41%|████      | 3996/9708 [00:19<00:31, 183.22batch/s]

Training batches on cuda:  41%|████▏     | 4015/9708 [00:19<00:31, 182.03batch/s]

Training batches on cuda:  42%|████▏     | 4035/9708 [00:19<00:30, 186.02batch/s]

Training batches on cuda:  42%|████▏     | 4055/9708 [00:19<00:30, 187.76batch/s]

Training batches on cuda:  42%|████▏     | 4077/9708 [00:19<00:28, 194.77batch/s]

Training batches on cuda:  42%|████▏     | 4100/9708 [00:20<00:27, 201.82batch/s]

Training batches on cuda:  42%|████▏     | 4121/9708 [00:20<00:27, 204.06batch/s]

Training batches on cuda:  43%|████▎     | 4145/9708 [00:20<00:26, 212.64batch/s]

Training batches on cuda:  43%|████▎     | 4169/9708 [00:20<00:25, 218.00batch/s]

Training batches on cuda:  43%|████▎     | 4191/9708 [00:20<00:26, 209.26batch/s]

Training batches on cuda:  43%|████▎     | 4213/9708 [00:20<00:27, 200.37batch/s]

Training batches on cuda:  44%|████▎     | 4234/9708 [00:20<00:27, 198.29batch/s]

Training batches on cuda:  44%|████▍     | 4254/9708 [00:20<00:27, 198.39batch/s]

Training batches on cuda:  44%|████▍     | 4274/9708 [00:20<00:27, 198.77batch/s]

Training batches on cuda:  44%|████▍     | 4294/9708 [00:21<00:27, 198.65batch/s]

Training batches on cuda:  44%|████▍     | 4316/9708 [00:21<00:26, 203.62batch/s]

Training batches on cuda:  45%|████▍     | 4339/9708 [00:21<00:25, 208.69batch/s]

Training batches on cuda:  45%|████▍     | 4362/9708 [00:21<00:25, 212.23batch/s]

Training batches on cuda:  45%|████▌     | 4385/9708 [00:21<00:24, 215.97batch/s]

Training batches on cuda:  45%|████▌     | 4408/9708 [00:21<00:24, 217.32batch/s]

Training batches on cuda:  46%|████▌     | 4430/9708 [00:21<00:24, 216.11batch/s]

Training batches on cuda:  46%|████▌     | 4452/9708 [00:21<00:24, 211.87batch/s]

Training batches on cuda:  46%|████▌     | 4474/9708 [00:21<00:25, 204.48batch/s]

Training batches on cuda:  46%|████▋     | 4495/9708 [00:21<00:25, 204.00batch/s]

Training batches on cuda:  47%|████▋     | 4516/9708 [00:22<00:26, 194.89batch/s]

Training batches on cuda:  47%|████▋     | 4537/9708 [00:22<00:26, 196.94batch/s]

Training batches on cuda:  47%|████▋     | 4557/9708 [00:22<00:26, 197.24batch/s]

Training batches on cuda:  47%|████▋     | 4577/9708 [00:22<00:26, 194.38batch/s]

Training batches on cuda:  47%|████▋     | 4597/9708 [00:22<00:26, 196.00batch/s]

Training batches on cuda:  48%|████▊     | 4617/9708 [00:22<00:25, 196.24batch/s]

Training batches on cuda:  48%|████▊     | 4639/9708 [00:22<00:25, 201.11batch/s]

Training batches on cuda:  48%|████▊     | 4660/9708 [00:22<00:27, 184.28batch/s]

Training batches on cuda:  48%|████▊     | 4682/9708 [00:22<00:25, 193.63batch/s]

Training batches on cuda:  48%|████▊     | 4702/9708 [00:23<00:26, 189.92batch/s]

Training batches on cuda:  49%|████▊     | 4723/9708 [00:23<00:25, 195.23batch/s]

Training batches on cuda:  49%|████▉     | 4745/9708 [00:23<00:24, 200.13batch/s]

Training batches on cuda:  49%|████▉     | 4767/9708 [00:23<00:24, 202.97batch/s]

Training batches on cuda:  49%|████▉     | 4788/9708 [00:23<00:24, 199.41batch/s]

Training batches on cuda:  50%|████▉     | 4809/9708 [00:23<00:24, 200.19batch/s]

Training batches on cuda:  50%|████▉     | 4830/9708 [00:23<00:24, 195.43batch/s]

Training batches on cuda:  50%|████▉     | 4852/9708 [00:23<00:24, 199.89batch/s]

Training batches on cuda:  50%|█████     | 4873/9708 [00:23<00:24, 198.51batch/s]

Training batches on cuda:  50%|█████     | 4894/9708 [00:23<00:24, 199.75batch/s]

Training batches on cuda:  51%|█████     | 4915/9708 [00:24<00:26, 180.53batch/s]

Training batches on cuda:  51%|█████     | 4934/9708 [00:24<00:27, 175.46batch/s]

Training batches on cuda:  51%|█████     | 4953/9708 [00:24<00:26, 179.35batch/s]

Training batches on cuda:  51%|█████     | 4974/9708 [00:24<00:25, 185.99batch/s]

Training batches on cuda:  51%|█████▏    | 4995/9708 [00:24<00:24, 190.30batch/s]

Training batches on cuda:  52%|█████▏    | 5015/9708 [00:24<00:24, 192.46batch/s]

Training batches on cuda:  52%|█████▏    | 5036/9708 [00:24<00:23, 195.85batch/s]

Training batches on cuda:  52%|█████▏    | 5056/9708 [00:24<00:24, 192.36batch/s]

Training batches on cuda:  52%|█████▏    | 5076/9708 [00:24<00:24, 192.85batch/s]

Training batches on cuda:  52%|█████▏    | 5096/9708 [00:25<00:23, 194.77batch/s]

Training batches on cuda:  53%|█████▎    | 5117/9708 [00:25<00:23, 198.96batch/s]

Training batches on cuda:  53%|█████▎    | 5137/9708 [00:25<00:23, 195.78batch/s]

Training batches on cuda:  53%|█████▎    | 5157/9708 [00:25<00:23, 190.02batch/s]

Training batches on cuda:  53%|█████▎    | 5178/9708 [00:25<00:23, 193.67batch/s]

Training batches on cuda:  54%|█████▎    | 5200/9708 [00:25<00:22, 199.73batch/s]

Training batches on cuda:  54%|█████▍    | 5221/9708 [00:25<00:22, 198.00batch/s]

Training batches on cuda:  54%|█████▍    | 5241/9708 [00:25<00:24, 185.28batch/s]

Training batches on cuda:  54%|█████▍    | 5262/9708 [00:25<00:23, 191.21batch/s]

Training batches on cuda:  54%|█████▍    | 5285/9708 [00:26<00:22, 199.75batch/s]

Training batches on cuda:  55%|█████▍    | 5306/9708 [00:26<00:22, 197.01batch/s]

Training batches on cuda:  55%|█████▍    | 5328/9708 [00:26<00:21, 201.50batch/s]

Training batches on cuda:  55%|█████▌    | 5353/9708 [00:26<00:20, 212.64batch/s]

Training batches on cuda:  55%|█████▌    | 5379/9708 [00:26<00:19, 222.87batch/s]

Training batches on cuda:  56%|█████▌    | 5405/9708 [00:26<00:18, 230.59batch/s]

Training batches on cuda:  56%|█████▌    | 5430/9708 [00:26<00:18, 235.39batch/s]

Training batches on cuda:  56%|█████▌    | 5454/9708 [00:26<00:18, 234.08batch/s]

Training batches on cuda:  56%|█████▋    | 5478/9708 [00:26<00:20, 208.75batch/s]

Training batches on cuda:  57%|█████▋    | 5500/9708 [00:27<00:20, 202.61batch/s]

Training batches on cuda:  57%|█████▋    | 5523/9708 [00:27<00:20, 207.98batch/s]

Training batches on cuda:  57%|█████▋    | 5546/9708 [00:27<00:19, 211.64batch/s]

Training batches on cuda:  57%|█████▋    | 5568/9708 [00:27<00:19, 210.53batch/s]

Training batches on cuda:  58%|█████▊    | 5590/9708 [00:27<00:19, 210.50batch/s]

Training batches on cuda:  58%|█████▊    | 5612/9708 [00:27<00:19, 211.94batch/s]

Training batches on cuda:  58%|█████▊    | 5634/9708 [00:27<00:19, 209.74batch/s]

Training batches on cuda:  58%|█████▊    | 5656/9708 [00:27<00:19, 209.55batch/s]

Training batches on cuda:  58%|█████▊    | 5678/9708 [00:27<00:19, 202.97batch/s]

Training batches on cuda:  59%|█████▊    | 5699/9708 [00:27<00:19, 204.65batch/s]

Training batches on cuda:  59%|█████▉    | 5720/9708 [00:28<00:19, 204.55batch/s]

Training batches on cuda:  59%|█████▉    | 5741/9708 [00:28<00:19, 203.21batch/s]

Training batches on cuda:  59%|█████▉    | 5762/9708 [00:28<00:19, 199.75batch/s]

Training batches on cuda:  60%|█████▉    | 5783/9708 [00:28<00:19, 196.88batch/s]

Training batches on cuda:  60%|█████▉    | 5804/9708 [00:28<00:19, 198.47batch/s]

Training batches on cuda:  60%|██████    | 5825/9708 [00:28<00:19, 199.74batch/s]

Training batches on cuda:  60%|██████    | 5846/9708 [00:28<00:19, 201.82batch/s]

Training batches on cuda:  60%|██████    | 5867/9708 [00:28<00:19, 198.69batch/s]

Training batches on cuda:  61%|██████    | 5889/9708 [00:28<00:18, 203.41batch/s]

Training batches on cuda:  61%|██████    | 5910/9708 [00:29<00:20, 189.15batch/s]

Training batches on cuda:  61%|██████    | 5931/9708 [00:29<00:19, 192.69batch/s]

Training batches on cuda:  61%|██████▏   | 5951/9708 [00:29<00:19, 194.48batch/s]

Training batches on cuda:  62%|██████▏   | 5974/9708 [00:29<00:18, 203.25batch/s]

Training batches on cuda:  62%|██████▏   | 5995/9708 [00:29<00:18, 195.81batch/s]

Training batches on cuda:  62%|██████▏   | 6015/9708 [00:29<00:19, 190.54batch/s]

Training batches on cuda:  62%|██████▏   | 6035/9708 [00:29<00:19, 192.18batch/s]

Training batches on cuda:  62%|██████▏   | 6056/9708 [00:29<00:18, 194.61batch/s]

Training batches on cuda:  63%|██████▎   | 6077/9708 [00:29<00:18, 197.28batch/s]

Training batches on cuda:  63%|██████▎   | 6098/9708 [00:29<00:18, 200.21batch/s]

Training batches on cuda:  63%|██████▎   | 6119/9708 [00:30<00:17, 202.89batch/s]

Training batches on cuda:  63%|██████▎   | 6141/9708 [00:30<00:17, 206.12batch/s]

Training batches on cuda:  63%|██████▎   | 6162/9708 [00:30<00:17, 204.42batch/s]

Training batches on cuda:  64%|██████▎   | 6183/9708 [00:30<00:17, 205.01batch/s]

Training batches on cuda:  64%|██████▍   | 6205/9708 [00:30<00:16, 206.72batch/s]

Training batches on cuda:  64%|██████▍   | 6227/9708 [00:30<00:16, 208.69batch/s]

Training batches on cuda:  64%|██████▍   | 6248/9708 [00:30<00:16, 204.63batch/s]

Training batches on cuda:  65%|██████▍   | 6270/9708 [00:30<00:16, 207.50batch/s]

Training batches on cuda:  65%|██████▍   | 6291/9708 [00:30<00:17, 198.62batch/s]

Training batches on cuda:  65%|██████▌   | 6315/9708 [00:31<00:16, 207.21batch/s]

Training batches on cuda:  65%|██████▌   | 6341/9708 [00:31<00:15, 219.14batch/s]

Training batches on cuda:  66%|██████▌   | 6365/9708 [00:31<00:14, 223.56batch/s]

Training batches on cuda:  66%|██████▌   | 6388/9708 [00:31<00:15, 215.41batch/s]

Training batches on cuda:  66%|██████▌   | 6410/9708 [00:31<00:15, 207.29batch/s]

Training batches on cuda:  66%|██████▌   | 6431/9708 [00:31<00:15, 207.76batch/s]

Training batches on cuda:  66%|██████▋   | 6454/9708 [00:31<00:15, 213.47batch/s]

Training batches on cuda:  67%|██████▋   | 6477/9708 [00:31<00:14, 217.17batch/s]

Training batches on cuda:  67%|██████▋   | 6500/9708 [00:31<00:14, 220.10batch/s]

Training batches on cuda:  67%|██████▋   | 6524/9708 [00:31<00:14, 223.56batch/s]

Training batches on cuda:  67%|██████▋   | 6547/9708 [00:32<00:14, 215.56batch/s]

Training batches on cuda:  68%|██████▊   | 6569/9708 [00:32<00:15, 202.54batch/s]

Training batches on cuda:  68%|██████▊   | 6590/9708 [00:32<00:16, 193.62batch/s]

Training batches on cuda:  68%|██████▊   | 6610/9708 [00:32<00:16, 189.68batch/s]

Training batches on cuda:  68%|██████▊   | 6632/9708 [00:32<00:15, 197.68batch/s]

Training batches on cuda:  69%|██████▊   | 6654/9708 [00:32<00:15, 203.26batch/s]

Training batches on cuda:  69%|██████▉   | 6677/9708 [00:32<00:14, 209.03batch/s]

Training batches on cuda:  69%|██████▉   | 6699/9708 [00:32<00:14, 211.25batch/s]

Training batches on cuda:  69%|██████▉   | 6722/9708 [00:32<00:13, 216.42batch/s]

Training batches on cuda:  69%|██████▉   | 6745/9708 [00:33<00:13, 218.48batch/s]

Training batches on cuda:  70%|██████▉   | 6768/9708 [00:33<00:13, 220.03batch/s]

Training batches on cuda:  70%|██████▉   | 6791/9708 [00:33<00:13, 219.54batch/s]

Training batches on cuda:  70%|███████   | 6814/9708 [00:33<00:13, 218.38batch/s]

Training batches on cuda:  70%|███████   | 6836/9708 [00:33<00:13, 218.08batch/s]

Training batches on cuda:  71%|███████   | 6859/9708 [00:33<00:12, 219.52batch/s]

Training batches on cuda:  71%|███████   | 6882/9708 [00:33<00:12, 220.80batch/s]

Training batches on cuda:  71%|███████   | 6905/9708 [00:33<00:12, 222.29batch/s]

Training batches on cuda:  71%|███████▏  | 6928/9708 [00:33<00:12, 221.66batch/s]

Training batches on cuda:  72%|███████▏  | 6951/9708 [00:33<00:12, 222.58batch/s]

Training batches on cuda:  72%|███████▏  | 6974/9708 [00:34<00:12, 222.66batch/s]

Training batches on cuda:  72%|███████▏  | 6997/9708 [00:34<00:13, 200.62batch/s]

Training batches on cuda:  72%|███████▏  | 7020/9708 [00:34<00:13, 206.59batch/s]

Training batches on cuda:  73%|███████▎  | 7043/9708 [00:34<00:12, 211.98batch/s]

Training batches on cuda:  73%|███████▎  | 7067/9708 [00:34<00:12, 216.77batch/s]

Training batches on cuda:  73%|███████▎  | 7090/9708 [00:34<00:11, 220.34batch/s]

Training batches on cuda:  73%|███████▎  | 7113/9708 [00:34<00:11, 221.00batch/s]

Training batches on cuda:  74%|███████▎  | 7137/9708 [00:34<00:11, 225.04batch/s]

Training batches on cuda:  74%|███████▍  | 7160/9708 [00:34<00:11, 224.69batch/s]

Training batches on cuda:  74%|███████▍  | 7185/9708 [00:35<00:10, 230.19batch/s]

Training batches on cuda:  74%|███████▍  | 7209/9708 [00:35<00:11, 225.50batch/s]

Training batches on cuda:  74%|███████▍  | 7232/9708 [00:35<00:11, 222.77batch/s]

Training batches on cuda:  75%|███████▍  | 7255/9708 [00:35<00:11, 217.06batch/s]

Training batches on cuda:  75%|███████▍  | 7277/9708 [00:35<00:11, 216.86batch/s]

Training batches on cuda:  75%|███████▌  | 7299/9708 [00:35<00:11, 214.27batch/s]

Training batches on cuda:  75%|███████▌  | 7321/9708 [00:35<00:11, 213.35batch/s]

Training batches on cuda:  76%|███████▌  | 7344/9708 [00:35<00:10, 215.80batch/s]

Training batches on cuda:  76%|███████▌  | 7367/9708 [00:35<00:10, 219.18batch/s]

Training batches on cuda:  76%|███████▌  | 7390/9708 [00:36<00:10, 220.78batch/s]

Training batches on cuda:  76%|███████▋  | 7413/9708 [00:36<00:10, 221.98batch/s]

Training batches on cuda:  77%|███████▋  | 7436/9708 [00:36<00:10, 222.47batch/s]

Training batches on cuda:  77%|███████▋  | 7459/9708 [00:36<00:10, 223.40batch/s]

Training batches on cuda:  77%|███████▋  | 7482/9708 [00:36<00:09, 224.97batch/s]

Training batches on cuda:  77%|███████▋  | 7505/9708 [00:36<00:09, 225.02batch/s]

Training batches on cuda:  78%|███████▊  | 7528/9708 [00:36<00:09, 224.55batch/s]

Training batches on cuda:  78%|███████▊  | 7551/9708 [00:36<00:09, 224.11batch/s]

Training batches on cuda:  78%|███████▊  | 7574/9708 [00:36<00:09, 219.68batch/s]

Training batches on cuda:  78%|███████▊  | 7598/9708 [00:36<00:09, 224.82batch/s]

Training batches on cuda:  79%|███████▊  | 7621/9708 [00:37<00:09, 221.39batch/s]

Training batches on cuda:  79%|███████▊  | 7644/9708 [00:37<00:09, 214.76batch/s]

Training batches on cuda:  79%|███████▉  | 7666/9708 [00:37<00:09, 212.50batch/s]

Training batches on cuda:  79%|███████▉  | 7688/9708 [00:37<00:09, 212.01batch/s]

Training batches on cuda:  79%|███████▉  | 7710/9708 [00:37<00:09, 210.77batch/s]

Training batches on cuda:  80%|███████▉  | 7732/9708 [00:37<00:09, 209.29batch/s]

Training batches on cuda:  80%|███████▉  | 7753/9708 [00:37<00:09, 206.98batch/s]

Training batches on cuda:  80%|████████  | 7775/9708 [00:37<00:09, 208.73batch/s]

Training batches on cuda:  80%|████████  | 7797/9708 [00:37<00:09, 210.69batch/s]

Training batches on cuda:  81%|████████  | 7819/9708 [00:37<00:08, 211.39batch/s]

Training batches on cuda:  81%|████████  | 7841/9708 [00:38<00:08, 211.10batch/s]

Training batches on cuda:  81%|████████  | 7863/9708 [00:38<00:08, 211.67batch/s]

Training batches on cuda:  81%|████████  | 7885/9708 [00:38<00:08, 209.11batch/s]

Training batches on cuda:  81%|████████▏ | 7906/9708 [00:38<00:08, 206.84batch/s]

Training batches on cuda:  82%|████████▏ | 7927/9708 [00:38<00:08, 200.56batch/s]

Training batches on cuda:  82%|████████▏ | 7948/9708 [00:38<00:08, 203.17batch/s]

Training batches on cuda:  82%|████████▏ | 7970/9708 [00:38<00:08, 205.97batch/s]

Training batches on cuda:  82%|████████▏ | 7991/9708 [00:38<00:08, 205.04batch/s]

Training batches on cuda:  83%|████████▎ | 8014/9708 [00:38<00:08, 209.97batch/s]

Training batches on cuda:  83%|████████▎ | 8036/9708 [00:39<00:07, 211.66batch/s]

Training batches on cuda:  83%|████████▎ | 8058/9708 [00:39<00:07, 213.64batch/s]

Training batches on cuda:  83%|████████▎ | 8080/9708 [00:39<00:08, 196.04batch/s]

Training batches on cuda:  83%|████████▎ | 8101/9708 [00:39<00:08, 198.72batch/s]

Training batches on cuda:  84%|████████▎ | 8122/9708 [00:39<00:07, 198.30batch/s]

Training batches on cuda:  84%|████████▍ | 8142/9708 [00:39<00:07, 196.46batch/s]

Training batches on cuda:  84%|████████▍ | 8165/9708 [00:39<00:07, 204.92batch/s]

Training batches on cuda:  84%|████████▍ | 8188/9708 [00:39<00:07, 210.33batch/s]

Training batches on cuda:  85%|████████▍ | 8213/9708 [00:39<00:06, 219.14batch/s]

Training batches on cuda:  85%|████████▍ | 8238/9708 [00:39<00:06, 225.67batch/s]

Training batches on cuda:  85%|████████▌ | 8262/9708 [00:40<00:06, 227.60batch/s]

Training batches on cuda:  85%|████████▌ | 8285/9708 [00:40<00:06, 219.14batch/s]

Training batches on cuda:  86%|████████▌ | 8309/9708 [00:40<00:06, 223.19batch/s]

Training batches on cuda:  86%|████████▌ | 8332/9708 [00:40<00:06, 224.34batch/s]

Training batches on cuda:  86%|████████▌ | 8355/9708 [00:40<00:06, 224.74batch/s]

Training batches on cuda:  86%|████████▋ | 8378/9708 [00:40<00:06, 216.53batch/s]

Training batches on cuda:  87%|████████▋ | 8402/9708 [00:40<00:05, 222.83batch/s]

Training batches on cuda:  87%|████████▋ | 8425/9708 [00:40<00:05, 217.02batch/s]

Training batches on cuda:  87%|████████▋ | 8447/9708 [00:40<00:05, 216.98batch/s]

Training batches on cuda:  87%|████████▋ | 8469/9708 [00:41<00:05, 211.43batch/s]

Training batches on cuda:  87%|████████▋ | 8491/9708 [00:41<00:05, 208.85batch/s]

Training batches on cuda:  88%|████████▊ | 8513/9708 [00:41<00:05, 209.69batch/s]

Training batches on cuda:  88%|████████▊ | 8536/9708 [00:41<00:05, 212.62batch/s]

Training batches on cuda:  88%|████████▊ | 8559/9708 [00:41<00:05, 216.34batch/s]

Training batches on cuda:  88%|████████▊ | 8582/9708 [00:41<00:05, 217.05batch/s]

Training batches on cuda:  89%|████████▊ | 8604/9708 [00:41<00:05, 214.99batch/s]

Training batches on cuda:  89%|████████▉ | 8627/9708 [00:41<00:04, 218.13batch/s]

Training batches on cuda:  89%|████████▉ | 8650/9708 [00:41<00:04, 220.62batch/s]

Training batches on cuda:  89%|████████▉ | 8673/9708 [00:41<00:04, 218.71batch/s]

Training batches on cuda:  90%|████████▉ | 8695/9708 [00:42<00:04, 218.87batch/s]

Training batches on cuda:  90%|████████▉ | 8717/9708 [00:42<00:04, 218.56batch/s]

Training batches on cuda:  90%|█████████ | 8740/9708 [00:42<00:04, 219.96batch/s]

Training batches on cuda:  90%|█████████ | 8763/9708 [00:42<00:04, 215.73batch/s]

Training batches on cuda:  91%|█████████ | 8786/9708 [00:42<00:04, 216.76batch/s]

Training batches on cuda:  91%|█████████ | 8808/9708 [00:42<00:04, 214.02batch/s]

Training batches on cuda:  91%|█████████ | 8830/9708 [00:42<00:04, 211.60batch/s]

Training batches on cuda:  91%|█████████ | 8852/9708 [00:42<00:04, 210.09batch/s]

Training batches on cuda:  91%|█████████▏| 8874/9708 [00:42<00:03, 210.70batch/s]

Training batches on cuda:  92%|█████████▏| 8896/9708 [00:43<00:03, 208.97batch/s]

Training batches on cuda:  92%|█████████▏| 8917/9708 [00:43<00:03, 202.78batch/s]

Training batches on cuda:  92%|█████████▏| 8938/9708 [00:43<00:03, 202.14batch/s]

Training batches on cuda:  92%|█████████▏| 8959/9708 [00:43<00:03, 198.36batch/s]

Training batches on cuda:  92%|█████████▏| 8979/9708 [00:43<00:03, 192.57batch/s]

Training batches on cuda:  93%|█████████▎| 8999/9708 [00:43<00:03, 188.37batch/s]

Training batches on cuda:  93%|█████████▎| 9020/9708 [00:43<00:03, 192.65batch/s]

Training batches on cuda:  93%|█████████▎| 9040/9708 [00:43<00:03, 193.06batch/s]

Training batches on cuda:  93%|█████████▎| 9060/9708 [00:43<00:03, 194.49batch/s]

Training batches on cuda:  94%|█████████▎| 9080/9708 [00:43<00:03, 194.74batch/s]

Training batches on cuda:  94%|█████████▍| 9104/9708 [00:44<00:02, 205.09batch/s]

Training batches on cuda:  94%|█████████▍| 9127/9708 [00:44<00:02, 210.06batch/s]

Training batches on cuda:  94%|█████████▍| 9152/9708 [00:44<00:02, 217.95batch/s]

Training batches on cuda:  95%|█████████▍| 9176/9708 [00:44<00:02, 223.60batch/s]

Training batches on cuda:  95%|█████████▍| 9200/9708 [00:44<00:02, 227.53batch/s]

Training batches on cuda:  95%|█████████▌| 9224/9708 [00:44<00:02, 230.99batch/s]

Training batches on cuda:  95%|█████████▌| 9249/9708 [00:44<00:01, 234.99batch/s]

Training batches on cuda:  96%|█████████▌| 9273/9708 [00:44<00:01, 236.13batch/s]

Training batches on cuda:  96%|█████████▌| 9297/9708 [00:44<00:01, 233.88batch/s]

Training batches on cuda:  96%|█████████▌| 9321/9708 [00:45<00:01, 232.89batch/s]

Training batches on cuda:  96%|█████████▋| 9345/9708 [00:45<00:01, 221.46batch/s]

Training batches on cuda:  96%|█████████▋| 9368/9708 [00:45<00:01, 216.15batch/s]

Training batches on cuda:  97%|█████████▋| 9390/9708 [00:45<00:01, 213.82batch/s]

Training batches on cuda:  97%|█████████▋| 9412/9708 [00:45<00:01, 211.14batch/s]

Training batches on cuda:  97%|█████████▋| 9434/9708 [00:45<00:01, 210.28batch/s]

Training batches on cuda:  97%|█████████▋| 9456/9708 [00:45<00:01, 209.54batch/s]

Training batches on cuda:  98%|█████████▊| 9477/9708 [00:45<00:01, 208.68batch/s]

Training batches on cuda:  98%|█████████▊| 9499/9708 [00:45<00:00, 209.93batch/s]

Training batches on cuda:  98%|█████████▊| 9521/9708 [00:45<00:00, 209.66batch/s]

Training batches on cuda:  98%|█████████▊| 9542/9708 [00:46<00:00, 209.19batch/s]

Training batches on cuda:  99%|█████████▊| 9564/9708 [00:46<00:00, 212.02batch/s]

Training batches on cuda:  99%|█████████▊| 9586/9708 [00:46<00:00, 213.17batch/s]

Training batches on cuda:  99%|█████████▉| 9608/9708 [00:46<00:00, 213.73batch/s]

Training batches on cuda:  99%|█████████▉| 9630/9708 [00:46<00:00, 212.84batch/s]

Training batches on cuda:  99%|█████████▉| 9652/9708 [00:46<00:00, 202.73batch/s]

Training batches on cuda: 100%|█████████▉| 9673/9708 [00:46<00:00, 203.86batch/s]

Training batches on cuda: 100%|█████████▉| 9695/9708 [00:46<00:00, 207.37batch/s]

                                                                                 
Training epochs on cuda:   0%|          | 0/10 [00:47<?, ?epoch/s, loss=0.0416, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:47<07:04, 47.14s/epoch, loss=0.0416, prev_loss=nan]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 16/9708 [00:00<01:02, 154.09batch/s]

Training batches on cuda:   0%|          | 38/9708 [00:00<00:57, 167.59batch/s]

Training batches on cuda:   1%|          | 59/9708 [00:00<00:54, 178.13batch/s]

Training batches on cuda:   1%|          | 80/9708 [00:00<00:51, 185.74batch/s]

Training batches on cuda:   1%|          | 102/9708 [00:00<00:49, 193.06batch/s]

Training batches on cuda:   1%|▏         | 124/9708 [00:00<00:48, 197.87batch/s]

Training batches on cuda:   2%|▏         | 147/9708 [00:00<00:46, 204.58batch/s]

Training batches on cuda:   2%|▏         | 168/9708 [00:00<00:46, 205.46batch/s]

Training batches on cuda:   2%|▏         | 189/9708 [00:00<00:46, 206.20batch/s]

Training batches on cuda:   2%|▏         | 210/9708 [00:01<00:45, 206.94batch/s]

Training batches on cuda:   2%|▏         | 231/9708 [00:01<00:45, 206.69batch/s]

Training batches on cuda:   3%|▎         | 252/9708 [00:01<00:45, 205.70batch/s]

Training batches on cuda:   3%|▎         | 273/9708 [00:01<00:45, 205.78batch/s]

Training batches on cuda:   3%|▎         | 294/9708 [00:01<00:45, 206.05batch/s]

Training batches on cuda:   3%|▎         | 315/9708 [00:01<00:45, 204.46batch/s]

Training batches on cuda:   3%|▎         | 338/9708 [00:01<00:44, 210.70batch/s]

Training batches on cuda:   4%|▎         | 362/9708 [00:01<00:43, 216.66batch/s]

Training batches on cuda:   4%|▍         | 384/9708 [00:01<00:43, 213.54batch/s]

Training batches on cuda:   4%|▍         | 408/9708 [00:01<00:42, 220.50batch/s]

Training batches on cuda:   4%|▍         | 432/9708 [00:02<00:41, 224.88batch/s]

Training batches on cuda:   5%|▍         | 455/9708 [00:02<00:44, 206.09batch/s]

Training batches on cuda:   5%|▍         | 478/9708 [00:02<00:43, 211.69batch/s]

Training batches on cuda:   5%|▌         | 500/9708 [00:02<00:43, 212.12batch/s]

Training batches on cuda:   5%|▌         | 522/9708 [00:02<00:42, 213.97batch/s]

Training batches on cuda:   6%|▌         | 545/9708 [00:02<00:41, 218.26batch/s]

Training batches on cuda:   6%|▌         | 569/9708 [00:02<00:41, 222.07batch/s]

Training batches on cuda:   6%|▌         | 593/9708 [00:02<00:40, 224.40batch/s]

Training batches on cuda:   6%|▋         | 616/9708 [00:02<00:40, 224.58batch/s]

Training batches on cuda:   7%|▋         | 639/9708 [00:03<00:40, 222.52batch/s]

Training batches on cuda:   7%|▋         | 662/9708 [00:03<00:41, 219.17batch/s]

Training batches on cuda:   7%|▋         | 684/9708 [00:03<00:41, 217.59batch/s]

Training batches on cuda:   7%|▋         | 706/9708 [00:03<00:41, 216.67batch/s]

Training batches on cuda:   7%|▋         | 728/9708 [00:03<00:42, 212.65batch/s]

Training batches on cuda:   8%|▊         | 750/9708 [00:03<00:42, 210.69batch/s]

Training batches on cuda:   8%|▊         | 772/9708 [00:03<00:43, 207.53batch/s]

Training batches on cuda:   8%|▊         | 793/9708 [00:03<00:43, 205.58batch/s]

Training batches on cuda:   8%|▊         | 814/9708 [00:03<00:43, 202.86batch/s]

Training batches on cuda:   9%|▊         | 835/9708 [00:03<00:43, 203.81batch/s]

Training batches on cuda:   9%|▉         | 857/9708 [00:04<00:42, 206.38batch/s]

Training batches on cuda:   9%|▉         | 879/9708 [00:04<00:42, 207.87batch/s]

Training batches on cuda:   9%|▉         | 900/9708 [00:04<00:43, 204.54batch/s]

Training batches on cuda:   9%|▉         | 922/9708 [00:04<00:42, 207.55batch/s]

Training batches on cuda:  10%|▉         | 946/9708 [00:04<00:40, 215.64batch/s]

Training batches on cuda:  10%|▉         | 969/9708 [00:04<00:40, 217.59batch/s]

Training batches on cuda:  10%|█         | 992/9708 [00:04<00:39, 220.86batch/s]

Training batches on cuda:  10%|█         | 1016/9708 [00:04<00:38, 224.03batch/s]

Training batches on cuda:  11%|█         | 1039/9708 [00:04<00:38, 224.90batch/s]

Training batches on cuda:  11%|█         | 1062/9708 [00:04<00:38, 224.14batch/s]

Training batches on cuda:  11%|█         | 1085/9708 [00:05<00:38, 224.88batch/s]

Training batches on cuda:  11%|█▏        | 1109/9708 [00:05<00:37, 226.60batch/s]

Training batches on cuda:  12%|█▏        | 1133/9708 [00:05<00:37, 230.41batch/s]

Training batches on cuda:  12%|█▏        | 1157/9708 [00:05<00:36, 231.39batch/s]

Training batches on cuda:  12%|█▏        | 1181/9708 [00:05<00:36, 231.43batch/s]

Training batches on cuda:  12%|█▏        | 1205/9708 [00:05<00:36, 231.54batch/s]

Training batches on cuda:  13%|█▎        | 1229/9708 [00:05<00:37, 228.85batch/s]

Training batches on cuda:  13%|█▎        | 1253/9708 [00:05<00:36, 229.54batch/s]

Training batches on cuda:  13%|█▎        | 1277/9708 [00:05<00:36, 229.73batch/s]

Training batches on cuda:  13%|█▎        | 1300/9708 [00:06<00:36, 229.23batch/s]

Training batches on cuda:  14%|█▎        | 1324/9708 [00:06<00:36, 230.73batch/s]

Training batches on cuda:  14%|█▍        | 1348/9708 [00:06<00:36, 230.95batch/s]

Training batches on cuda:  14%|█▍        | 1372/9708 [00:06<00:36, 229.82batch/s]

Training batches on cuda:  14%|█▍        | 1395/9708 [00:06<00:36, 227.09batch/s]

Training batches on cuda:  15%|█▍        | 1419/9708 [00:06<00:36, 228.18batch/s]

Training batches on cuda:  15%|█▍        | 1442/9708 [00:06<00:36, 227.84batch/s]

Training batches on cuda:  15%|█▌        | 1465/9708 [00:06<00:36, 227.29batch/s]

Training batches on cuda:  15%|█▌        | 1489/9708 [00:06<00:35, 228.39batch/s]

Training batches on cuda:  16%|█▌        | 1512/9708 [00:06<00:35, 228.47batch/s]

Training batches on cuda:  16%|█▌        | 1535/9708 [00:07<00:35, 228.42batch/s]

Training batches on cuda:  16%|█▌        | 1558/9708 [00:07<00:36, 223.57batch/s]

Training batches on cuda:  16%|█▋        | 1581/9708 [00:07<00:37, 216.54batch/s]

Training batches on cuda:  17%|█▋        | 1603/9708 [00:07<00:38, 210.94batch/s]

Training batches on cuda:  17%|█▋        | 1625/9708 [00:07<00:38, 210.94batch/s]

Training batches on cuda:  17%|█▋        | 1647/9708 [00:07<00:37, 212.39batch/s]

Training batches on cuda:  17%|█▋        | 1669/9708 [00:07<00:37, 213.25batch/s]

Training batches on cuda:  17%|█▋        | 1692/9708 [00:07<00:37, 215.75batch/s]

Training batches on cuda:  18%|█▊        | 1714/9708 [00:07<00:37, 214.05batch/s]

Training batches on cuda:  18%|█▊        | 1738/9708 [00:08<00:36, 219.29batch/s]

Training batches on cuda:  18%|█▊        | 1760/9708 [00:08<00:36, 216.10batch/s]

Training batches on cuda:  18%|█▊        | 1784/9708 [00:08<00:35, 222.61batch/s]

Training batches on cuda:  19%|█▊        | 1809/9708 [00:08<00:34, 229.88batch/s]

Training batches on cuda:  19%|█▉        | 1835/9708 [00:08<00:33, 235.78batch/s]

Training batches on cuda:  19%|█▉        | 1860/9708 [00:08<00:33, 237.73batch/s]

Training batches on cuda:  19%|█▉        | 1884/9708 [00:08<00:33, 232.82batch/s]

Training batches on cuda:  20%|█▉        | 1908/9708 [00:08<00:34, 227.31batch/s]

Training batches on cuda:  20%|█▉        | 1931/9708 [00:08<00:34, 227.04batch/s]

Training batches on cuda:  20%|██        | 1954/9708 [00:08<00:34, 225.32batch/s]

Training batches on cuda:  20%|██        | 1977/9708 [00:09<00:34, 225.67batch/s]

Training batches on cuda:  21%|██        | 2001/9708 [00:09<00:33, 227.33batch/s]

Training batches on cuda:  21%|██        | 2024/9708 [00:09<00:34, 223.71batch/s]

Training batches on cuda:  21%|██        | 2047/9708 [00:09<00:36, 210.56batch/s]

Training batches on cuda:  21%|██▏       | 2069/9708 [00:09<00:36, 211.54batch/s]

Training batches on cuda:  22%|██▏       | 2091/9708 [00:09<00:36, 210.32batch/s]

Training batches on cuda:  22%|██▏       | 2113/9708 [00:09<00:36, 210.17batch/s]

Training batches on cuda:  22%|██▏       | 2137/9708 [00:09<00:35, 215.67batch/s]

Training batches on cuda:  22%|██▏       | 2159/9708 [00:09<00:35, 214.68batch/s]

Training batches on cuda:  22%|██▏       | 2181/9708 [00:10<00:35, 212.22batch/s]

Training batches on cuda:  23%|██▎       | 2204/9708 [00:10<00:34, 217.07batch/s]

Training batches on cuda:  23%|██▎       | 2226/9708 [00:10<00:34, 216.08batch/s]

Training batches on cuda:  23%|██▎       | 2248/9708 [00:10<00:37, 196.90batch/s]

Training batches on cuda:  23%|██▎       | 2270/9708 [00:10<00:36, 202.62batch/s]

Training batches on cuda:  24%|██▎       | 2291/9708 [00:10<00:36, 204.53batch/s]

Training batches on cuda:  24%|██▍       | 2313/9708 [00:10<00:35, 206.82batch/s]

Training batches on cuda:  24%|██▍       | 2335/9708 [00:10<00:35, 208.03batch/s]

Training batches on cuda:  24%|██▍       | 2356/9708 [00:10<00:35, 208.00batch/s]

Training batches on cuda:  24%|██▍       | 2377/9708 [00:10<00:36, 202.58batch/s]

Training batches on cuda:  25%|██▍       | 2398/9708 [00:11<00:37, 195.47batch/s]

Training batches on cuda:  25%|██▍       | 2418/9708 [00:11<00:38, 188.29batch/s]

Training batches on cuda:  25%|██▌       | 2438/9708 [00:11<00:38, 190.20batch/s]

Training batches on cuda:  25%|██▌       | 2460/9708 [00:11<00:36, 196.63batch/s]

Training batches on cuda:  26%|██▌       | 2481/9708 [00:11<00:36, 199.47batch/s]

Training batches on cuda:  26%|██▌       | 2502/9708 [00:11<00:35, 200.25batch/s]

Training batches on cuda:  26%|██▌       | 2523/9708 [00:11<00:35, 202.36batch/s]

Training batches on cuda:  26%|██▌       | 2544/9708 [00:11<00:35, 201.38batch/s]

Training batches on cuda:  26%|██▋       | 2565/9708 [00:11<00:35, 202.29batch/s]

Training batches on cuda:  27%|██▋       | 2586/9708 [00:12<00:35, 203.47batch/s]

Training batches on cuda:  27%|██▋       | 2607/9708 [00:12<00:34, 203.88batch/s]

Training batches on cuda:  27%|██▋       | 2628/9708 [00:12<00:34, 203.01batch/s]

Training batches on cuda:  27%|██▋       | 2649/9708 [00:12<00:34, 202.23batch/s]

Training batches on cuda:  28%|██▊       | 2670/9708 [00:12<00:35, 200.45batch/s]

Training batches on cuda:  28%|██▊       | 2692/9708 [00:12<00:34, 205.17batch/s]

Training batches on cuda:  28%|██▊       | 2714/9708 [00:12<00:33, 209.01batch/s]

Training batches on cuda:  28%|██▊       | 2735/9708 [00:12<00:33, 208.77batch/s]

Training batches on cuda:  28%|██▊       | 2759/9708 [00:12<00:32, 214.97batch/s]

Training batches on cuda:  29%|██▊       | 2781/9708 [00:12<00:34, 201.76batch/s]

Training batches on cuda:  29%|██▉       | 2802/9708 [00:13<00:34, 197.63batch/s]

Training batches on cuda:  29%|██▉       | 2822/9708 [00:13<00:35, 192.27batch/s]

Training batches on cuda:  29%|██▉       | 2844/9708 [00:13<00:34, 198.14batch/s]

Training batches on cuda:  30%|██▉       | 2864/9708 [00:13<00:35, 193.49batch/s]

Training batches on cuda:  30%|██▉       | 2884/9708 [00:13<00:34, 195.24batch/s]

Training batches on cuda:  30%|██▉       | 2906/9708 [00:13<00:33, 200.34batch/s]

Training batches on cuda:  30%|███       | 2927/9708 [00:13<00:33, 200.96batch/s]

Training batches on cuda:  30%|███       | 2950/9708 [00:13<00:32, 206.38batch/s]

Training batches on cuda:  31%|███       | 2973/9708 [00:13<00:31, 210.81batch/s]

Training batches on cuda:  31%|███       | 2996/9708 [00:14<00:31, 214.31batch/s]

Training batches on cuda:  31%|███       | 3018/9708 [00:14<00:31, 213.84batch/s]

Training batches on cuda:  31%|███▏      | 3041/9708 [00:14<00:30, 217.15batch/s]

Training batches on cuda:  32%|███▏      | 3063/9708 [00:14<00:30, 214.58batch/s]

Training batches on cuda:  32%|███▏      | 3085/9708 [00:14<00:33, 200.62batch/s]

Training batches on cuda:  32%|███▏      | 3106/9708 [00:14<00:32, 202.79batch/s]

Training batches on cuda:  32%|███▏      | 3128/9708 [00:14<00:31, 207.40batch/s]

Training batches on cuda:  32%|███▏      | 3149/9708 [00:14<00:32, 203.96batch/s]

Training batches on cuda:  33%|███▎      | 3170/9708 [00:14<00:33, 197.58batch/s]

Training batches on cuda:  33%|███▎      | 3190/9708 [00:14<00:33, 195.93batch/s]

Training batches on cuda:  33%|███▎      | 3211/9708 [00:15<00:32, 199.48batch/s]

Training batches on cuda:  33%|███▎      | 3232/9708 [00:15<00:32, 197.49batch/s]

Training batches on cuda:  33%|███▎      | 3252/9708 [00:15<00:34, 186.01batch/s]

Training batches on cuda:  34%|███▎      | 3272/9708 [00:15<00:33, 189.59batch/s]

Training batches on cuda:  34%|███▍      | 3293/9708 [00:15<00:33, 193.81batch/s]

Training batches on cuda:  34%|███▍      | 3313/9708 [00:15<00:32, 194.63batch/s]

Training batches on cuda:  34%|███▍      | 3333/9708 [00:15<00:35, 180.73batch/s]

Training batches on cuda:  35%|███▍      | 3355/9708 [00:15<00:33, 189.57batch/s]

Training batches on cuda:  35%|███▍      | 3378/9708 [00:15<00:31, 198.20batch/s]

Training batches on cuda:  35%|███▌      | 3399/9708 [00:16<00:31, 199.57batch/s]

Training batches on cuda:  35%|███▌      | 3420/9708 [00:16<00:31, 197.29batch/s]

Training batches on cuda:  35%|███▌      | 3440/9708 [00:16<00:33, 187.33batch/s]

Training batches on cuda:  36%|███▌      | 3460/9708 [00:16<00:33, 189.05batch/s]

Training batches on cuda:  36%|███▌      | 3481/9708 [00:16<00:32, 192.82batch/s]

Training batches on cuda:  36%|███▌      | 3502/9708 [00:16<00:31, 195.88batch/s]

Training batches on cuda:  36%|███▋      | 3523/9708 [00:16<00:31, 199.01batch/s]

Training batches on cuda:  37%|███▋      | 3544/9708 [00:16<00:30, 201.21batch/s]

Training batches on cuda:  37%|███▋      | 3565/9708 [00:16<00:30, 199.61batch/s]

Training batches on cuda:  37%|███▋      | 3586/9708 [00:17<00:30, 199.43batch/s]

Training batches on cuda:  37%|███▋      | 3606/9708 [00:17<00:30, 196.88batch/s]

Training batches on cuda:  37%|███▋      | 3626/9708 [00:17<00:34, 177.59batch/s]

Training batches on cuda:  38%|███▊      | 3648/9708 [00:17<00:32, 187.53batch/s]

Training batches on cuda:  38%|███▊      | 3671/9708 [00:17<00:30, 198.36batch/s]

Training batches on cuda:  38%|███▊      | 3693/9708 [00:17<00:29, 203.73batch/s]

Training batches on cuda:  38%|███▊      | 3714/9708 [00:17<00:29, 203.29batch/s]

Training batches on cuda:  38%|███▊      | 3735/9708 [00:17<00:29, 202.52batch/s]

Training batches on cuda:  39%|███▊      | 3756/9708 [00:17<00:29, 202.39batch/s]

Training batches on cuda:  39%|███▉      | 3777/9708 [00:17<00:29, 197.84batch/s]

Training batches on cuda:  39%|███▉      | 3798/9708 [00:18<00:29, 201.22batch/s]

Training batches on cuda:  39%|███▉      | 3819/9708 [00:18<00:29, 202.99batch/s]

Training batches on cuda:  40%|███▉      | 3841/9708 [00:18<00:28, 205.51batch/s]

Training batches on cuda:  40%|███▉      | 3863/9708 [00:18<00:27, 208.77batch/s]

Training batches on cuda:  40%|████      | 3885/9708 [00:18<00:27, 209.47batch/s]

Training batches on cuda:  40%|████      | 3906/9708 [00:18<00:27, 207.58batch/s]

Training batches on cuda:  40%|████      | 3929/9708 [00:18<00:27, 212.38batch/s]

Training batches on cuda:  41%|████      | 3951/9708 [00:18<00:27, 211.88batch/s]

Training batches on cuda:  41%|████      | 3973/9708 [00:18<00:27, 211.25batch/s]

Training batches on cuda:  41%|████      | 3995/9708 [00:19<00:27, 210.85batch/s]

Training batches on cuda:  41%|████▏     | 4017/9708 [00:19<00:27, 209.00batch/s]

Training batches on cuda:  42%|████▏     | 4038/9708 [00:19<00:27, 206.42batch/s]

Training batches on cuda:  42%|████▏     | 4059/9708 [00:19<00:28, 199.88batch/s]

Training batches on cuda:  42%|████▏     | 4080/9708 [00:19<00:28, 194.82batch/s]

Training batches on cuda:  42%|████▏     | 4101/9708 [00:19<00:28, 198.69batch/s]

Training batches on cuda:  42%|████▏     | 4121/9708 [00:19<00:29, 191.38batch/s]

Training batches on cuda:  43%|████▎     | 4142/9708 [00:19<00:28, 196.55batch/s]

Training batches on cuda:  43%|████▎     | 4162/9708 [00:19<00:28, 197.30batch/s]

Training batches on cuda:  43%|████▎     | 4185/9708 [00:19<00:27, 204.35batch/s]

Training batches on cuda:  43%|████▎     | 4206/9708 [00:20<00:26, 204.72batch/s]

Training batches on cuda:  44%|████▎     | 4228/9708 [00:20<00:26, 207.70batch/s]

Training batches on cuda:  44%|████▍     | 4249/9708 [00:20<00:26, 207.28batch/s]

Training batches on cuda:  44%|████▍     | 4270/9708 [00:20<00:26, 204.03batch/s]

Training batches on cuda:  44%|████▍     | 4291/9708 [00:20<00:26, 204.40batch/s]

Training batches on cuda:  44%|████▍     | 4313/9708 [00:20<00:26, 206.16batch/s]

Training batches on cuda:  45%|████▍     | 4334/9708 [00:20<00:26, 206.52batch/s]

Training batches on cuda:  45%|████▍     | 4355/9708 [00:20<00:26, 204.18batch/s]

Training batches on cuda:  45%|████▌     | 4376/9708 [00:20<00:25, 205.57batch/s]

Training batches on cuda:  45%|████▌     | 4397/9708 [00:21<00:26, 203.79batch/s]

Training batches on cuda:  46%|████▌     | 4418/9708 [00:21<00:25, 204.16batch/s]

Training batches on cuda:  46%|████▌     | 4441/9708 [00:21<00:25, 210.33batch/s]

Training batches on cuda:  46%|████▌     | 4465/9708 [00:21<00:24, 217.73batch/s]

Training batches on cuda:  46%|████▋     | 4490/9708 [00:21<00:23, 225.67batch/s]

Training batches on cuda:  46%|████▋     | 4513/9708 [00:21<00:23, 223.55batch/s]

Training batches on cuda:  47%|████▋     | 4536/9708 [00:21<00:24, 214.10batch/s]

Training batches on cuda:  47%|████▋     | 4558/9708 [00:21<00:24, 213.08batch/s]

Training batches on cuda:  47%|████▋     | 4580/9708 [00:21<00:24, 206.76batch/s]

Training batches on cuda:  47%|████▋     | 4602/9708 [00:21<00:24, 208.66batch/s]

Training batches on cuda:  48%|████▊     | 4625/9708 [00:22<00:23, 212.77batch/s]

Training batches on cuda:  48%|████▊     | 4647/9708 [00:22<00:23, 214.75batch/s]

Training batches on cuda:  48%|████▊     | 4669/9708 [00:22<00:26, 193.25batch/s]

Training batches on cuda:  48%|████▊     | 4691/9708 [00:22<00:25, 198.77batch/s]

Training batches on cuda:  49%|████▊     | 4712/9708 [00:22<00:24, 201.88batch/s]

Training batches on cuda:  49%|████▉     | 4733/9708 [00:22<00:24, 202.45batch/s]

Training batches on cuda:  49%|████▉     | 4754/9708 [00:22<00:24, 204.65batch/s]

Training batches on cuda:  49%|████▉     | 4775/9708 [00:22<00:24, 204.36batch/s]

Training batches on cuda:  49%|████▉     | 4796/9708 [00:22<00:24, 204.46batch/s]

Training batches on cuda:  50%|████▉     | 4818/9708 [00:23<00:23, 206.42batch/s]

Training batches on cuda:  50%|████▉     | 4840/9708 [00:23<00:23, 208.73batch/s]

Training batches on cuda:  50%|█████     | 4861/9708 [00:23<00:23, 208.62batch/s]

Training batches on cuda:  50%|█████     | 4883/9708 [00:23<00:22, 209.85batch/s]

Training batches on cuda:  51%|█████     | 4905/9708 [00:23<00:22, 208.90batch/s]

Training batches on cuda:  51%|█████     | 4928/9708 [00:23<00:22, 212.34batch/s]

Training batches on cuda:  51%|█████     | 4950/9708 [00:23<00:22, 209.33batch/s]

Training batches on cuda:  51%|█████     | 4971/9708 [00:23<00:22, 209.36batch/s]

Training batches on cuda:  51%|█████▏    | 4993/9708 [00:23<00:22, 209.72batch/s]

Training batches on cuda:  52%|█████▏    | 5014/9708 [00:23<00:22, 207.15batch/s]

Training batches on cuda:  52%|█████▏    | 5035/9708 [00:24<00:22, 204.75batch/s]

Training batches on cuda:  52%|█████▏    | 5056/9708 [00:24<00:22, 205.58batch/s]

Training batches on cuda:  52%|█████▏    | 5077/9708 [00:24<00:22, 206.64batch/s]

Training batches on cuda:  53%|█████▎    | 5098/9708 [00:24<00:22, 205.52batch/s]

Training batches on cuda:  53%|█████▎    | 5119/9708 [00:24<00:24, 190.53batch/s]

Training batches on cuda:  53%|█████▎    | 5139/9708 [00:24<00:24, 186.75batch/s]

Training batches on cuda:  53%|█████▎    | 5160/9708 [00:24<00:23, 190.92batch/s]

Training batches on cuda:  53%|█████▎    | 5181/9708 [00:24<00:23, 194.55batch/s]

Training batches on cuda:  54%|█████▎    | 5201/9708 [00:24<00:22, 196.05batch/s]

Training batches on cuda:  54%|█████▍    | 5222/9708 [00:25<00:22, 198.92batch/s]

Training batches on cuda:  54%|█████▍    | 5243/9708 [00:25<00:22, 199.72batch/s]

Training batches on cuda:  54%|█████▍    | 5264/9708 [00:25<00:22, 199.36batch/s]

Training batches on cuda:  54%|█████▍    | 5284/9708 [00:25<00:22, 197.85batch/s]

Training batches on cuda:  55%|█████▍    | 5305/9708 [00:25<00:21, 200.81batch/s]

Training batches on cuda:  55%|█████▍    | 5326/9708 [00:25<00:22, 199.02batch/s]

Training batches on cuda:  55%|█████▌    | 5348/9708 [00:25<00:21, 202.81batch/s]

Training batches on cuda:  55%|█████▌    | 5369/9708 [00:25<00:21, 202.71batch/s]

Training batches on cuda:  56%|█████▌    | 5390/9708 [00:25<00:21, 204.35batch/s]

Training batches on cuda:  56%|█████▌    | 5411/9708 [00:25<00:21, 199.26batch/s]

Training batches on cuda:  56%|█████▌    | 5433/9708 [00:26<00:20, 204.29batch/s]

Training batches on cuda:  56%|█████▌    | 5455/9708 [00:26<00:20, 207.68batch/s]

Training batches on cuda:  56%|█████▋    | 5478/9708 [00:26<00:19, 212.32batch/s]

Training batches on cuda:  57%|█████▋    | 5501/9708 [00:26<00:19, 215.98batch/s]

Training batches on cuda:  57%|█████▋    | 5524/9708 [00:26<00:19, 218.12batch/s]

Training batches on cuda:  57%|█████▋    | 5546/9708 [00:26<00:19, 210.69batch/s]

Training batches on cuda:  57%|█████▋    | 5569/9708 [00:26<00:19, 213.97batch/s]

Training batches on cuda:  58%|█████▊    | 5592/9708 [00:26<00:19, 215.88batch/s]

Training batches on cuda:  58%|█████▊    | 5615/9708 [00:26<00:18, 218.22batch/s]

Training batches on cuda:  58%|█████▊    | 5638/9708 [00:26<00:18, 219.36batch/s]

Training batches on cuda:  58%|█████▊    | 5661/9708 [00:27<00:18, 221.13batch/s]

Training batches on cuda:  59%|█████▊    | 5684/9708 [00:27<00:18, 222.73batch/s]

Training batches on cuda:  59%|█████▉    | 5707/9708 [00:27<00:20, 199.99batch/s]

Training batches on cuda:  59%|█████▉    | 5730/9708 [00:27<00:19, 207.09batch/s]

Training batches on cuda:  59%|█████▉    | 5754/9708 [00:27<00:18, 213.85batch/s]

Training batches on cuda:  60%|█████▉    | 5777/9708 [00:27<00:18, 217.94batch/s]

Training batches on cuda:  60%|█████▉    | 5800/9708 [00:27<00:17, 218.85batch/s]

Training batches on cuda:  60%|█████▉    | 5823/9708 [00:27<00:17, 220.16batch/s]

Training batches on cuda:  60%|██████    | 5846/9708 [00:27<00:17, 222.12batch/s]

Training batches on cuda:  60%|██████    | 5869/9708 [00:28<00:17, 220.34batch/s]

Training batches on cuda:  61%|██████    | 5892/9708 [00:28<00:17, 219.73batch/s]

Training batches on cuda:  61%|██████    | 5915/9708 [00:28<00:17, 220.16batch/s]

Training batches on cuda:  61%|██████    | 5938/9708 [00:28<00:17, 218.80batch/s]

Training batches on cuda:  61%|██████▏   | 5960/9708 [00:28<00:17, 218.70batch/s]

Training batches on cuda:  62%|██████▏   | 5984/9708 [00:28<00:16, 224.37batch/s]

Training batches on cuda:  62%|██████▏   | 6010/9708 [00:28<00:15, 231.82batch/s]

Training batches on cuda:  62%|██████▏   | 6035/9708 [00:28<00:15, 236.08batch/s]

Training batches on cuda:  62%|██████▏   | 6060/9708 [00:28<00:15, 239.47batch/s]

Training batches on cuda:  63%|██████▎   | 6085/9708 [00:29<00:15, 227.54batch/s]

Training batches on cuda:  63%|██████▎   | 6108/9708 [00:29<00:17, 208.26batch/s]

Training batches on cuda:  63%|██████▎   | 6130/9708 [00:29<00:17, 206.29batch/s]

Training batches on cuda:  63%|██████▎   | 6151/9708 [00:29<00:17, 199.64batch/s]

Training batches on cuda:  64%|██████▎   | 6173/9708 [00:29<00:17, 203.99batch/s]

Training batches on cuda:  64%|██████▍   | 6196/9708 [00:29<00:16, 209.58batch/s]

Training batches on cuda:  64%|██████▍   | 6218/9708 [00:29<00:17, 202.19batch/s]

Training batches on cuda:  64%|██████▍   | 6240/9708 [00:29<00:16, 205.73batch/s]

Training batches on cuda:  64%|██████▍   | 6261/9708 [00:29<00:16, 206.74batch/s]

Training batches on cuda:  65%|██████▍   | 6282/9708 [00:29<00:16, 206.65batch/s]

Training batches on cuda:  65%|██████▍   | 6304/9708 [00:30<00:16, 207.90batch/s]

Training batches on cuda:  65%|██████▌   | 6325/9708 [00:30<00:16, 199.04batch/s]

Training batches on cuda:  65%|██████▌   | 6348/9708 [00:30<00:16, 207.03batch/s]

Training batches on cuda:  66%|██████▌   | 6369/9708 [00:30<00:16, 207.08batch/s]

Training batches on cuda:  66%|██████▌   | 6390/9708 [00:30<00:16, 205.84batch/s]

Training batches on cuda:  66%|██████▌   | 6411/9708 [00:30<00:15, 206.88batch/s]

Training batches on cuda:  66%|██████▋   | 6432/9708 [00:30<00:15, 207.73batch/s]

Training batches on cuda:  66%|██████▋   | 6453/9708 [00:30<00:15, 203.98batch/s]

Training batches on cuda:  67%|██████▋   | 6474/9708 [00:30<00:17, 189.16batch/s]

Training batches on cuda:  67%|██████▋   | 6494/9708 [00:31<00:16, 189.42batch/s]

Training batches on cuda:  67%|██████▋   | 6514/9708 [00:31<00:17, 184.20batch/s]

Training batches on cuda:  67%|██████▋   | 6533/9708 [00:31<00:17, 185.23batch/s]

Training batches on cuda:  68%|██████▊   | 6555/9708 [00:31<00:16, 192.38batch/s]

Training batches on cuda:  68%|██████▊   | 6575/9708 [00:31<00:16, 193.07batch/s]

Training batches on cuda:  68%|██████▊   | 6598/9708 [00:31<00:15, 202.19batch/s]

Training batches on cuda:  68%|██████▊   | 6620/9708 [00:31<00:15, 205.53batch/s]

Training batches on cuda:  68%|██████▊   | 6643/9708 [00:31<00:14, 211.00batch/s]

Training batches on cuda:  69%|██████▊   | 6665/9708 [00:31<00:14, 211.99batch/s]

Training batches on cuda:  69%|██████▉   | 6687/9708 [00:31<00:14, 213.16batch/s]

Training batches on cuda:  69%|██████▉   | 6709/9708 [00:32<00:15, 199.63batch/s]

Training batches on cuda:  69%|██████▉   | 6730/9708 [00:32<00:14, 199.60batch/s]

Training batches on cuda:  70%|██████▉   | 6751/9708 [00:32<00:15, 186.12batch/s]

Training batches on cuda:  70%|██████▉   | 6774/9708 [00:32<00:14, 197.08batch/s]

Training batches on cuda:  70%|███████   | 6799/9708 [00:32<00:13, 208.51batch/s]

Training batches on cuda:  70%|███████   | 6823/9708 [00:32<00:13, 215.20batch/s]

Training batches on cuda:  71%|███████   | 6847/9708 [00:32<00:13, 218.79batch/s]

Training batches on cuda:  71%|███████   | 6870/9708 [00:32<00:13, 217.57batch/s]

Training batches on cuda:  71%|███████   | 6894/9708 [00:32<00:12, 223.31batch/s]

Training batches on cuda:  71%|███████▏  | 6917/9708 [00:33<00:12, 221.24batch/s]

Training batches on cuda:  71%|███████▏  | 6940/9708 [00:33<00:13, 211.74batch/s]

Training batches on cuda:  72%|███████▏  | 6962/9708 [00:33<00:13, 203.29batch/s]

Training batches on cuda:  72%|███████▏  | 6983/9708 [00:33<00:13, 202.55batch/s]

Training batches on cuda:  72%|███████▏  | 7004/9708 [00:33<00:13, 201.67batch/s]

Training batches on cuda:  72%|███████▏  | 7025/9708 [00:33<00:13, 198.78batch/s]

Training batches on cuda:  73%|███████▎  | 7045/9708 [00:33<00:13, 198.98batch/s]

Training batches on cuda:  73%|███████▎  | 7065/9708 [00:33<00:13, 193.89batch/s]

Training batches on cuda:  73%|███████▎  | 7086/9708 [00:33<00:13, 195.92batch/s]

Training batches on cuda:  73%|███████▎  | 7109/9708 [00:34<00:12, 202.76batch/s]

Training batches on cuda:  73%|███████▎  | 7130/9708 [00:34<00:12, 202.08batch/s]

Training batches on cuda:  74%|███████▎  | 7153/9708 [00:34<00:12, 207.23batch/s]

Training batches on cuda:  74%|███████▍  | 7176/9708 [00:34<00:11, 211.54batch/s]

Training batches on cuda:  74%|███████▍  | 7199/9708 [00:34<00:11, 214.70batch/s]

Training batches on cuda:  74%|███████▍  | 7222/9708 [00:34<00:11, 216.49batch/s]

Training batches on cuda:  75%|███████▍  | 7245/9708 [00:34<00:11, 219.00batch/s]

Training batches on cuda:  75%|███████▍  | 7267/9708 [00:34<00:11, 218.70batch/s]

Training batches on cuda:  75%|███████▌  | 7289/9708 [00:34<00:11, 214.18batch/s]

Training batches on cuda:  75%|███████▌  | 7311/9708 [00:34<00:11, 215.19batch/s]

Training batches on cuda:  76%|███████▌  | 7334/9708 [00:35<00:10, 218.07batch/s]

Training batches on cuda:  76%|███████▌  | 7357/9708 [00:35<00:10, 221.31batch/s]

Training batches on cuda:  76%|███████▌  | 7381/9708 [00:35<00:10, 225.27batch/s]

Training batches on cuda:  76%|███████▋  | 7409/9708 [00:35<00:09, 237.16batch/s]

Training batches on cuda:  77%|███████▋  | 7435/9708 [00:35<00:09, 243.29batch/s]

Training batches on cuda:  77%|███████▋  | 7460/9708 [00:35<00:09, 237.43batch/s]

Training batches on cuda:  77%|███████▋  | 7484/9708 [00:35<00:09, 232.73batch/s]

Training batches on cuda:  77%|███████▋  | 7508/9708 [00:35<00:09, 232.03batch/s]

Training batches on cuda:  78%|███████▊  | 7532/9708 [00:35<00:09, 228.01batch/s]

Training batches on cuda:  78%|███████▊  | 7555/9708 [00:36<00:09, 227.34batch/s]

Training batches on cuda:  78%|███████▊  | 7578/9708 [00:36<00:09, 214.03batch/s]

Training batches on cuda:  78%|███████▊  | 7600/9708 [00:36<00:10, 200.36batch/s]

Training batches on cuda:  79%|███████▊  | 7621/9708 [00:36<00:10, 191.81batch/s]

Training batches on cuda:  79%|███████▊  | 7643/9708 [00:36<00:10, 199.47batch/s]

Training batches on cuda:  79%|███████▉  | 7666/9708 [00:36<00:09, 205.93batch/s]

Training batches on cuda:  79%|███████▉  | 7688/9708 [00:36<00:09, 209.46batch/s]

Training batches on cuda:  79%|███████▉  | 7710/9708 [00:36<00:09, 210.51batch/s]

Training batches on cuda:  80%|███████▉  | 7733/9708 [00:36<00:09, 214.12batch/s]

Training batches on cuda:  80%|███████▉  | 7755/9708 [00:37<00:09, 214.65batch/s]

Training batches on cuda:  80%|████████  | 7777/9708 [00:37<00:09, 197.13batch/s]

Training batches on cuda:  80%|████████  | 7799/9708 [00:37<00:09, 203.32batch/s]

Training batches on cuda:  81%|████████  | 7820/9708 [00:37<00:10, 185.89batch/s]

Training batches on cuda:  81%|████████  | 7840/9708 [00:37<00:10, 183.01batch/s]

Training batches on cuda:  81%|████████  | 7860/9708 [00:37<00:09, 186.12batch/s]

Training batches on cuda:  81%|████████  | 7882/9708 [00:37<00:09, 193.63batch/s]

Training batches on cuda:  81%|████████▏ | 7906/9708 [00:37<00:08, 203.69batch/s]

Training batches on cuda:  82%|████████▏ | 7930/9708 [00:37<00:08, 212.24batch/s]

Training batches on cuda:  82%|████████▏ | 7954/9708 [00:38<00:08, 218.54batch/s]

Training batches on cuda:  82%|████████▏ | 7978/9708 [00:38<00:07, 222.99batch/s]

Training batches on cuda:  82%|████████▏ | 8001/9708 [00:38<00:07, 222.19batch/s]

Training batches on cuda:  83%|████████▎ | 8024/9708 [00:38<00:07, 223.37batch/s]

Training batches on cuda:  83%|████████▎ | 8047/9708 [00:38<00:07, 223.04batch/s]

Training batches on cuda:  83%|████████▎ | 8070/9708 [00:38<00:07, 223.89batch/s]

Training batches on cuda:  83%|████████▎ | 8093/9708 [00:38<00:07, 225.26batch/s]

Training batches on cuda:  84%|████████▎ | 8117/9708 [00:38<00:07, 227.03batch/s]

Training batches on cuda:  84%|████████▍ | 8140/9708 [00:38<00:06, 226.00batch/s]

Training batches on cuda:  84%|████████▍ | 8163/9708 [00:38<00:06, 225.90batch/s]

Training batches on cuda:  84%|████████▍ | 8186/9708 [00:39<00:06, 223.67batch/s]

Training batches on cuda:  85%|████████▍ | 8209/9708 [00:39<00:07, 210.80batch/s]

Training batches on cuda:  85%|████████▍ | 8231/9708 [00:39<00:07, 203.91batch/s]

Training batches on cuda:  85%|████████▌ | 8252/9708 [00:39<00:07, 198.46batch/s]

Training batches on cuda:  85%|████████▌ | 8275/9708 [00:39<00:06, 204.83batch/s]

Training batches on cuda:  85%|████████▌ | 8297/9708 [00:39<00:06, 208.20batch/s]

Training batches on cuda:  86%|████████▌ | 8319/9708 [00:39<00:06, 209.19batch/s]

Training batches on cuda:  86%|████████▌ | 8341/9708 [00:39<00:06, 202.75batch/s]

Training batches on cuda:  86%|████████▌ | 8362/9708 [00:39<00:06, 196.28batch/s]

Training batches on cuda:  86%|████████▋ | 8382/9708 [00:40<00:06, 194.10batch/s]

Training batches on cuda:  87%|████████▋ | 8402/9708 [00:40<00:06, 191.46batch/s]

Training batches on cuda:  87%|████████▋ | 8422/9708 [00:40<00:06, 191.19batch/s]

Training batches on cuda:  87%|████████▋ | 8442/9708 [00:40<00:06, 191.59batch/s]

Training batches on cuda:  87%|████████▋ | 8462/9708 [00:40<00:06, 189.79batch/s]

Training batches on cuda:  87%|████████▋ | 8482/9708 [00:40<00:06, 188.59batch/s]

Training batches on cuda:  88%|████████▊ | 8501/9708 [00:40<00:06, 188.14batch/s]

Training batches on cuda:  88%|████████▊ | 8520/9708 [00:40<00:06, 186.64batch/s]

Training batches on cuda:  88%|████████▊ | 8539/9708 [00:40<00:06, 185.86batch/s]

Training batches on cuda:  88%|████████▊ | 8558/9708 [00:40<00:06, 186.64batch/s]

Training batches on cuda:  88%|████████▊ | 8577/9708 [00:41<00:06, 183.92batch/s]

Training batches on cuda:  89%|████████▊ | 8596/9708 [00:41<00:06, 181.78batch/s]

Training batches on cuda:  89%|████████▉ | 8616/9708 [00:41<00:05, 185.05batch/s]

Training batches on cuda:  89%|████████▉ | 8637/9708 [00:41<00:05, 189.78batch/s]

Training batches on cuda:  89%|████████▉ | 8657/9708 [00:41<00:05, 191.13batch/s]

Training batches on cuda:  89%|████████▉ | 8678/9708 [00:41<00:05, 193.72batch/s]

Training batches on cuda:  90%|████████▉ | 8699/9708 [00:41<00:05, 195.64batch/s]

Training batches on cuda:  90%|████████▉ | 8720/9708 [00:41<00:04, 198.41batch/s]

Training batches on cuda:  90%|█████████ | 8741/9708 [00:41<00:04, 201.50batch/s]

Training batches on cuda:  90%|█████████ | 8762/9708 [00:42<00:04, 203.61batch/s]

Training batches on cuda:  90%|█████████ | 8784/9708 [00:42<00:04, 205.73batch/s]

Training batches on cuda:  91%|█████████ | 8805/9708 [00:42<00:04, 195.51batch/s]

Training batches on cuda:  91%|█████████ | 8827/9708 [00:42<00:04, 200.98batch/s]

Training batches on cuda:  91%|█████████ | 8850/9708 [00:42<00:04, 207.44batch/s]

Training batches on cuda:  91%|█████████▏| 8872/9708 [00:42<00:04, 208.18batch/s]

Training batches on cuda:  92%|█████████▏| 8893/9708 [00:42<00:03, 207.02batch/s]

Training batches on cuda:  92%|█████████▏| 8914/9708 [00:42<00:03, 207.23batch/s]

Training batches on cuda:  92%|█████████▏| 8935/9708 [00:42<00:03, 206.14batch/s]

Training batches on cuda:  92%|█████████▏| 8957/9708 [00:42<00:03, 209.55batch/s]

Training batches on cuda:  92%|█████████▏| 8979/9708 [00:43<00:03, 210.63batch/s]

Training batches on cuda:  93%|█████████▎| 9001/9708 [00:43<00:03, 211.99batch/s]

Training batches on cuda:  93%|█████████▎| 9023/9708 [00:43<00:03, 212.81batch/s]

Training batches on cuda:  93%|█████████▎| 9045/9708 [00:43<00:03, 212.15batch/s]

Training batches on cuda:  93%|█████████▎| 9067/9708 [00:43<00:03, 207.15batch/s]

Training batches on cuda:  94%|█████████▎| 9088/9708 [00:43<00:03, 206.56batch/s]

Training batches on cuda:  94%|█████████▍| 9109/9708 [00:43<00:02, 205.99batch/s]

Training batches on cuda:  94%|█████████▍| 9130/9708 [00:43<00:02, 204.95batch/s]

Training batches on cuda:  94%|█████████▍| 9152/9708 [00:43<00:02, 207.93batch/s]

Training batches on cuda:  94%|█████████▍| 9173/9708 [00:43<00:02, 207.20batch/s]

Training batches on cuda:  95%|█████████▍| 9194/9708 [00:44<00:02, 206.97batch/s]

Training batches on cuda:  95%|█████████▍| 9216/9708 [00:44<00:02, 210.30batch/s]

Training batches on cuda:  95%|█████████▌| 9238/9708 [00:44<00:02, 212.89batch/s]

Training batches on cuda:  95%|█████████▌| 9260/9708 [00:44<00:02, 211.06batch/s]

Training batches on cuda:  96%|█████████▌| 9282/9708 [00:44<00:02, 211.51batch/s]

Training batches on cuda:  96%|█████████▌| 9305/9708 [00:44<00:01, 215.84batch/s]

Training batches on cuda:  96%|█████████▌| 9327/9708 [00:44<00:01, 211.92batch/s]

Training batches on cuda:  96%|█████████▋| 9349/9708 [00:44<00:01, 209.58batch/s]

Training batches on cuda:  97%|█████████▋| 9370/9708 [00:44<00:01, 207.50batch/s]

Training batches on cuda:  97%|█████████▋| 9391/9708 [00:45<00:01, 205.44batch/s]

Training batches on cuda:  97%|█████████▋| 9412/9708 [00:45<00:01, 205.17batch/s]

Training batches on cuda:  97%|█████████▋| 9433/9708 [00:45<00:01, 206.01batch/s]

Training batches on cuda:  97%|█████████▋| 9454/9708 [00:45<00:01, 202.08batch/s]

Training batches on cuda:  98%|█████████▊| 9475/9708 [00:45<00:01, 203.09batch/s]

Training batches on cuda:  98%|█████████▊| 9496/9708 [00:45<00:01, 199.90batch/s]

Training batches on cuda:  98%|█████████▊| 9517/9708 [00:45<00:00, 197.61batch/s]

Training batches on cuda:  98%|█████████▊| 9537/9708 [00:45<00:00, 195.98batch/s]

Training batches on cuda:  98%|█████████▊| 9558/9708 [00:45<00:00, 197.76batch/s]

Training batches on cuda:  99%|█████████▊| 9579/9708 [00:45<00:00, 200.21batch/s]

Training batches on cuda:  99%|█████████▉| 9600/9708 [00:46<00:00, 197.04batch/s]

Training batches on cuda:  99%|█████████▉| 9620/9708 [00:46<00:00, 187.42batch/s]

Training batches on cuda:  99%|█████████▉| 9642/9708 [00:46<00:00, 195.46batch/s]

Training batches on cuda: 100%|█████████▉| 9663/9708 [00:46<00:00, 199.09batch/s]

Training batches on cuda: 100%|█████████▉| 9685/9708 [00:46<00:00, 203.36batch/s]

Training batches on cuda: 100%|█████████▉| 9707/9708 [00:46<00:00, 207.17batch/s]

                                                                                 
Training epochs on cuda:  10%|█         | 1/10 [01:34<07:04, 47.14s/epoch, loss=0.0417, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [01:34<06:16, 47.06s/epoch, loss=0.0417, prev_loss=nan]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 17/9708 [00:00<00:57, 169.73batch/s]

Training batches on cuda:   0%|          | 39/9708 [00:00<00:53, 181.06batch/s]

Training batches on cuda:   1%|          | 59/9708 [00:00<00:52, 184.70batch/s]

Training batches on cuda:   1%|          | 78/9708 [00:00<00:51, 186.08batch/s]

Training batches on cuda:   1%|          | 97/9708 [00:00<00:51, 185.22batch/s]

Training batches on cuda:   1%|          | 113/9708 [00:00<00:56, 168.76batch/s]

Training batches on cuda:   1%|▏         | 133/9708 [00:00<00:54, 176.46batch/s]

Training batches on cuda:   2%|▏         | 155/9708 [00:00<00:51, 184.15batch/s]

Training batches on cuda:   2%|▏         | 175/9708 [00:00<00:50, 188.50batch/s]

Training batches on cuda:   2%|▏         | 195/9708 [00:01<00:49, 190.83batch/s]

Training batches on cuda:   2%|▏         | 215/9708 [00:01<00:49, 192.12batch/s]

Training batches on cuda:   2%|▏         | 236/9708 [00:01<00:48, 194.43batch/s]

Training batches on cuda:   3%|▎         | 259/9708 [00:01<00:46, 202.31batch/s]

Training batches on cuda:   3%|▎         | 282/9708 [00:01<00:45, 207.48batch/s]

Training batches on cuda:   3%|▎         | 304/9708 [00:01<00:45, 208.52batch/s]

Training batches on cuda:   3%|▎         | 326/9708 [00:01<00:44, 211.21batch/s]

Training batches on cuda:   4%|▎         | 348/9708 [00:01<00:44, 211.52batch/s]

Training batches on cuda:   4%|▍         | 370/9708 [00:01<00:44, 211.27batch/s]

Training batches on cuda:   4%|▍         | 392/9708 [00:01<00:44, 210.87batch/s]

Training batches on cuda:   4%|▍         | 415/9708 [00:02<00:43, 215.12batch/s]

Training batches on cuda:   5%|▍         | 438/9708 [00:02<00:42, 218.36batch/s]

Training batches on cuda:   5%|▍         | 460/9708 [00:02<00:42, 217.56batch/s]

Training batches on cuda:   5%|▍         | 484/9708 [00:02<00:41, 221.53batch/s]

Training batches on cuda:   5%|▌         | 507/9708 [00:02<00:41, 222.64batch/s]

Training batches on cuda:   5%|▌         | 530/9708 [00:02<00:40, 224.27batch/s]

Training batches on cuda:   6%|▌         | 553/9708 [00:02<00:40, 225.14batch/s]

Training batches on cuda:   6%|▌         | 577/9708 [00:02<00:40, 226.89batch/s]

Training batches on cuda:   6%|▌         | 600/9708 [00:02<00:40, 225.93batch/s]

Training batches on cuda:   6%|▋         | 623/9708 [00:02<00:40, 223.44batch/s]

Training batches on cuda:   7%|▋         | 647/9708 [00:03<00:40, 226.09batch/s]

Training batches on cuda:   7%|▋         | 670/9708 [00:03<00:40, 222.16batch/s]

Training batches on cuda:   7%|▋         | 693/9708 [00:03<00:41, 216.05batch/s]

Training batches on cuda:   7%|▋         | 715/9708 [00:03<00:42, 213.56batch/s]

Training batches on cuda:   8%|▊         | 738/9708 [00:03<00:41, 217.62batch/s]

Training batches on cuda:   8%|▊         | 761/9708 [00:03<00:40, 219.88batch/s]

Training batches on cuda:   8%|▊         | 784/9708 [00:03<00:40, 220.76batch/s]

Training batches on cuda:   8%|▊         | 807/9708 [00:03<00:40, 221.79batch/s]

Training batches on cuda:   9%|▊         | 830/9708 [00:03<00:39, 222.70batch/s]

Training batches on cuda:   9%|▉         | 853/9708 [00:04<00:39, 223.68batch/s]

Training batches on cuda:   9%|▉         | 876/9708 [00:04<00:39, 222.91batch/s]

Training batches on cuda:   9%|▉         | 899/9708 [00:04<00:39, 222.37batch/s]

Training batches on cuda:   9%|▉         | 922/9708 [00:04<00:39, 223.32batch/s]

Training batches on cuda:  10%|▉         | 945/9708 [00:04<00:39, 223.39batch/s]

Training batches on cuda:  10%|▉         | 970/9708 [00:04<00:38, 228.54batch/s]

Training batches on cuda:  10%|█         | 993/9708 [00:04<00:38, 224.40batch/s]

Training batches on cuda:  10%|█         | 1016/9708 [00:04<00:39, 219.08batch/s]

Training batches on cuda:  11%|█         | 1038/9708 [00:04<00:40, 214.69batch/s]

Training batches on cuda:  11%|█         | 1062/9708 [00:04<00:39, 219.17batch/s]

Training batches on cuda:  11%|█         | 1086/9708 [00:05<00:38, 222.47batch/s]

Training batches on cuda:  11%|█▏        | 1109/9708 [00:05<00:38, 224.60batch/s]

Training batches on cuda:  12%|█▏        | 1132/9708 [00:05<00:38, 224.10batch/s]

Training batches on cuda:  12%|█▏        | 1155/9708 [00:05<00:38, 224.75batch/s]

Training batches on cuda:  12%|█▏        | 1178/9708 [00:05<00:37, 225.07batch/s]

Training batches on cuda:  12%|█▏        | 1201/9708 [00:05<00:37, 225.02batch/s]

Training batches on cuda:  13%|█▎        | 1224/9708 [00:05<00:37, 224.95batch/s]

Training batches on cuda:  13%|█▎        | 1248/9708 [00:05<00:37, 227.75batch/s]

Training batches on cuda:  13%|█▎        | 1272/9708 [00:05<00:36, 230.92batch/s]

Training batches on cuda:  13%|█▎        | 1296/9708 [00:06<00:38, 221.36batch/s]

Training batches on cuda:  14%|█▎        | 1320/9708 [00:06<00:37, 224.44batch/s]

Training batches on cuda:  14%|█▍        | 1343/9708 [00:06<00:37, 223.86batch/s]

Training batches on cuda:  14%|█▍        | 1366/9708 [00:06<00:37, 223.09batch/s]

Training batches on cuda:  14%|█▍        | 1389/9708 [00:06<00:37, 223.42batch/s]

Training batches on cuda:  15%|█▍        | 1412/9708 [00:06<00:37, 219.54batch/s]

Training batches on cuda:  15%|█▍        | 1434/9708 [00:06<00:38, 213.54batch/s]

Training batches on cuda:  15%|█▍        | 1456/9708 [00:06<00:39, 211.18batch/s]

Training batches on cuda:  15%|█▌        | 1478/9708 [00:06<00:39, 207.19batch/s]

Training batches on cuda:  15%|█▌        | 1499/9708 [00:06<00:41, 198.64batch/s]

Training batches on cuda:  16%|█▌        | 1522/9708 [00:07<00:39, 206.72batch/s]

Training batches on cuda:  16%|█▌        | 1547/9708 [00:07<00:37, 216.91batch/s]

Training batches on cuda:  16%|█▌        | 1572/9708 [00:07<00:36, 224.23batch/s]

Training batches on cuda:  16%|█▋        | 1599/9708 [00:07<00:34, 234.19batch/s]

Training batches on cuda:  17%|█▋        | 1623/9708 [00:07<00:35, 227.85batch/s]

Training batches on cuda:  17%|█▋        | 1647/9708 [00:07<00:37, 213.48batch/s]

Training batches on cuda:  17%|█▋        | 1669/9708 [00:07<00:38, 206.18batch/s]

Training batches on cuda:  17%|█▋        | 1691/9708 [00:07<00:38, 208.41batch/s]

Training batches on cuda:  18%|█▊        | 1713/9708 [00:07<00:37, 210.66batch/s]

Training batches on cuda:  18%|█▊        | 1735/9708 [00:08<00:37, 211.33batch/s]

Training batches on cuda:  18%|█▊        | 1757/9708 [00:08<00:38, 208.44batch/s]

Training batches on cuda:  18%|█▊        | 1778/9708 [00:08<00:38, 207.88batch/s]

Training batches on cuda:  19%|█▊        | 1799/9708 [00:08<00:38, 203.62batch/s]

Training batches on cuda:  19%|█▊        | 1820/9708 [00:08<00:39, 199.72batch/s]

Training batches on cuda:  19%|█▉        | 1841/9708 [00:08<00:41, 189.03batch/s]

Training batches on cuda:  19%|█▉        | 1861/9708 [00:08<00:44, 177.20batch/s]

Training batches on cuda:  19%|█▉        | 1879/9708 [00:08<00:44, 177.80batch/s]

Training batches on cuda:  20%|█▉        | 1898/9708 [00:08<00:43, 178.57batch/s]

Training batches on cuda:  20%|█▉        | 1916/9708 [00:09<00:44, 176.43batch/s]

Training batches on cuda:  20%|█▉        | 1936/9708 [00:09<00:43, 180.37batch/s]

Training batches on cuda:  20%|██        | 1955/9708 [00:09<00:42, 180.99batch/s]

Training batches on cuda:  20%|██        | 1974/9708 [00:09<00:42, 181.12batch/s]

Training batches on cuda:  21%|██        | 1993/9708 [00:09<00:44, 174.04batch/s]

Training batches on cuda:  21%|██        | 2011/9708 [00:09<00:44, 174.28batch/s]

Training batches on cuda:  21%|██        | 2029/9708 [00:09<00:43, 175.87batch/s]

Training batches on cuda:  21%|██        | 2047/9708 [00:09<00:43, 177.03batch/s]

Training batches on cuda:  21%|██▏       | 2066/9708 [00:09<00:42, 178.97batch/s]

Training batches on cuda:  21%|██▏       | 2085/9708 [00:09<00:42, 180.30batch/s]

Training batches on cuda:  22%|██▏       | 2105/9708 [00:10<00:41, 184.01batch/s]

Training batches on cuda:  22%|██▏       | 2125/9708 [00:10<00:40, 187.75batch/s]

Training batches on cuda:  22%|██▏       | 2145/9708 [00:10<00:39, 189.66batch/s]

Training batches on cuda:  22%|██▏       | 2165/9708 [00:10<00:39, 191.84batch/s]

Training batches on cuda:  23%|██▎       | 2185/9708 [00:10<00:39, 192.11batch/s]

Training batches on cuda:  23%|██▎       | 2205/9708 [00:10<00:39, 187.83batch/s]

Training batches on cuda:  23%|██▎       | 2224/9708 [00:10<00:43, 170.30batch/s]

Training batches on cuda:  23%|██▎       | 2243/9708 [00:10<00:42, 175.35batch/s]

Training batches on cuda:  23%|██▎       | 2261/9708 [00:10<00:42, 173.90batch/s]

Training batches on cuda:  23%|██▎       | 2280/9708 [00:11<00:42, 176.53batch/s]

Training batches on cuda:  24%|██▎       | 2300/9708 [00:11<00:40, 182.05batch/s]

Training batches on cuda:  24%|██▍       | 2319/9708 [00:11<00:40, 183.20batch/s]

Training batches on cuda:  24%|██▍       | 2339/9708 [00:11<00:39, 187.35batch/s]

Training batches on cuda:  24%|██▍       | 2359/9708 [00:11<00:38, 189.06batch/s]

Training batches on cuda:  25%|██▍       | 2379/9708 [00:11<00:38, 192.00batch/s]

Training batches on cuda:  25%|██▍       | 2401/9708 [00:11<00:36, 197.94batch/s]

Training batches on cuda:  25%|██▍       | 2423/9708 [00:11<00:36, 201.97batch/s]

Training batches on cuda:  25%|██▌       | 2445/9708 [00:11<00:35, 205.88batch/s]

Training batches on cuda:  25%|██▌       | 2467/9708 [00:11<00:34, 207.65batch/s]

Training batches on cuda:  26%|██▌       | 2488/9708 [00:12<00:34, 207.34batch/s]

Training batches on cuda:  26%|██▌       | 2510/9708 [00:12<00:34, 208.17batch/s]

Training batches on cuda:  26%|██▌       | 2531/9708 [00:12<00:34, 207.56batch/s]

Training batches on cuda:  26%|██▋       | 2552/9708 [00:12<00:34, 207.83batch/s]

Training batches on cuda:  27%|██▋       | 2573/9708 [00:12<00:34, 207.37batch/s]

Training batches on cuda:  27%|██▋       | 2594/9708 [00:12<00:34, 206.80batch/s]

Training batches on cuda:  27%|██▋       | 2615/9708 [00:12<00:34, 205.56batch/s]

Training batches on cuda:  27%|██▋       | 2636/9708 [00:12<00:35, 200.35batch/s]

Training batches on cuda:  27%|██▋       | 2657/9708 [00:12<00:35, 199.99batch/s]

Training batches on cuda:  28%|██▊       | 2678/9708 [00:13<00:35, 199.54batch/s]

Training batches on cuda:  28%|██▊       | 2699/9708 [00:13<00:34, 200.29batch/s]

Training batches on cuda:  28%|██▊       | 2721/9708 [00:13<00:34, 203.53batch/s]

Training batches on cuda:  28%|██▊       | 2742/9708 [00:13<00:34, 204.26batch/s]

Training batches on cuda:  28%|██▊       | 2765/9708 [00:13<00:33, 209.54batch/s]

Training batches on cuda:  29%|██▊       | 2788/9708 [00:13<00:32, 213.79batch/s]

Training batches on cuda:  29%|██▉       | 2811/9708 [00:13<00:31, 216.20batch/s]

Training batches on cuda:  29%|██▉       | 2833/9708 [00:13<00:32, 213.72batch/s]

Training batches on cuda:  29%|██▉       | 2855/9708 [00:13<00:31, 215.42batch/s]

Training batches on cuda:  30%|██▉       | 2878/9708 [00:13<00:31, 217.75batch/s]

Training batches on cuda:  30%|██▉       | 2900/9708 [00:14<00:31, 217.83batch/s]

Training batches on cuda:  30%|███       | 2922/9708 [00:14<00:31, 214.64batch/s]

Training batches on cuda:  30%|███       | 2945/9708 [00:14<00:31, 216.45batch/s]

Training batches on cuda:  31%|███       | 2967/9708 [00:14<00:31, 215.65batch/s]

Training batches on cuda:  31%|███       | 2990/9708 [00:14<00:30, 218.04batch/s]

Training batches on cuda:  31%|███       | 3013/9708 [00:14<00:30, 220.05batch/s]

Training batches on cuda:  31%|███▏      | 3036/9708 [00:14<00:30, 222.17batch/s]

Training batches on cuda:  32%|███▏      | 3059/9708 [00:14<00:29, 224.36batch/s]

Training batches on cuda:  32%|███▏      | 3082/9708 [00:14<00:29, 224.94batch/s]

Training batches on cuda:  32%|███▏      | 3105/9708 [00:14<00:29, 223.86batch/s]

Training batches on cuda:  32%|███▏      | 3128/9708 [00:15<00:29, 223.10batch/s]

Training batches on cuda:  32%|███▏      | 3151/9708 [00:15<00:29, 222.85batch/s]

Training batches on cuda:  33%|███▎      | 3174/9708 [00:15<00:29, 222.15batch/s]

Training batches on cuda:  33%|███▎      | 3197/9708 [00:15<00:29, 223.03batch/s]

Training batches on cuda:  33%|███▎      | 3220/9708 [00:15<00:29, 222.86batch/s]

Training batches on cuda:  33%|███▎      | 3243/9708 [00:15<00:29, 222.30batch/s]

Training batches on cuda:  34%|███▎      | 3266/9708 [00:15<00:28, 222.81batch/s]

Training batches on cuda:  34%|███▍      | 3289/9708 [00:15<00:28, 224.39batch/s]

Training batches on cuda:  34%|███▍      | 3312/9708 [00:15<00:30, 213.10batch/s]

Training batches on cuda:  34%|███▍      | 3334/9708 [00:16<00:29, 213.81batch/s]

Training batches on cuda:  35%|███▍      | 3356/9708 [00:16<00:29, 214.05batch/s]

Training batches on cuda:  35%|███▍      | 3378/9708 [00:16<00:29, 214.19batch/s]

Training batches on cuda:  35%|███▌      | 3400/9708 [00:16<00:30, 205.00batch/s]

Training batches on cuda:  35%|███▌      | 3421/9708 [00:16<00:30, 205.76batch/s]

Training batches on cuda:  35%|███▌      | 3443/9708 [00:16<00:29, 209.58batch/s]

Training batches on cuda:  36%|███▌      | 3466/9708 [00:16<00:29, 213.31batch/s]

Training batches on cuda:  36%|███▌      | 3489/9708 [00:16<00:28, 215.60batch/s]

Training batches on cuda:  36%|███▌      | 3511/9708 [00:16<00:28, 216.47batch/s]

Training batches on cuda:  36%|███▋      | 3534/9708 [00:16<00:28, 218.39batch/s]

Training batches on cuda:  37%|███▋      | 3557/9708 [00:17<00:28, 219.54batch/s]

Training batches on cuda:  37%|███▋      | 3579/9708 [00:17<00:28, 217.80batch/s]

Training batches on cuda:  37%|███▋      | 3601/9708 [00:17<00:29, 210.31batch/s]

Training batches on cuda:  37%|███▋      | 3623/9708 [00:17<00:29, 209.60batch/s]

Training batches on cuda:  38%|███▊      | 3646/9708 [00:17<00:28, 213.13batch/s]

Training batches on cuda:  38%|███▊      | 3670/9708 [00:17<00:27, 219.72batch/s]

Training batches on cuda:  38%|███▊      | 3693/9708 [00:17<00:27, 221.57batch/s]

Training batches on cuda:  38%|███▊      | 3716/9708 [00:17<00:27, 221.77batch/s]

Training batches on cuda:  39%|███▊      | 3739/9708 [00:17<00:27, 217.98batch/s]

Training batches on cuda:  39%|███▉      | 3762/9708 [00:18<00:26, 220.92batch/s]

Training batches on cuda:  39%|███▉      | 3785/9708 [00:18<00:28, 210.80batch/s]

Training batches on cuda:  39%|███▉      | 3807/9708 [00:18<00:27, 211.58batch/s]

Training batches on cuda:  39%|███▉      | 3829/9708 [00:18<00:28, 207.53batch/s]

Training batches on cuda:  40%|███▉      | 3850/9708 [00:18<00:28, 206.17batch/s]

Training batches on cuda:  40%|███▉      | 3872/9708 [00:18<00:27, 209.85batch/s]

Training batches on cuda:  40%|████      | 3894/9708 [00:18<00:27, 209.74batch/s]

Training batches on cuda:  40%|████      | 3916/9708 [00:18<00:27, 211.24batch/s]

Training batches on cuda:  41%|████      | 3938/9708 [00:18<00:27, 213.43batch/s]

Training batches on cuda:  41%|████      | 3960/9708 [00:18<00:26, 213.99batch/s]

Training batches on cuda:  41%|████      | 3982/9708 [00:19<00:27, 209.10batch/s]

Training batches on cuda:  41%|████      | 4003/9708 [00:19<00:27, 208.46batch/s]

Training batches on cuda:  41%|████▏     | 4025/9708 [00:19<00:26, 211.54batch/s]

Training batches on cuda:  42%|████▏     | 4047/9708 [00:19<00:26, 212.24batch/s]

Training batches on cuda:  42%|████▏     | 4069/9708 [00:19<00:28, 200.74batch/s]

Training batches on cuda:  42%|████▏     | 4092/9708 [00:19<00:27, 206.47batch/s]

Training batches on cuda:  42%|████▏     | 4115/9708 [00:19<00:26, 211.28batch/s]

Training batches on cuda:  43%|████▎     | 4140/9708 [00:19<00:25, 219.28batch/s]

Training batches on cuda:  43%|████▎     | 4163/9708 [00:19<00:25, 213.63batch/s]

Training batches on cuda:  43%|████▎     | 4185/9708 [00:20<00:26, 210.48batch/s]

Training batches on cuda:  43%|████▎     | 4207/9708 [00:20<00:26, 209.79batch/s]

Training batches on cuda:  44%|████▎     | 4229/9708 [00:20<00:26, 208.54batch/s]

Training batches on cuda:  44%|████▍     | 4250/9708 [00:20<00:26, 208.54batch/s]

Training batches on cuda:  44%|████▍     | 4271/9708 [00:20<00:26, 206.15batch/s]

Training batches on cuda:  44%|████▍     | 4293/9708 [00:20<00:25, 208.37batch/s]

Training batches on cuda:  44%|████▍     | 4314/9708 [00:20<00:26, 207.40batch/s]

Training batches on cuda:  45%|████▍     | 4335/9708 [00:20<00:26, 203.27batch/s]

Training batches on cuda:  45%|████▍     | 4356/9708 [00:20<00:26, 202.43batch/s]

Training batches on cuda:  45%|████▌     | 4377/9708 [00:20<00:28, 188.57batch/s]

Training batches on cuda:  45%|████▌     | 4397/9708 [00:21<00:29, 179.07batch/s]

Training batches on cuda:  45%|████▌     | 4417/9708 [00:21<00:28, 182.80batch/s]

Training batches on cuda:  46%|████▌     | 4439/9708 [00:21<00:27, 191.66batch/s]

Training batches on cuda:  46%|████▌     | 4460/9708 [00:21<00:26, 195.71batch/s]

Training batches on cuda:  46%|████▌     | 4482/9708 [00:21<00:25, 201.20batch/s]

Training batches on cuda:  46%|████▋     | 4505/9708 [00:21<00:25, 207.34batch/s]

Training batches on cuda:  47%|████▋     | 4528/9708 [00:21<00:24, 213.50batch/s]

Training batches on cuda:  47%|████▋     | 4551/9708 [00:21<00:23, 217.56batch/s]

Training batches on cuda:  47%|████▋     | 4573/9708 [00:21<00:23, 217.47batch/s]

Training batches on cuda:  47%|████▋     | 4596/9708 [00:22<00:23, 220.98batch/s]

Training batches on cuda:  48%|████▊     | 4619/9708 [00:22<00:23, 217.55batch/s]

Training batches on cuda:  48%|████▊     | 4641/9708 [00:22<00:23, 213.93batch/s]

Training batches on cuda:  48%|████▊     | 4663/9708 [00:22<00:24, 203.21batch/s]

Training batches on cuda:  48%|████▊     | 4686/9708 [00:22<00:23, 210.37batch/s]

Training batches on cuda:  49%|████▊     | 4709/9708 [00:22<00:23, 213.62batch/s]

Training batches on cuda:  49%|████▊     | 4731/9708 [00:22<00:23, 208.06batch/s]

Training batches on cuda:  49%|████▉     | 4752/9708 [00:22<00:24, 203.79batch/s]

Training batches on cuda:  49%|████▉     | 4773/9708 [00:22<00:24, 198.23batch/s]

Training batches on cuda:  49%|████▉     | 4793/9708 [00:23<00:25, 191.32batch/s]

Training batches on cuda:  50%|████▉     | 4813/9708 [00:23<00:25, 190.59batch/s]

Training batches on cuda:  50%|████▉     | 4833/9708 [00:23<00:26, 183.23batch/s]

Training batches on cuda:  50%|████▉     | 4853/9708 [00:23<00:25, 187.48batch/s]

Training batches on cuda:  50%|█████     | 4874/9708 [00:23<00:25, 192.51batch/s]

Training batches on cuda:  50%|█████     | 4894/9708 [00:23<00:24, 192.97batch/s]

Training batches on cuda:  51%|█████     | 4915/9708 [00:23<00:24, 195.50batch/s]

Training batches on cuda:  51%|█████     | 4937/9708 [00:23<00:23, 200.30batch/s]

Training batches on cuda:  51%|█████     | 4959/9708 [00:23<00:23, 203.17batch/s]

Training batches on cuda:  51%|█████▏    | 4982/9708 [00:23<00:22, 209.32batch/s]

Training batches on cuda:  52%|█████▏    | 5004/9708 [00:24<00:22, 206.73batch/s]

Training batches on cuda:  52%|█████▏    | 5027/9708 [00:24<00:22, 211.89batch/s]

Training batches on cuda:  52%|█████▏    | 5050/9708 [00:24<00:21, 216.27batch/s]

Training batches on cuda:  52%|█████▏    | 5072/9708 [00:24<00:21, 216.30batch/s]

Training batches on cuda:  52%|█████▏    | 5095/9708 [00:24<00:21, 218.96batch/s]

Training batches on cuda:  53%|█████▎    | 5118/9708 [00:24<00:20, 219.32batch/s]

Training batches on cuda:  53%|█████▎    | 5140/9708 [00:24<00:20, 218.82batch/s]

Training batches on cuda:  53%|█████▎    | 5163/9708 [00:24<00:20, 219.52batch/s]

Training batches on cuda:  53%|█████▎    | 5185/9708 [00:24<00:21, 207.07batch/s]

Training batches on cuda:  54%|█████▎    | 5206/9708 [00:25<00:21, 207.62batch/s]

Training batches on cuda:  54%|█████▍    | 5227/9708 [00:25<00:22, 202.87batch/s]

Training batches on cuda:  54%|█████▍    | 5249/9708 [00:25<00:21, 206.15batch/s]

Training batches on cuda:  54%|█████▍    | 5270/9708 [00:25<00:21, 203.48batch/s]

Training batches on cuda:  55%|█████▍    | 5292/9708 [00:25<00:21, 206.25batch/s]

Training batches on cuda:  55%|█████▍    | 5313/9708 [00:25<00:21, 202.77batch/s]

Training batches on cuda:  55%|█████▍    | 5334/9708 [00:25<00:21, 199.51batch/s]

Training batches on cuda:  55%|█████▌    | 5354/9708 [00:25<00:22, 196.54batch/s]

Training batches on cuda:  55%|█████▌    | 5374/9708 [00:25<00:22, 194.91batch/s]

Training batches on cuda:  56%|█████▌    | 5394/9708 [00:25<00:22, 194.50batch/s]

Training batches on cuda:  56%|█████▌    | 5414/9708 [00:26<00:22, 193.67batch/s]

Training batches on cuda:  56%|█████▌    | 5434/9708 [00:26<00:22, 193.99batch/s]

Training batches on cuda:  56%|█████▌    | 5454/9708 [00:26<00:22, 191.27batch/s]

Training batches on cuda:  56%|█████▋    | 5474/9708 [00:26<00:22, 187.98batch/s]

Training batches on cuda:  57%|█████▋    | 5493/9708 [00:26<00:23, 183.20batch/s]

Training batches on cuda:  57%|█████▋    | 5512/9708 [00:26<00:23, 179.89batch/s]

Training batches on cuda:  57%|█████▋    | 5531/9708 [00:26<00:23, 181.27batch/s]

Training batches on cuda:  57%|█████▋    | 5552/9708 [00:26<00:22, 187.76batch/s]

Training batches on cuda:  57%|█████▋    | 5571/9708 [00:26<00:22, 185.17batch/s]

Training batches on cuda:  58%|█████▊    | 5593/9708 [00:27<00:21, 192.73batch/s]

Training batches on cuda:  58%|█████▊    | 5615/9708 [00:27<00:20, 198.98batch/s]

Training batches on cuda:  58%|█████▊    | 5636/9708 [00:27<00:20, 194.46batch/s]

Training batches on cuda:  58%|█████▊    | 5656/9708 [00:27<00:22, 181.55batch/s]

Training batches on cuda:  58%|█████▊    | 5676/9708 [00:27<00:21, 185.89batch/s]

Training batches on cuda:  59%|█████▊    | 5695/9708 [00:27<00:21, 186.26batch/s]

Training batches on cuda:  59%|█████▉    | 5716/9708 [00:27<00:20, 190.77batch/s]

Training batches on cuda:  59%|█████▉    | 5738/9708 [00:27<00:20, 196.71batch/s]

Training batches on cuda:  59%|█████▉    | 5758/9708 [00:27<00:20, 196.04batch/s]

Training batches on cuda:  60%|█████▉    | 5778/9708 [00:28<00:21, 185.41batch/s]

Training batches on cuda:  60%|█████▉    | 5798/9708 [00:28<00:20, 188.84batch/s]

Training batches on cuda:  60%|█████▉    | 5818/9708 [00:28<00:20, 190.80batch/s]

Training batches on cuda:  60%|██████    | 5838/9708 [00:28<00:20, 192.81batch/s]

Training batches on cuda:  60%|██████    | 5860/9708 [00:28<00:19, 200.09batch/s]

Training batches on cuda:  61%|██████    | 5881/9708 [00:28<00:19, 199.36batch/s]

Training batches on cuda:  61%|██████    | 5902/9708 [00:28<00:18, 201.46batch/s]

Training batches on cuda:  61%|██████    | 5923/9708 [00:28<00:18, 201.30batch/s]

Training batches on cuda:  61%|██████    | 5944/9708 [00:28<00:18, 202.70batch/s]

Training batches on cuda:  61%|██████▏   | 5965/9708 [00:28<00:18, 202.21batch/s]

Training batches on cuda:  62%|██████▏   | 5987/9708 [00:29<00:18, 204.91batch/s]

Training batches on cuda:  62%|██████▏   | 6008/9708 [00:29<00:18, 204.29batch/s]

Training batches on cuda:  62%|██████▏   | 6029/9708 [00:29<00:18, 194.02batch/s]

Training batches on cuda:  62%|██████▏   | 6049/9708 [00:29<00:19, 189.51batch/s]

Training batches on cuda:  63%|██████▎   | 6069/9708 [00:29<00:19, 186.62batch/s]

Training batches on cuda:  63%|██████▎   | 6088/9708 [00:29<00:19, 184.04batch/s]

Training batches on cuda:  63%|██████▎   | 6107/9708 [00:29<00:19, 182.17batch/s]

Training batches on cuda:  63%|██████▎   | 6126/9708 [00:29<00:19, 181.69batch/s]

Training batches on cuda:  63%|██████▎   | 6148/9708 [00:29<00:18, 189.57batch/s]

Training batches on cuda:  64%|██████▎   | 6170/9708 [00:30<00:18, 196.04batch/s]

Training batches on cuda:  64%|██████▍   | 6191/9708 [00:30<00:17, 199.05batch/s]

Training batches on cuda:  64%|██████▍   | 6212/9708 [00:30<00:17, 201.71batch/s]

Training batches on cuda:  64%|██████▍   | 6233/9708 [00:30<00:17, 203.52batch/s]

Training batches on cuda:  64%|██████▍   | 6255/9708 [00:30<00:16, 206.03batch/s]

Training batches on cuda:  65%|██████▍   | 6278/9708 [00:30<00:16, 212.51batch/s]

Training batches on cuda:  65%|██████▍   | 6301/9708 [00:30<00:15, 216.58batch/s]

Training batches on cuda:  65%|██████▌   | 6325/9708 [00:30<00:15, 222.34batch/s]

Training batches on cuda:  65%|██████▌   | 6349/9708 [00:30<00:14, 224.68batch/s]

Training batches on cuda:  66%|██████▌   | 6372/9708 [00:30<00:14, 222.95batch/s]

Training batches on cuda:  66%|██████▌   | 6396/9708 [00:31<00:14, 226.09batch/s]

Training batches on cuda:  66%|██████▌   | 6419/9708 [00:31<00:14, 226.32batch/s]

Training batches on cuda:  66%|██████▋   | 6443/9708 [00:31<00:14, 229.79batch/s]

Training batches on cuda:  67%|██████▋   | 6467/9708 [00:31<00:15, 213.20batch/s]

Training batches on cuda:  67%|██████▋   | 6489/9708 [00:31<00:15, 211.45batch/s]

Training batches on cuda:  67%|██████▋   | 6511/9708 [00:31<00:15, 204.09batch/s]

Training batches on cuda:  67%|██████▋   | 6532/9708 [00:31<00:16, 194.02batch/s]

Training batches on cuda:  67%|██████▋   | 6552/9708 [00:31<00:16, 192.33batch/s]

Training batches on cuda:  68%|██████▊   | 6573/9708 [00:31<00:15, 196.30batch/s]

Training batches on cuda:  68%|██████▊   | 6595/9708 [00:32<00:15, 200.87batch/s]

Training batches on cuda:  68%|██████▊   | 6617/9708 [00:32<00:15, 203.76batch/s]

Training batches on cuda:  68%|██████▊   | 6639/9708 [00:32<00:14, 207.42batch/s]

Training batches on cuda:  69%|██████▊   | 6660/9708 [00:32<00:15, 201.07batch/s]

Training batches on cuda:  69%|██████▉   | 6681/9708 [00:32<00:15, 195.73batch/s]

Training batches on cuda:  69%|██████▉   | 6701/9708 [00:32<00:15, 193.03batch/s]

Training batches on cuda:  69%|██████▉   | 6723/9708 [00:32<00:14, 200.15batch/s]

Training batches on cuda:  69%|██████▉   | 6746/9708 [00:32<00:14, 207.56batch/s]

Training batches on cuda:  70%|██████▉   | 6768/9708 [00:32<00:14, 209.97batch/s]

Training batches on cuda:  70%|██████▉   | 6791/9708 [00:32<00:13, 214.00batch/s]

Training batches on cuda:  70%|███████   | 6814/9708 [00:33<00:13, 216.30batch/s]

Training batches on cuda:  70%|███████   | 6837/9708 [00:33<00:13, 219.32batch/s]

Training batches on cuda:  71%|███████   | 6860/9708 [00:33<00:12, 220.23batch/s]

Training batches on cuda:  71%|███████   | 6883/9708 [00:33<00:12, 220.88batch/s]

Training batches on cuda:  71%|███████   | 6906/9708 [00:33<00:12, 222.44batch/s]

Training batches on cuda:  71%|███████▏  | 6929/9708 [00:33<00:12, 221.37batch/s]

Training batches on cuda:  72%|███████▏  | 6952/9708 [00:33<00:12, 223.42batch/s]

Training batches on cuda:  72%|███████▏  | 6975/9708 [00:33<00:12, 225.25batch/s]

Training batches on cuda:  72%|███████▏  | 6998/9708 [00:33<00:11, 226.49batch/s]

Training batches on cuda:  72%|███████▏  | 7021/9708 [00:33<00:11, 227.18batch/s]

Training batches on cuda:  73%|███████▎  | 7045/9708 [00:34<00:11, 227.99batch/s]

Training batches on cuda:  73%|███████▎  | 7068/9708 [00:34<00:11, 226.81batch/s]

Training batches on cuda:  73%|███████▎  | 7091/9708 [00:34<00:11, 226.17batch/s]

Training batches on cuda:  73%|███████▎  | 7114/9708 [00:34<00:12, 215.63batch/s]

Training batches on cuda:  74%|███████▎  | 7136/9708 [00:34<00:12, 206.73batch/s]

Training batches on cuda:  74%|███████▎  | 7158/9708 [00:34<00:12, 207.93batch/s]

Training batches on cuda:  74%|███████▍  | 7179/9708 [00:34<00:12, 205.98batch/s]

Training batches on cuda:  74%|███████▍  | 7200/9708 [00:34<00:12, 206.82batch/s]

Training batches on cuda:  74%|███████▍  | 7221/9708 [00:34<00:11, 207.37batch/s]

Training batches on cuda:  75%|███████▍  | 7242/9708 [00:35<00:12, 205.49batch/s]

Training batches on cuda:  75%|███████▍  | 7263/9708 [00:35<00:11, 204.88batch/s]

Training batches on cuda:  75%|███████▌  | 7284/9708 [00:35<00:11, 205.40batch/s]

Training batches on cuda:  75%|███████▌  | 7305/9708 [00:35<00:11, 205.86batch/s]

Training batches on cuda:  75%|███████▌  | 7326/9708 [00:35<00:11, 205.21batch/s]

Training batches on cuda:  76%|███████▌  | 7348/9708 [00:35<00:11, 207.01batch/s]

Training batches on cuda:  76%|███████▌  | 7370/9708 [00:35<00:11, 209.11batch/s]

Training batches on cuda:  76%|███████▌  | 7392/9708 [00:35<00:11, 210.32batch/s]

Training batches on cuda:  76%|███████▋  | 7414/9708 [00:35<00:12, 186.76batch/s]

Training batches on cuda:  77%|███████▋  | 7434/9708 [00:36<00:12, 188.45batch/s]

Training batches on cuda:  77%|███████▋  | 7454/9708 [00:36<00:11, 191.04batch/s]

Training batches on cuda:  77%|███████▋  | 7474/9708 [00:36<00:11, 192.05batch/s]

Training batches on cuda:  77%|███████▋  | 7495/9708 [00:36<00:11, 194.82batch/s]

Training batches on cuda:  77%|███████▋  | 7516/9708 [00:36<00:11, 198.72batch/s]

Training batches on cuda:  78%|███████▊  | 7538/9708 [00:36<00:10, 203.29batch/s]

Training batches on cuda:  78%|███████▊  | 7560/9708 [00:36<00:10, 205.77batch/s]

Training batches on cuda:  78%|███████▊  | 7581/9708 [00:36<00:10, 204.71batch/s]

Training batches on cuda:  78%|███████▊  | 7602/9708 [00:36<00:10, 197.08batch/s]

Training batches on cuda:  79%|███████▊  | 7622/9708 [00:36<00:11, 187.91batch/s]

Training batches on cuda:  79%|███████▊  | 7641/9708 [00:37<00:11, 182.62batch/s]

Training batches on cuda:  79%|███████▉  | 7660/9708 [00:37<00:11, 181.95batch/s]

Training batches on cuda:  79%|███████▉  | 7681/9708 [00:37<00:10, 187.52batch/s]

Training batches on cuda:  79%|███████▉  | 7702/9708 [00:37<00:10, 193.52batch/s]

Training batches on cuda:  80%|███████▉  | 7722/9708 [00:37<00:10, 193.61batch/s]

Training batches on cuda:  80%|███████▉  | 7745/9708 [00:37<00:09, 201.54batch/s]

Training batches on cuda:  80%|████████  | 7768/9708 [00:37<00:09, 207.13batch/s]

Training batches on cuda:  80%|████████  | 7791/9708 [00:37<00:08, 213.04batch/s]

Training batches on cuda:  81%|████████  | 7815/9708 [00:37<00:08, 218.47batch/s]

Training batches on cuda:  81%|████████  | 7839/9708 [00:38<00:08, 221.96batch/s]

Training batches on cuda:  81%|████████  | 7862/9708 [00:38<00:08, 223.38batch/s]

Training batches on cuda:  81%|████████  | 7885/9708 [00:38<00:08, 219.25batch/s]

Training batches on cuda:  81%|████████▏ | 7908/9708 [00:38<00:08, 221.44batch/s]

Training batches on cuda:  82%|████████▏ | 7933/9708 [00:38<00:07, 228.31batch/s]

Training batches on cuda:  82%|████████▏ | 7957/9708 [00:38<00:07, 231.32batch/s]

Training batches on cuda:  82%|████████▏ | 7981/9708 [00:38<00:07, 224.58batch/s]

Training batches on cuda:  82%|████████▏ | 8004/9708 [00:38<00:07, 221.76batch/s]

Training batches on cuda:  83%|████████▎ | 8027/9708 [00:38<00:07, 215.49batch/s]

Training batches on cuda:  83%|████████▎ | 8049/9708 [00:38<00:07, 210.43batch/s]

Training batches on cuda:  83%|████████▎ | 8071/9708 [00:39<00:07, 212.43batch/s]

Training batches on cuda:  83%|████████▎ | 8093/9708 [00:39<00:07, 207.02batch/s]

Training batches on cuda:  84%|████████▎ | 8114/9708 [00:39<00:07, 206.47batch/s]

Training batches on cuda:  84%|████████▍ | 8137/9708 [00:39<00:07, 212.84batch/s]

Training batches on cuda:  84%|████████▍ | 8161/9708 [00:39<00:07, 220.02batch/s]

Training batches on cuda:  84%|████████▍ | 8185/9708 [00:39<00:06, 223.71batch/s]

Training batches on cuda:  85%|████████▍ | 8210/9708 [00:39<00:06, 228.89batch/s]

Training batches on cuda:  85%|████████▍ | 8233/9708 [00:39<00:06, 229.15batch/s]

Training batches on cuda:  85%|████████▌ | 8256/9708 [00:39<00:06, 227.68batch/s]

Training batches on cuda:  85%|████████▌ | 8279/9708 [00:39<00:06, 228.05batch/s]

Training batches on cuda:  86%|████████▌ | 8302/9708 [00:40<00:06, 227.14batch/s]

Training batches on cuda:  86%|████████▌ | 8327/9708 [00:40<00:05, 231.79batch/s]

Training batches on cuda:  86%|████████▌ | 8351/9708 [00:40<00:05, 226.90batch/s]

Training batches on cuda:  86%|████████▋ | 8374/9708 [00:40<00:06, 211.91batch/s]

Training batches on cuda:  86%|████████▋ | 8396/9708 [00:40<00:06, 210.26batch/s]

Training batches on cuda:  87%|████████▋ | 8418/9708 [00:40<00:06, 210.81batch/s]

Training batches on cuda:  87%|████████▋ | 8440/9708 [00:40<00:06, 207.96batch/s]

Training batches on cuda:  87%|████████▋ | 8462/9708 [00:40<00:05, 209.64batch/s]

Training batches on cuda:  87%|████████▋ | 8485/9708 [00:40<00:05, 215.19batch/s]

Training batches on cuda:  88%|████████▊ | 8508/9708 [00:41<00:05, 219.00batch/s]

Training batches on cuda:  88%|████████▊ | 8531/9708 [00:41<00:05, 222.03batch/s]

Training batches on cuda:  88%|████████▊ | 8554/9708 [00:41<00:05, 224.26batch/s]

Training batches on cuda:  88%|████████▊ | 8577/9708 [00:41<00:05, 225.73batch/s]

Training batches on cuda:  89%|████████▊ | 8600/9708 [00:41<00:04, 226.76batch/s]

Training batches on cuda:  89%|████████▉ | 8623/9708 [00:41<00:04, 225.42batch/s]

Training batches on cuda:  89%|████████▉ | 8646/9708 [00:41<00:04, 223.31batch/s]

Training batches on cuda:  89%|████████▉ | 8670/9708 [00:41<00:04, 225.54batch/s]

Training batches on cuda:  90%|████████▉ | 8693/9708 [00:41<00:04, 226.28batch/s]

Training batches on cuda:  90%|████████▉ | 8716/9708 [00:41<00:04, 227.30batch/s]

Training batches on cuda:  90%|█████████ | 8739/9708 [00:42<00:04, 228.08batch/s]

Training batches on cuda:  90%|█████████ | 8762/9708 [00:42<00:04, 228.27batch/s]

Training batches on cuda:  90%|█████████ | 8785/9708 [00:42<00:04, 228.75batch/s]

Training batches on cuda:  91%|█████████ | 8808/9708 [00:42<00:03, 228.47batch/s]

Training batches on cuda:  91%|█████████ | 8832/9708 [00:42<00:03, 228.47batch/s]

Training batches on cuda:  91%|█████████ | 8855/9708 [00:42<00:03, 220.22batch/s]

Training batches on cuda:  91%|█████████▏| 8878/9708 [00:42<00:03, 214.21batch/s]

Training batches on cuda:  92%|█████████▏| 8900/9708 [00:42<00:03, 215.91batch/s]

Training batches on cuda:  92%|█████████▏| 8922/9708 [00:42<00:03, 209.41batch/s]

Training batches on cuda:  92%|█████████▏| 8944/9708 [00:43<00:03, 199.66batch/s]

Training batches on cuda:  92%|█████████▏| 8965/9708 [00:43<00:03, 199.91batch/s]

Training batches on cuda:  93%|█████████▎| 8986/9708 [00:43<00:03, 200.15batch/s]

Training batches on cuda:  93%|█████████▎| 9008/9708 [00:43<00:03, 204.00batch/s]

Training batches on cuda:  93%|█████████▎| 9029/9708 [00:43<00:03, 204.15batch/s]

Training batches on cuda:  93%|█████████▎| 9052/9708 [00:43<00:03, 210.33batch/s]

Training batches on cuda:  94%|█████████▎| 9079/9708 [00:43<00:02, 224.01batch/s]

Training batches on cuda:  94%|█████████▍| 9105/9708 [00:43<00:02, 231.59batch/s]

Training batches on cuda:  94%|█████████▍| 9129/9708 [00:43<00:02, 228.25batch/s]

Training batches on cuda:  94%|█████████▍| 9153/9708 [00:43<00:02, 228.36batch/s]

Training batches on cuda:  95%|█████████▍| 9176/9708 [00:44<00:02, 225.53batch/s]

Training batches on cuda:  95%|█████████▍| 9199/9708 [00:44<00:02, 222.62batch/s]

Training batches on cuda:  95%|█████████▍| 9222/9708 [00:44<00:02, 219.20batch/s]

Training batches on cuda:  95%|█████████▌| 9244/9708 [00:44<00:02, 215.50batch/s]

Training batches on cuda:  95%|█████████▌| 9267/9708 [00:44<00:02, 219.08batch/s]

Training batches on cuda:  96%|█████████▌| 9289/9708 [00:44<00:01, 217.73batch/s]

Training batches on cuda:  96%|█████████▌| 9312/9708 [00:44<00:01, 219.91batch/s]

Training batches on cuda:  96%|█████████▌| 9335/9708 [00:44<00:01, 219.88batch/s]

Training batches on cuda:  96%|█████████▋| 9358/9708 [00:44<00:01, 221.65batch/s]

Training batches on cuda:  97%|█████████▋| 9381/9708 [00:45<00:01, 214.29batch/s]

Training batches on cuda:  97%|█████████▋| 9403/9708 [00:45<00:01, 213.32batch/s]

Training batches on cuda:  97%|█████████▋| 9426/9708 [00:45<00:01, 215.79batch/s]

Training batches on cuda:  97%|█████████▋| 9448/9708 [00:45<00:01, 212.91batch/s]

Training batches on cuda:  98%|█████████▊| 9471/9708 [00:45<00:01, 215.81batch/s]

Training batches on cuda:  98%|█████████▊| 9494/9708 [00:45<00:00, 217.77batch/s]

Training batches on cuda:  98%|█████████▊| 9516/9708 [00:45<00:00, 213.19batch/s]

Training batches on cuda:  98%|█████████▊| 9538/9708 [00:45<00:00, 212.59batch/s]

Training batches on cuda:  98%|█████████▊| 9560/9708 [00:45<00:00, 208.94batch/s]

Training batches on cuda:  99%|█████████▊| 9581/9708 [00:45<00:00, 206.38batch/s]

Training batches on cuda:  99%|█████████▉| 9603/9708 [00:46<00:00, 208.25batch/s]

Training batches on cuda:  99%|█████████▉| 9624/9708 [00:46<00:00, 207.28batch/s]

Training batches on cuda:  99%|█████████▉| 9645/9708 [00:46<00:00, 207.44batch/s]

Training batches on cuda: 100%|█████████▉| 9666/9708 [00:46<00:00, 206.23batch/s]

Training batches on cuda: 100%|█████████▉| 9687/9708 [00:46<00:00, 203.97batch/s]

Training batches on cuda: 100%|██████████| 9708/9708 [00:46<00:00, 193.17batch/s]

                                                                                 
Training epochs on cuda:  20%|██        | 2/10 [02:20<06:16, 47.06s/epoch, loss=0.0417, prev_loss=0.0417]
Training epochs on cuda:  30%|███       | 3/10 [02:20<05:28, 46.99s/epoch, loss=0.0417, prev_loss=0.0417]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 14/9708 [00:00<01:11, 135.89batch/s]

Training batches on cuda:   0%|          | 36/9708 [00:00<01:03, 152.48batch/s]

Training batches on cuda:   1%|          | 59/9708 [00:00<00:56, 169.56batch/s]

Training batches on cuda:   1%|          | 82/9708 [00:00<00:52, 183.78batch/s]

Training batches on cuda:   1%|          | 105/9708 [00:00<00:49, 194.00batch/s]

Training batches on cuda:   1%|▏         | 128/9708 [00:00<00:47, 202.34batch/s]

Training batches on cuda:   2%|▏         | 148/9708 [00:00<00:48, 198.39batch/s]

Training batches on cuda:   2%|▏         | 169/9708 [00:00<00:47, 200.90batch/s]

Training batches on cuda:   2%|▏         | 191/9708 [00:00<00:46, 203.70batch/s]

Training batches on cuda:   2%|▏         | 213/9708 [00:01<00:46, 205.72batch/s]

Training batches on cuda:   2%|▏         | 235/9708 [00:01<00:45, 207.95batch/s]

Training batches on cuda:   3%|▎         | 257/9708 [00:01<00:45, 209.85batch/s]

Training batches on cuda:   3%|▎         | 279/9708 [00:01<00:44, 211.30batch/s]

Training batches on cuda:   3%|▎         | 301/9708 [00:01<00:44, 211.29batch/s]

Training batches on cuda:   3%|▎         | 323/9708 [00:01<00:44, 211.17batch/s]

Training batches on cuda:   4%|▎         | 346/9708 [00:01<00:43, 215.23batch/s]

Training batches on cuda:   4%|▍         | 369/9708 [00:01<00:42, 219.44batch/s]

Training batches on cuda:   4%|▍         | 394/9708 [00:01<00:41, 225.78batch/s]

Training batches on cuda:   4%|▍         | 422/9708 [00:01<00:39, 237.57batch/s]

Training batches on cuda:   5%|▍         | 446/9708 [00:02<00:39, 234.93batch/s]

Training batches on cuda:   5%|▍         | 470/9708 [00:02<00:39, 234.38batch/s]

Training batches on cuda:   5%|▌         | 494/9708 [00:02<00:39, 233.21batch/s]

Training batches on cuda:   5%|▌         | 518/9708 [00:02<00:39, 232.65batch/s]

Training batches on cuda:   6%|▌         | 542/9708 [00:02<00:39, 230.66batch/s]

Training batches on cuda:   6%|▌         | 566/9708 [00:02<00:39, 230.50batch/s]

Training batches on cuda:   6%|▌         | 590/9708 [00:02<00:39, 229.01batch/s]

Training batches on cuda:   6%|▋         | 613/9708 [00:02<00:40, 226.18batch/s]

Training batches on cuda:   7%|▋         | 640/9708 [00:02<00:38, 236.95batch/s]

Training batches on cuda:   7%|▋         | 668/9708 [00:02<00:36, 246.39batch/s]

Training batches on cuda:   7%|▋         | 693/9708 [00:03<00:37, 239.30batch/s]

Training batches on cuda:   7%|▋         | 718/9708 [00:03<00:38, 235.10batch/s]

Training batches on cuda:   8%|▊         | 742/9708 [00:03<00:38, 234.70batch/s]

Training batches on cuda:   8%|▊         | 766/9708 [00:03<00:39, 228.82batch/s]

Training batches on cuda:   8%|▊         | 793/9708 [00:03<00:37, 237.46batch/s]

Training batches on cuda:   8%|▊         | 820/9708 [00:03<00:36, 244.45batch/s]

Training batches on cuda:   9%|▊         | 847/9708 [00:03<00:35, 251.28batch/s]

Training batches on cuda:   9%|▉         | 874/9708 [00:03<00:34, 255.94batch/s]

Training batches on cuda:   9%|▉         | 901/9708 [00:03<00:33, 259.27batch/s]

Training batches on cuda:  10%|▉         | 929/9708 [00:04<00:33, 262.63batch/s]

Training batches on cuda:  10%|▉         | 957/9708 [00:04<00:33, 265.02batch/s]

Training batches on cuda:  10%|█         | 984/9708 [00:04<00:36, 238.43batch/s]

Training batches on cuda:  10%|█         | 1012/9708 [00:04<00:35, 247.83batch/s]

Training batches on cuda:  11%|█         | 1039/9708 [00:04<00:34, 251.48batch/s]

Training batches on cuda:  11%|█         | 1065/9708 [00:04<00:36, 239.70batch/s]

Training batches on cuda:  11%|█         | 1090/9708 [00:04<00:36, 237.00batch/s]

Training batches on cuda:  11%|█▏        | 1114/9708 [00:04<00:36, 235.03batch/s]

Training batches on cuda:  12%|█▏        | 1138/9708 [00:04<00:36, 233.37batch/s]

Training batches on cuda:  12%|█▏        | 1162/9708 [00:05<00:36, 232.89batch/s]

Training batches on cuda:  12%|█▏        | 1186/9708 [00:05<00:36, 231.57batch/s]

Training batches on cuda:  12%|█▏        | 1210/9708 [00:05<00:36, 233.43batch/s]

Training batches on cuda:  13%|█▎        | 1234/9708 [00:05<00:36, 232.06batch/s]

Training batches on cuda:  13%|█▎        | 1258/9708 [00:05<00:36, 232.67batch/s]

Training batches on cuda:  13%|█▎        | 1282/9708 [00:05<00:36, 232.92batch/s]

Training batches on cuda:  13%|█▎        | 1306/9708 [00:05<00:36, 232.83batch/s]

Training batches on cuda:  14%|█▎        | 1330/9708 [00:05<00:36, 231.40batch/s]

Training batches on cuda:  14%|█▍        | 1354/9708 [00:05<00:36, 231.27batch/s]

Training batches on cuda:  14%|█▍        | 1378/9708 [00:05<00:36, 229.81batch/s]

Training batches on cuda:  14%|█▍        | 1401/9708 [00:06<00:37, 221.46batch/s]

Training batches on cuda:  15%|█▍        | 1424/9708 [00:06<00:37, 218.59batch/s]

Training batches on cuda:  15%|█▍        | 1446/9708 [00:06<00:38, 216.49batch/s]

Training batches on cuda:  15%|█▌        | 1468/9708 [00:06<00:39, 209.20batch/s]

Training batches on cuda:  15%|█▌        | 1490/9708 [00:06<00:39, 208.16batch/s]

Training batches on cuda:  16%|█▌        | 1511/9708 [00:06<00:39, 207.46batch/s]

Training batches on cuda:  16%|█▌        | 1532/9708 [00:06<00:39, 207.52batch/s]

Training batches on cuda:  16%|█▌        | 1553/9708 [00:06<00:39, 204.14batch/s]

Training batches on cuda:  16%|█▌        | 1574/9708 [00:06<00:40, 201.98batch/s]

Training batches on cuda:  16%|█▋        | 1595/9708 [00:07<00:39, 203.64batch/s]

Training batches on cuda:  17%|█▋        | 1616/9708 [00:07<00:40, 197.86batch/s]

Training batches on cuda:  17%|█▋        | 1636/9708 [00:07<00:40, 198.21batch/s]

Training batches on cuda:  17%|█▋        | 1658/9708 [00:07<00:39, 202.34batch/s]

Training batches on cuda:  17%|█▋        | 1681/9708 [00:07<00:38, 208.16batch/s]

Training batches on cuda:  18%|█▊        | 1704/9708 [00:07<00:37, 212.51batch/s]

Training batches on cuda:  18%|█▊        | 1727/9708 [00:07<00:37, 215.39batch/s]

Training batches on cuda:  18%|█▊        | 1750/9708 [00:07<00:36, 218.35batch/s]

Training batches on cuda:  18%|█▊        | 1772/9708 [00:07<00:37, 210.14batch/s]

Training batches on cuda:  18%|█▊        | 1794/9708 [00:07<00:38, 205.88batch/s]

Training batches on cuda:  19%|█▊        | 1815/9708 [00:08<00:38, 205.49batch/s]

Training batches on cuda:  19%|█▉        | 1838/9708 [00:08<00:37, 211.02batch/s]

Training batches on cuda:  19%|█▉        | 1861/9708 [00:08<00:36, 215.65batch/s]

Training batches on cuda:  19%|█▉        | 1884/9708 [00:08<00:35, 219.42batch/s]

Training batches on cuda:  20%|█▉        | 1907/9708 [00:08<00:35, 221.34batch/s]

Training batches on cuda:  20%|█▉        | 1930/9708 [00:08<00:34, 222.85batch/s]

Training batches on cuda:  20%|██        | 1953/9708 [00:08<00:34, 224.19batch/s]

Training batches on cuda:  20%|██        | 1976/9708 [00:08<00:34, 225.26batch/s]

Training batches on cuda:  21%|██        | 1999/9708 [00:08<00:34, 225.38batch/s]

Training batches on cuda:  21%|██        | 2022/9708 [00:08<00:33, 226.45batch/s]

Training batches on cuda:  21%|██        | 2045/9708 [00:09<00:33, 226.66batch/s]

Training batches on cuda:  21%|██▏       | 2068/9708 [00:09<00:33, 226.77batch/s]

Training batches on cuda:  22%|██▏       | 2091/9708 [00:09<00:35, 216.99batch/s]

Training batches on cuda:  22%|██▏       | 2114/9708 [00:09<00:34, 220.15batch/s]

Training batches on cuda:  22%|██▏       | 2137/9708 [00:09<00:34, 221.75batch/s]

Training batches on cuda:  22%|██▏       | 2160/9708 [00:09<00:35, 214.19batch/s]

Training batches on cuda:  22%|██▏       | 2182/9708 [00:09<00:35, 210.04batch/s]

Training batches on cuda:  23%|██▎       | 2204/9708 [00:09<00:36, 207.69batch/s]

Training batches on cuda:  23%|██▎       | 2225/9708 [00:09<00:36, 205.93batch/s]

Training batches on cuda:  23%|██▎       | 2246/9708 [00:10<00:36, 203.88batch/s]

Training batches on cuda:  23%|██▎       | 2267/9708 [00:10<00:37, 199.13batch/s]

Training batches on cuda:  24%|██▎       | 2287/9708 [00:10<00:37, 198.35batch/s]

Training batches on cuda:  24%|██▍       | 2308/9708 [00:10<00:37, 199.53batch/s]

Training batches on cuda:  24%|██▍       | 2329/9708 [00:10<00:36, 202.31batch/s]

Training batches on cuda:  24%|██▍       | 2351/9708 [00:10<00:35, 206.16batch/s]

Training batches on cuda:  24%|██▍       | 2372/9708 [00:10<00:36, 202.82batch/s]

Training batches on cuda:  25%|██▍       | 2393/9708 [00:10<00:35, 203.82batch/s]

Training batches on cuda:  25%|██▍       | 2415/9708 [00:10<00:35, 207.67batch/s]

Training batches on cuda:  25%|██▌       | 2437/9708 [00:10<00:34, 209.98batch/s]

Training batches on cuda:  25%|██▌       | 2461/9708 [00:11<00:33, 217.99batch/s]

Training batches on cuda:  26%|██▌       | 2486/9708 [00:11<00:31, 225.93batch/s]

Training batches on cuda:  26%|██▌       | 2511/9708 [00:11<00:31, 231.40batch/s]

Training batches on cuda:  26%|██▌       | 2538/9708 [00:11<00:29, 241.54batch/s]

Training batches on cuda:  26%|██▋       | 2566/9708 [00:11<00:28, 250.04batch/s]

Training batches on cuda:  27%|██▋       | 2594/9708 [00:11<00:27, 256.14batch/s]

Training batches on cuda:  27%|██▋       | 2622/9708 [00:11<00:27, 260.86batch/s]

Training batches on cuda:  27%|██▋       | 2649/9708 [00:11<00:27, 252.72batch/s]

Training batches on cuda:  28%|██▊       | 2675/9708 [00:11<00:29, 241.23batch/s]

Training batches on cuda:  28%|██▊       | 2700/9708 [00:12<00:31, 219.02batch/s]

Training batches on cuda:  28%|██▊       | 2723/9708 [00:12<00:34, 204.19batch/s]

Training batches on cuda:  28%|██▊       | 2745/9708 [00:12<00:35, 196.94batch/s]

Training batches on cuda:  28%|██▊       | 2766/9708 [00:12<00:36, 192.30batch/s]

Training batches on cuda:  29%|██▊       | 2788/9708 [00:12<00:34, 198.73batch/s]

Training batches on cuda:  29%|██▉       | 2811/9708 [00:12<00:33, 204.76batch/s]

Training batches on cuda:  29%|██▉       | 2832/9708 [00:12<00:33, 203.92batch/s]

Training batches on cuda:  29%|██▉       | 2855/9708 [00:12<00:32, 209.56batch/s]

Training batches on cuda:  30%|██▉       | 2877/9708 [00:12<00:32, 207.19batch/s]

Training batches on cuda:  30%|██▉       | 2898/9708 [00:13<00:32, 207.81batch/s]

Training batches on cuda:  30%|███       | 2920/9708 [00:13<00:32, 209.81batch/s]

Training batches on cuda:  30%|███       | 2942/9708 [00:13<00:32, 209.68batch/s]

Training batches on cuda:  31%|███       | 2964/9708 [00:13<00:32, 210.64batch/s]

Training batches on cuda:  31%|███       | 2986/9708 [00:13<00:31, 210.78batch/s]

Training batches on cuda:  31%|███       | 3008/9708 [00:13<00:31, 210.71batch/s]

Training batches on cuda:  31%|███       | 3030/9708 [00:13<00:31, 211.58batch/s]

Training batches on cuda:  31%|███▏      | 3052/9708 [00:13<00:31, 210.26batch/s]

Training batches on cuda:  32%|███▏      | 3074/9708 [00:13<00:31, 209.40batch/s]

Training batches on cuda:  32%|███▏      | 3095/9708 [00:14<00:31, 209.27batch/s]

Training batches on cuda:  32%|███▏      | 3117/9708 [00:14<00:31, 210.79batch/s]

Training batches on cuda:  32%|███▏      | 3139/9708 [00:14<00:31, 210.26batch/s]

Training batches on cuda:  33%|███▎      | 3161/9708 [00:14<00:35, 182.87batch/s]

Training batches on cuda:  33%|███▎      | 3185/9708 [00:14<00:33, 195.14batch/s]

Training batches on cuda:  33%|███▎      | 3207/9708 [00:14<00:32, 200.38batch/s]

Training batches on cuda:  33%|███▎      | 3228/9708 [00:14<00:33, 194.74batch/s]

Training batches on cuda:  33%|███▎      | 3248/9708 [00:14<00:34, 186.76batch/s]

Training batches on cuda:  34%|███▎      | 3268/9708 [00:14<00:35, 182.33batch/s]

Training batches on cuda:  34%|███▍      | 3287/9708 [00:15<00:35, 181.55batch/s]

Training batches on cuda:  34%|███▍      | 3306/9708 [00:15<00:35, 182.84batch/s]

Training batches on cuda:  34%|███▍      | 3326/9708 [00:15<00:34, 187.67batch/s]

Training batches on cuda:  34%|███▍      | 3346/9708 [00:15<00:33, 190.37batch/s]

Training batches on cuda:  35%|███▍      | 3366/9708 [00:15<00:33, 188.31batch/s]

Training batches on cuda:  35%|███▍      | 3385/9708 [00:15<00:33, 186.16batch/s]

Training batches on cuda:  35%|███▌      | 3406/9708 [00:15<00:32, 191.95batch/s]

Training batches on cuda:  35%|███▌      | 3426/9708 [00:15<00:32, 191.02batch/s]

Training batches on cuda:  36%|███▌      | 3447/9708 [00:15<00:32, 194.26batch/s]

Training batches on cuda:  36%|███▌      | 3468/9708 [00:15<00:31, 196.37batch/s]

Training batches on cuda:  36%|███▌      | 3489/9708 [00:16<00:31, 199.53batch/s]

Training batches on cuda:  36%|███▌      | 3509/9708 [00:16<00:31, 196.87batch/s]

Training batches on cuda:  36%|███▋      | 3531/9708 [00:16<00:30, 201.06batch/s]

Training batches on cuda:  37%|███▋      | 3552/9708 [00:16<00:30, 201.70batch/s]

Training batches on cuda:  37%|███▋      | 3574/9708 [00:16<00:29, 206.07batch/s]

Training batches on cuda:  37%|███▋      | 3595/9708 [00:16<00:30, 199.22batch/s]

Training batches on cuda:  37%|███▋      | 3616/9708 [00:16<00:30, 200.62batch/s]

Training batches on cuda:  37%|███▋      | 3637/9708 [00:16<00:31, 194.24batch/s]

Training batches on cuda:  38%|███▊      | 3658/9708 [00:16<00:30, 196.53batch/s]

Training batches on cuda:  38%|███▊      | 3678/9708 [00:17<00:31, 188.87batch/s]

Training batches on cuda:  38%|███▊      | 3699/9708 [00:17<00:31, 192.36batch/s]

Training batches on cuda:  38%|███▊      | 3719/9708 [00:17<00:31, 190.86batch/s]

Training batches on cuda:  39%|███▊      | 3741/9708 [00:17<00:30, 196.75batch/s]

Training batches on cuda:  39%|███▉      | 3764/9708 [00:17<00:29, 204.13batch/s]

Training batches on cuda:  39%|███▉      | 3785/9708 [00:17<00:29, 202.07batch/s]

Training batches on cuda:  39%|███▉      | 3808/9708 [00:17<00:28, 208.71batch/s]

Training batches on cuda:  39%|███▉      | 3829/9708 [00:17<00:28, 204.11batch/s]

Training batches on cuda:  40%|███▉      | 3850/9708 [00:17<00:29, 201.26batch/s]

Training batches on cuda:  40%|███▉      | 3871/9708 [00:17<00:29, 197.57batch/s]

Training batches on cuda:  40%|████      | 3893/9708 [00:18<00:28, 202.47batch/s]

Training batches on cuda:  40%|████      | 3914/9708 [00:18<00:29, 198.20batch/s]

Training batches on cuda:  41%|████      | 3934/9708 [00:18<00:30, 192.07batch/s]

Training batches on cuda:  41%|████      | 3954/9708 [00:18<00:30, 190.04batch/s]

Training batches on cuda:  41%|████      | 3974/9708 [00:18<00:30, 188.26batch/s]

Training batches on cuda:  41%|████      | 3993/9708 [00:18<00:30, 188.01batch/s]

Training batches on cuda:  41%|████▏     | 4015/9708 [00:18<00:29, 195.03batch/s]

Training batches on cuda:  42%|████▏     | 4036/9708 [00:18<00:28, 196.98batch/s]

Training batches on cuda:  42%|████▏     | 4057/9708 [00:18<00:28, 198.65batch/s]

Training batches on cuda:  42%|████▏     | 4080/9708 [00:19<00:27, 204.79batch/s]

Training batches on cuda:  42%|████▏     | 4101/9708 [00:19<00:27, 200.92batch/s]

Training batches on cuda:  42%|████▏     | 4124/9708 [00:19<00:26, 207.24batch/s]

Training batches on cuda:  43%|████▎     | 4145/9708 [00:19<00:28, 191.89batch/s]

Training batches on cuda:  43%|████▎     | 4166/9708 [00:19<00:28, 195.44batch/s]

Training batches on cuda:  43%|████▎     | 4190/9708 [00:19<00:26, 206.80batch/s]

Training batches on cuda:  43%|████▎     | 4216/9708 [00:19<00:25, 218.36batch/s]

Training batches on cuda:  44%|████▎     | 4242/9708 [00:19<00:24, 227.62batch/s]

Training batches on cuda:  44%|████▍     | 4266/9708 [00:19<00:23, 228.28batch/s]

Training batches on cuda:  44%|████▍     | 4290/9708 [00:20<00:24, 219.84batch/s]

Training batches on cuda:  44%|████▍     | 4313/9708 [00:20<00:24, 215.83batch/s]

Training batches on cuda:  45%|████▍     | 4335/9708 [00:20<00:25, 210.72batch/s]

Training batches on cuda:  45%|████▍     | 4357/9708 [00:20<00:25, 212.23batch/s]

Training batches on cuda:  45%|████▌     | 4379/9708 [00:20<00:25, 212.93batch/s]

Training batches on cuda:  45%|████▌     | 4401/9708 [00:20<00:26, 201.02batch/s]

Training batches on cuda:  46%|████▌     | 4422/9708 [00:20<00:26, 200.26batch/s]

Training batches on cuda:  46%|████▌     | 4443/9708 [00:20<00:26, 201.72batch/s]

Training batches on cuda:  46%|████▌     | 4464/9708 [00:20<00:26, 201.67batch/s]

Training batches on cuda:  46%|████▌     | 4485/9708 [00:20<00:26, 198.61batch/s]

Training batches on cuda:  46%|████▋     | 4506/9708 [00:21<00:25, 201.72batch/s]

Training batches on cuda:  47%|████▋     | 4529/9708 [00:21<00:24, 207.45batch/s]

Training batches on cuda:  47%|████▋     | 4551/9708 [00:21<00:24, 210.48batch/s]

Training batches on cuda:  47%|████▋     | 4574/9708 [00:21<00:23, 214.04batch/s]

Training batches on cuda:  47%|████▋     | 4596/9708 [00:21<00:23, 213.86batch/s]

Training batches on cuda:  48%|████▊     | 4620/9708 [00:21<00:23, 218.54batch/s]

Training batches on cuda:  48%|████▊     | 4642/9708 [00:21<00:23, 213.90batch/s]

Training batches on cuda:  48%|████▊     | 4665/9708 [00:21<00:23, 215.96batch/s]

Training batches on cuda:  48%|████▊     | 4687/9708 [00:21<00:23, 213.53batch/s]

Training batches on cuda:  49%|████▊     | 4709/9708 [00:22<00:23, 212.12batch/s]

Training batches on cuda:  49%|████▊     | 4732/9708 [00:22<00:23, 216.01batch/s]

Training batches on cuda:  49%|████▉     | 4757/9708 [00:22<00:22, 223.17batch/s]

Training batches on cuda:  49%|████▉     | 4780/9708 [00:22<00:22, 221.38batch/s]

Training batches on cuda:  49%|████▉     | 4803/9708 [00:22<00:22, 220.19batch/s]

Training batches on cuda:  50%|████▉     | 4826/9708 [00:22<00:22, 218.08batch/s]

Training batches on cuda:  50%|████▉     | 4848/9708 [00:22<00:22, 217.83batch/s]

Training batches on cuda:  50%|█████     | 4871/9708 [00:22<00:22, 219.14batch/s]

Training batches on cuda:  50%|█████     | 4893/9708 [00:22<00:22, 217.48batch/s]

Training batches on cuda:  51%|█████     | 4916/9708 [00:22<00:21, 220.78batch/s]

Training batches on cuda:  51%|█████     | 4939/9708 [00:23<00:21, 217.90batch/s]

Training batches on cuda:  51%|█████     | 4962/9708 [00:23<00:21, 220.05batch/s]

Training batches on cuda:  51%|█████▏    | 4985/9708 [00:23<00:21, 220.06batch/s]

Training batches on cuda:  52%|█████▏    | 5008/9708 [00:23<00:21, 216.98batch/s]

Training batches on cuda:  52%|█████▏    | 5030/9708 [00:23<00:21, 216.75batch/s]

Training batches on cuda:  52%|█████▏    | 5054/9708 [00:23<00:21, 221.53batch/s]

Training batches on cuda:  52%|█████▏    | 5077/9708 [00:23<00:21, 216.13batch/s]

Training batches on cuda:  53%|█████▎    | 5099/9708 [00:23<00:21, 213.04batch/s]

Training batches on cuda:  53%|█████▎    | 5122/9708 [00:23<00:21, 217.65batch/s]

Training batches on cuda:  53%|█████▎    | 5145/9708 [00:24<00:20, 220.07batch/s]

Training batches on cuda:  53%|█████▎    | 5168/9708 [00:24<00:20, 222.33batch/s]

Training batches on cuda:  53%|█████▎    | 5191/9708 [00:24<00:20, 222.26batch/s]

Training batches on cuda:  54%|█████▎    | 5216/9708 [00:24<00:19, 227.52batch/s]

Training batches on cuda:  54%|█████▍    | 5239/9708 [00:24<00:22, 200.97batch/s]

Training batches on cuda:  54%|█████▍    | 5260/9708 [00:24<00:22, 201.81batch/s]

Training batches on cuda:  54%|█████▍    | 5282/9708 [00:24<00:21, 205.76batch/s]

Training batches on cuda:  55%|█████▍    | 5307/9708 [00:24<00:20, 215.33batch/s]

Training batches on cuda:  55%|█████▍    | 5333/9708 [00:24<00:19, 224.98batch/s]

Training batches on cuda:  55%|█████▌    | 5358/9708 [00:24<00:18, 230.01batch/s]

Training batches on cuda:  55%|█████▌    | 5384/9708 [00:25<00:18, 236.52batch/s]

Training batches on cuda:  56%|█████▌    | 5408/9708 [00:25<00:19, 223.42batch/s]

Training batches on cuda:  56%|█████▌    | 5431/9708 [00:25<00:20, 210.60batch/s]

Training batches on cuda:  56%|█████▌    | 5453/9708 [00:25<00:21, 199.86batch/s]

Training batches on cuda:  56%|█████▋    | 5474/9708 [00:25<00:21, 197.88batch/s]

Training batches on cuda:  57%|█████▋    | 5495/9708 [00:25<00:21, 198.60batch/s]

Training batches on cuda:  57%|█████▋    | 5516/9708 [00:25<00:21, 194.47batch/s]

Training batches on cuda:  57%|█████▋    | 5536/9708 [00:25<00:21, 193.87batch/s]

Training batches on cuda:  57%|█████▋    | 5556/9708 [00:25<00:21, 190.89batch/s]

Training batches on cuda:  57%|█████▋    | 5577/9708 [00:26<00:21, 193.90batch/s]

Training batches on cuda:  58%|█████▊    | 5598/9708 [00:26<00:20, 195.73batch/s]

Training batches on cuda:  58%|█████▊    | 5618/9708 [00:26<00:20, 195.70batch/s]

Training batches on cuda:  58%|█████▊    | 5638/9708 [00:26<00:20, 193.83batch/s]

Training batches on cuda:  58%|█████▊    | 5658/9708 [00:26<00:21, 190.21batch/s]

Training batches on cuda:  58%|█████▊    | 5678/9708 [00:26<00:21, 188.97batch/s]

Training batches on cuda:  59%|█████▊    | 5699/9708 [00:26<00:20, 192.88batch/s]

Training batches on cuda:  59%|█████▉    | 5719/9708 [00:26<00:20, 194.41batch/s]

Training batches on cuda:  59%|█████▉    | 5739/9708 [00:26<00:20, 194.51batch/s]

Training batches on cuda:  59%|█████▉    | 5760/9708 [00:27<00:19, 198.29batch/s]

Training batches on cuda:  60%|█████▉    | 5781/9708 [00:27<00:19, 200.58batch/s]

Training batches on cuda:  60%|█████▉    | 5802/9708 [00:27<00:19, 202.92batch/s]

Training batches on cuda:  60%|█████▉    | 5823/9708 [00:27<00:19, 202.02batch/s]

Training batches on cuda:  60%|██████    | 5844/9708 [00:27<00:18, 204.07batch/s]

Training batches on cuda:  60%|██████    | 5867/9708 [00:27<00:18, 209.16batch/s]

Training batches on cuda:  61%|██████    | 5888/9708 [00:27<00:18, 207.13batch/s]

Training batches on cuda:  61%|██████    | 5909/9708 [00:27<00:18, 203.13batch/s]

Training batches on cuda:  61%|██████    | 5931/9708 [00:27<00:18, 207.75batch/s]

Training batches on cuda:  61%|██████▏   | 5954/9708 [00:27<00:17, 212.97batch/s]

Training batches on cuda:  62%|██████▏   | 5979/9708 [00:28<00:16, 221.03batch/s]

Training batches on cuda:  62%|██████▏   | 6003/9708 [00:28<00:16, 223.74batch/s]

Training batches on cuda:  62%|██████▏   | 6026/9708 [00:28<00:17, 215.90batch/s]

Training batches on cuda:  62%|██████▏   | 6048/9708 [00:28<00:17, 211.64batch/s]

Training batches on cuda:  63%|██████▎   | 6070/9708 [00:28<00:17, 210.94batch/s]

Training batches on cuda:  63%|██████▎   | 6092/9708 [00:28<00:18, 195.96batch/s]

Training batches on cuda:  63%|██████▎   | 6115/9708 [00:28<00:17, 203.18batch/s]

Training batches on cuda:  63%|██████▎   | 6138/9708 [00:28<00:17, 209.41batch/s]

Training batches on cuda:  63%|██████▎   | 6160/9708 [00:28<00:17, 205.92batch/s]

Training batches on cuda:  64%|██████▎   | 6181/9708 [00:29<00:17, 204.71batch/s]

Training batches on cuda:  64%|██████▍   | 6204/9708 [00:29<00:16, 209.67batch/s]

Training batches on cuda:  64%|██████▍   | 6227/9708 [00:29<00:16, 212.82batch/s]

Training batches on cuda:  64%|██████▍   | 6249/9708 [00:29<00:16, 213.72batch/s]

Training batches on cuda:  65%|██████▍   | 6271/9708 [00:29<00:17, 199.99batch/s]

Training batches on cuda:  65%|██████▍   | 6292/9708 [00:29<00:16, 202.78batch/s]

Training batches on cuda:  65%|██████▌   | 6314/9708 [00:29<00:16, 205.77batch/s]

Training batches on cuda:  65%|██████▌   | 6335/9708 [00:29<00:16, 198.66batch/s]

Training batches on cuda:  65%|██████▌   | 6356/9708 [00:29<00:16, 198.20batch/s]

Training batches on cuda:  66%|██████▌   | 6377/9708 [00:30<00:16, 199.84batch/s]

Training batches on cuda:  66%|██████▌   | 6398/9708 [00:30<00:16, 200.92batch/s]

Training batches on cuda:  66%|██████▌   | 6421/9708 [00:30<00:15, 208.54batch/s]

Training batches on cuda:  66%|██████▋   | 6443/9708 [00:30<00:15, 211.37batch/s]

Training batches on cuda:  67%|██████▋   | 6465/9708 [00:30<00:15, 212.60batch/s]

Training batches on cuda:  67%|██████▋   | 6487/9708 [00:30<00:15, 201.47batch/s]

Training batches on cuda:  67%|██████▋   | 6510/9708 [00:30<00:15, 207.89batch/s]

Training batches on cuda:  67%|██████▋   | 6534/9708 [00:30<00:14, 214.13batch/s]

Training batches on cuda:  68%|██████▊   | 6556/9708 [00:30<00:14, 212.28batch/s]

Training batches on cuda:  68%|██████▊   | 6578/9708 [00:30<00:15, 208.09batch/s]

Training batches on cuda:  68%|██████▊   | 6599/9708 [00:31<00:15, 203.06batch/s]

Training batches on cuda:  68%|██████▊   | 6621/9708 [00:31<00:15, 204.78batch/s]

Training batches on cuda:  68%|██████▊   | 6642/9708 [00:31<00:15, 203.89batch/s]

Training batches on cuda:  69%|██████▊   | 6663/9708 [00:31<00:14, 204.01batch/s]

Training batches on cuda:  69%|██████▉   | 6685/9708 [00:31<00:14, 206.61batch/s]

Training batches on cuda:  69%|██████▉   | 6707/9708 [00:31<00:14, 208.43batch/s]

Training batches on cuda:  69%|██████▉   | 6729/9708 [00:31<00:14, 211.27batch/s]

Training batches on cuda:  70%|██████▉   | 6751/9708 [00:31<00:14, 207.70batch/s]

Training batches on cuda:  70%|██████▉   | 6772/9708 [00:31<00:14, 202.83batch/s]

Training batches on cuda:  70%|██████▉   | 6793/9708 [00:32<00:14, 203.10batch/s]

Training batches on cuda:  70%|███████   | 6816/9708 [00:32<00:13, 209.04batch/s]

Training batches on cuda:  70%|███████   | 6838/9708 [00:32<00:13, 211.32batch/s]

Training batches on cuda:  71%|███████   | 6861/9708 [00:32<00:13, 215.42batch/s]

Training batches on cuda:  71%|███████   | 6883/9708 [00:32<00:13, 207.71batch/s]

Training batches on cuda:  71%|███████   | 6904/9708 [00:32<00:13, 207.65batch/s]

Training batches on cuda:  71%|███████▏  | 6925/9708 [00:32<00:13, 204.91batch/s]

Training batches on cuda:  72%|███████▏  | 6947/9708 [00:32<00:13, 208.89batch/s]

Training batches on cuda:  72%|███████▏  | 6969/9708 [00:32<00:12, 210.72batch/s]

Training batches on cuda:  72%|███████▏  | 6991/9708 [00:32<00:13, 201.40batch/s]

Training batches on cuda:  72%|███████▏  | 7012/9708 [00:33<00:13, 198.94batch/s]

Training batches on cuda:  72%|███████▏  | 7032/9708 [00:33<00:13, 195.91batch/s]

Training batches on cuda:  73%|███████▎  | 7054/9708 [00:33<00:13, 201.20batch/s]

Training batches on cuda:  73%|███████▎  | 7075/9708 [00:33<00:13, 200.74batch/s]

Training batches on cuda:  73%|███████▎  | 7096/9708 [00:33<00:13, 199.66batch/s]

Training batches on cuda:  73%|███████▎  | 7117/9708 [00:33<00:13, 198.46batch/s]

Training batches on cuda:  74%|███████▎  | 7141/9708 [00:33<00:12, 208.13batch/s]

Training batches on cuda:  74%|███████▍  | 7162/9708 [00:33<00:12, 206.34batch/s]

Training batches on cuda:  74%|███████▍  | 7183/9708 [00:33<00:12, 204.20batch/s]

Training batches on cuda:  74%|███████▍  | 7204/9708 [00:34<00:12, 195.39batch/s]

Training batches on cuda:  74%|███████▍  | 7225/9708 [00:34<00:12, 199.24batch/s]

Training batches on cuda:  75%|███████▍  | 7246/9708 [00:34<00:12, 196.29batch/s]

Training batches on cuda:  75%|███████▍  | 7271/9708 [00:34<00:11, 208.63batch/s]

Training batches on cuda:  75%|███████▌  | 7296/9708 [00:34<00:11, 218.20batch/s]

Training batches on cuda:  75%|███████▌  | 7321/9708 [00:34<00:10, 224.68batch/s]

Training batches on cuda:  76%|███████▌  | 7346/9708 [00:34<00:10, 230.10batch/s]

Training batches on cuda:  76%|███████▌  | 7371/9708 [00:34<00:09, 233.76batch/s]

Training batches on cuda:  76%|███████▌  | 7395/9708 [00:34<00:10, 230.08batch/s]

Training batches on cuda:  76%|███████▋  | 7419/9708 [00:34<00:09, 231.42batch/s]

Training batches on cuda:  77%|███████▋  | 7443/9708 [00:35<00:09, 229.93batch/s]

Training batches on cuda:  77%|███████▋  | 7467/9708 [00:35<00:10, 207.77batch/s]

Training batches on cuda:  77%|███████▋  | 7491/9708 [00:35<00:10, 214.30batch/s]

Training batches on cuda:  77%|███████▋  | 7513/9708 [00:35<00:10, 206.95batch/s]

Training batches on cuda:  78%|███████▊  | 7535/9708 [00:35<00:10, 199.49batch/s]

Training batches on cuda:  78%|███████▊  | 7556/9708 [00:35<00:11, 184.47batch/s]

Training batches on cuda:  78%|███████▊  | 7575/9708 [00:35<00:11, 182.92batch/s]

Training batches on cuda:  78%|███████▊  | 7594/9708 [00:35<00:11, 182.08batch/s]

Training batches on cuda:  78%|███████▊  | 7615/9708 [00:35<00:11, 189.62batch/s]

Training batches on cuda:  79%|███████▊  | 7636/9708 [00:36<00:10, 193.36batch/s]

Training batches on cuda:  79%|███████▉  | 7656/9708 [00:36<00:10, 192.60batch/s]

Training batches on cuda:  79%|███████▉  | 7677/9708 [00:36<00:10, 195.94batch/s]

Training batches on cuda:  79%|███████▉  | 7697/9708 [00:36<00:10, 195.05batch/s]

Training batches on cuda:  80%|███████▉  | 7719/9708 [00:36<00:09, 200.15batch/s]

Training batches on cuda:  80%|███████▉  | 7741/9708 [00:36<00:09, 204.82batch/s]

Training batches on cuda:  80%|███████▉  | 7763/9708 [00:36<00:09, 206.44batch/s]

Training batches on cuda:  80%|████████  | 7785/9708 [00:36<00:09, 208.73batch/s]

Training batches on cuda:  80%|████████  | 7807/9708 [00:36<00:09, 210.04batch/s]

Training batches on cuda:  81%|████████  | 7829/9708 [00:37<00:08, 211.93batch/s]

Training batches on cuda:  81%|████████  | 7851/9708 [00:37<00:08, 212.33batch/s]

Training batches on cuda:  81%|████████  | 7873/9708 [00:37<00:09, 195.31batch/s]

Training batches on cuda:  81%|████████▏ | 7894/9708 [00:37<00:09, 197.38batch/s]

Training batches on cuda:  82%|████████▏ | 7914/9708 [00:37<00:09, 186.35batch/s]

Training batches on cuda:  82%|████████▏ | 7935/9708 [00:37<00:09, 191.78batch/s]

Training batches on cuda:  82%|████████▏ | 7957/9708 [00:37<00:08, 197.35batch/s]

Training batches on cuda:  82%|████████▏ | 7979/9708 [00:37<00:08, 202.24batch/s]

Training batches on cuda:  82%|████████▏ | 8002/9708 [00:37<00:08, 207.42batch/s]

Training batches on cuda:  83%|████████▎ | 8024/9708 [00:38<00:08, 208.94batch/s]

Training batches on cuda:  83%|████████▎ | 8047/9708 [00:38<00:07, 212.49batch/s]

Training batches on cuda:  83%|████████▎ | 8070/9708 [00:38<00:07, 216.04batch/s]

Training batches on cuda:  83%|████████▎ | 8093/9708 [00:38<00:07, 217.61batch/s]

Training batches on cuda:  84%|████████▎ | 8116/9708 [00:38<00:07, 219.53batch/s]

Training batches on cuda:  84%|████████▍ | 8138/9708 [00:38<00:07, 217.83batch/s]

Training batches on cuda:  84%|████████▍ | 8160/9708 [00:38<00:07, 205.97batch/s]

Training batches on cuda:  84%|████████▍ | 8184/9708 [00:38<00:07, 212.14batch/s]

Training batches on cuda:  85%|████████▍ | 8206/9708 [00:38<00:07, 204.20batch/s]

Training batches on cuda:  85%|████████▍ | 8228/9708 [00:38<00:07, 208.56batch/s]

Training batches on cuda:  85%|████████▍ | 8251/9708 [00:39<00:06, 212.90batch/s]

Training batches on cuda:  85%|████████▌ | 8274/9708 [00:39<00:06, 215.79batch/s]

Training batches on cuda:  85%|████████▌ | 8296/9708 [00:39<00:06, 210.57batch/s]

Training batches on cuda:  86%|████████▌ | 8318/9708 [00:39<00:06, 208.65batch/s]

Training batches on cuda:  86%|████████▌ | 8339/9708 [00:39<00:06, 203.70batch/s]

Training batches on cuda:  86%|████████▌ | 8360/9708 [00:39<00:06, 202.40batch/s]

Training batches on cuda:  86%|████████▋ | 8381/9708 [00:39<00:06, 204.58batch/s]

Training batches on cuda:  87%|████████▋ | 8402/9708 [00:39<00:06, 204.31batch/s]

Training batches on cuda:  87%|████████▋ | 8423/9708 [00:39<00:06, 205.24batch/s]

Training batches on cuda:  87%|████████▋ | 8444/9708 [00:40<00:06, 198.46batch/s]

Training batches on cuda:  87%|████████▋ | 8464/9708 [00:40<00:06, 192.71batch/s]

Training batches on cuda:  87%|████████▋ | 8486/9708 [00:40<00:06, 199.58batch/s]

Training batches on cuda:  88%|████████▊ | 8507/9708 [00:40<00:06, 198.77batch/s]

Training batches on cuda:  88%|████████▊ | 8528/9708 [00:40<00:05, 200.41batch/s]

Training batches on cuda:  88%|████████▊ | 8549/9708 [00:40<00:05, 203.00batch/s]

Training batches on cuda:  88%|████████▊ | 8570/9708 [00:40<00:05, 203.27batch/s]

Training batches on cuda:  89%|████████▊ | 8592/9708 [00:40<00:05, 205.80batch/s]

Training batches on cuda:  89%|████████▊ | 8613/9708 [00:40<00:05, 206.99batch/s]

Training batches on cuda:  89%|████████▉ | 8634/9708 [00:40<00:05, 207.09batch/s]

Training batches on cuda:  89%|████████▉ | 8655/9708 [00:41<00:05, 203.69batch/s]

Training batches on cuda:  89%|████████▉ | 8677/9708 [00:41<00:04, 206.24batch/s]

Training batches on cuda:  90%|████████▉ | 8699/9708 [00:41<00:04, 207.99batch/s]

Training batches on cuda:  90%|████████▉ | 8720/9708 [00:41<00:04, 208.53batch/s]

Training batches on cuda:  90%|█████████ | 8741/9708 [00:41<00:04, 196.05batch/s]

Training batches on cuda:  90%|█████████ | 8761/9708 [00:41<00:04, 194.67batch/s]

Training batches on cuda:  90%|█████████ | 8782/9708 [00:41<00:04, 197.16batch/s]

Training batches on cuda:  91%|█████████ | 8803/9708 [00:41<00:04, 200.19batch/s]

Training batches on cuda:  91%|█████████ | 8825/9708 [00:41<00:04, 203.27batch/s]

Training batches on cuda:  91%|█████████ | 8846/9708 [00:41<00:04, 204.67batch/s]

Training batches on cuda:  91%|█████████▏| 8867/9708 [00:42<00:04, 191.20batch/s]

Training batches on cuda:  92%|█████████▏| 8889/9708 [00:42<00:04, 197.44batch/s]

Training batches on cuda:  92%|█████████▏| 8909/9708 [00:42<00:04, 192.37batch/s]

Training batches on cuda:  92%|█████████▏| 8929/9708 [00:42<00:04, 191.64batch/s]

Training batches on cuda:  92%|█████████▏| 8949/9708 [00:42<00:03, 192.75batch/s]

Training batches on cuda:  92%|█████████▏| 8970/9708 [00:42<00:03, 197.29batch/s]

Training batches on cuda:  93%|█████████▎| 8990/9708 [00:42<00:03, 194.22batch/s]

Training batches on cuda:  93%|█████████▎| 9010/9708 [00:42<00:03, 188.88batch/s]

Training batches on cuda:  93%|█████████▎| 9030/9708 [00:42<00:03, 190.29batch/s]

Training batches on cuda:  93%|█████████▎| 9052/9708 [00:43<00:03, 197.77batch/s]

Training batches on cuda:  93%|█████████▎| 9074/9708 [00:43<00:03, 202.89batch/s]

Training batches on cuda:  94%|█████████▎| 9096/9708 [00:43<00:02, 205.65batch/s]

Training batches on cuda:  94%|█████████▍| 9119/9708 [00:43<00:02, 211.32batch/s]

Training batches on cuda:  94%|█████████▍| 9143/9708 [00:43<00:02, 217.45batch/s]

Training batches on cuda:  94%|█████████▍| 9165/9708 [00:43<00:02, 205.19batch/s]

Training batches on cuda:  95%|█████████▍| 9186/9708 [00:43<00:02, 205.27batch/s]

Training batches on cuda:  95%|█████████▍| 9207/9708 [00:43<00:02, 202.42batch/s]

Training batches on cuda:  95%|█████████▌| 9229/9708 [00:43<00:02, 207.33batch/s]

Training batches on cuda:  95%|█████████▌| 9251/9708 [00:44<00:02, 209.80batch/s]

Training batches on cuda:  96%|█████████▌| 9274/9708 [00:44<00:02, 214.39batch/s]

Training batches on cuda:  96%|█████████▌| 9296/9708 [00:44<00:02, 205.39batch/s]

Training batches on cuda:  96%|█████████▌| 9317/9708 [00:44<00:01, 205.71batch/s]

Training batches on cuda:  96%|█████████▌| 9340/9708 [00:44<00:01, 210.35batch/s]

Training batches on cuda:  96%|█████████▋| 9362/9708 [00:44<00:01, 209.26batch/s]

Training batches on cuda:  97%|█████████▋| 9383/9708 [00:44<00:01, 209.32batch/s]

Training batches on cuda:  97%|█████████▋| 9404/9708 [00:44<00:01, 208.06batch/s]

Training batches on cuda:  97%|█████████▋| 9426/9708 [00:44<00:01, 209.04batch/s]

Training batches on cuda:  97%|█████████▋| 9447/9708 [00:44<00:01, 207.86batch/s]

Training batches on cuda:  98%|█████████▊| 9468/9708 [00:45<00:01, 206.69batch/s]

Training batches on cuda:  98%|█████████▊| 9489/9708 [00:45<00:01, 207.38batch/s]

Training batches on cuda:  98%|█████████▊| 9511/9708 [00:45<00:00, 209.13batch/s]

Training batches on cuda:  98%|█████████▊| 9533/9708 [00:45<00:00, 210.61batch/s]

Training batches on cuda:  98%|█████████▊| 9555/9708 [00:45<00:00, 210.81batch/s]

Training batches on cuda:  99%|█████████▊| 9577/9708 [00:45<00:00, 210.19batch/s]

Training batches on cuda:  99%|█████████▉| 9599/9708 [00:45<00:00, 211.70batch/s]

Training batches on cuda:  99%|█████████▉| 9621/9708 [00:45<00:00, 207.31batch/s]

Training batches on cuda:  99%|█████████▉| 9642/9708 [00:45<00:00, 198.65batch/s]

Training batches on cuda: 100%|█████████▉| 9664/9708 [00:46<00:00, 202.46batch/s]

Training batches on cuda: 100%|█████████▉| 9686/9708 [00:46<00:00, 205.04batch/s]

Training batches on cuda: 100%|██████████| 9708/9708 [00:46<00:00, 203.73batch/s]

                                                                                 
Training epochs on cuda:  30%|███       | 3/10 [03:07<05:28, 46.99s/epoch, loss=0.0417, prev_loss=0.0417]
Training epochs on cuda:  40%|████      | 4/10 [03:07<04:41, 46.84s/epoch, loss=0.0417, prev_loss=0.0417]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 17/9708 [00:00<00:58, 164.80batch/s]

Training batches on cuda:   0%|          | 39/9708 [00:00<00:54, 178.20batch/s]

Training batches on cuda:   1%|          | 60/9708 [00:00<00:52, 185.44batch/s]

Training batches on cuda:   1%|          | 81/9708 [00:00<00:50, 191.42batch/s]

Training batches on cuda:   1%|          | 102/9708 [00:00<00:49, 195.90batch/s]

Training batches on cuda:   1%|▏         | 124/9708 [00:00<00:47, 200.68batch/s]

Training batches on cuda:   1%|▏         | 145/9708 [00:00<00:47, 202.37batch/s]

Training batches on cuda:   2%|▏         | 166/9708 [00:00<00:46, 203.21batch/s]

Training batches on cuda:   2%|▏         | 187/9708 [00:00<00:46, 204.65batch/s]

Training batches on cuda:   2%|▏         | 208/9708 [00:01<00:46, 205.12batch/s]

Training batches on cuda:   2%|▏         | 230/9708 [00:01<00:45, 207.34batch/s]

Training batches on cuda:   3%|▎         | 252/9708 [00:01<00:45, 209.73batch/s]

Training batches on cuda:   3%|▎         | 274/9708 [00:01<00:44, 209.99batch/s]

Training batches on cuda:   3%|▎         | 296/9708 [00:01<00:44, 210.64batch/s]

Training batches on cuda:   3%|▎         | 319/9708 [00:01<00:43, 214.12batch/s]

Training batches on cuda:   4%|▎         | 342/9708 [00:01<00:43, 217.10batch/s]

Training batches on cuda:   4%|▍         | 366/9708 [00:01<00:42, 221.97batch/s]

Training batches on cuda:   4%|▍         | 392/9708 [00:01<00:40, 229.97batch/s]

Training batches on cuda:   4%|▍         | 417/9708 [00:01<00:39, 235.57batch/s]

Training batches on cuda:   5%|▍         | 442/9708 [00:02<00:38, 239.53batch/s]

Training batches on cuda:   5%|▍         | 468/9708 [00:02<00:38, 242.69batch/s]

Training batches on cuda:   5%|▌         | 494/9708 [00:02<00:37, 245.15batch/s]

Training batches on cuda:   5%|▌         | 519/9708 [00:02<00:37, 244.67batch/s]

Training batches on cuda:   6%|▌         | 544/9708 [00:02<00:38, 238.37batch/s]

Training batches on cuda:   6%|▌         | 568/9708 [00:02<00:39, 231.26batch/s]

Training batches on cuda:   6%|▌         | 592/9708 [00:02<00:39, 230.78batch/s]

Training batches on cuda:   6%|▋         | 616/9708 [00:02<00:40, 224.31batch/s]

Training batches on cuda:   7%|▋         | 639/9708 [00:02<00:41, 218.63batch/s]

Training batches on cuda:   7%|▋         | 661/9708 [00:03<00:41, 215.73batch/s]

Training batches on cuda:   7%|▋         | 683/9708 [00:03<00:43, 209.00batch/s]

Training batches on cuda:   7%|▋         | 705/9708 [00:03<00:42, 209.99batch/s]

Training batches on cuda:   7%|▋         | 727/9708 [00:03<00:43, 206.38batch/s]

Training batches on cuda:   8%|▊         | 749/9708 [00:03<00:43, 207.84batch/s]

Training batches on cuda:   8%|▊         | 770/9708 [00:03<00:43, 205.07batch/s]

Training batches on cuda:   8%|▊         | 791/9708 [00:03<00:47, 187.96batch/s]

Training batches on cuda:   8%|▊         | 811/9708 [00:03<00:48, 184.21batch/s]

Training batches on cuda:   9%|▊         | 830/9708 [00:03<00:48, 182.87batch/s]

Training batches on cuda:   9%|▉         | 850/9708 [00:04<00:47, 186.06batch/s]

Training batches on cuda:   9%|▉         | 871/9708 [00:04<00:46, 191.81batch/s]

Training batches on cuda:   9%|▉         | 891/9708 [00:04<00:46, 191.41batch/s]

Training batches on cuda:   9%|▉         | 912/9708 [00:04<00:44, 196.36batch/s]

Training batches on cuda:  10%|▉         | 932/9708 [00:04<00:45, 192.91batch/s]

Training batches on cuda:  10%|▉         | 953/9708 [00:04<00:44, 195.94batch/s]

Training batches on cuda:  10%|█         | 975/9708 [00:04<00:43, 201.86batch/s]

Training batches on cuda:  10%|█         | 998/9708 [00:04<00:41, 207.95batch/s]

Training batches on cuda:  10%|█         | 1019/9708 [00:04<00:42, 205.00batch/s]

Training batches on cuda:  11%|█         | 1040/9708 [00:04<00:42, 203.56batch/s]

Training batches on cuda:  11%|█         | 1061/9708 [00:05<00:46, 184.79batch/s]

Training batches on cuda:  11%|█         | 1082/9708 [00:05<00:45, 191.66batch/s]

Training batches on cuda:  11%|█▏        | 1103/9708 [00:05<00:44, 195.31batch/s]

Training batches on cuda:  12%|█▏        | 1124/9708 [00:05<00:43, 196.97batch/s]

Training batches on cuda:  12%|█▏        | 1145/9708 [00:05<00:43, 197.60batch/s]

Training batches on cuda:  12%|█▏        | 1165/9708 [00:05<00:44, 192.19batch/s]

Training batches on cuda:  12%|█▏        | 1185/9708 [00:05<00:44, 193.63batch/s]

Training batches on cuda:  12%|█▏        | 1206/9708 [00:05<00:43, 195.98batch/s]

Training batches on cuda:  13%|█▎        | 1226/9708 [00:05<00:43, 195.16batch/s]

Training batches on cuda:  13%|█▎        | 1247/9708 [00:06<00:42, 198.22batch/s]

Training batches on cuda:  13%|█▎        | 1269/9708 [00:06<00:41, 204.22batch/s]

Training batches on cuda:  13%|█▎        | 1291/9708 [00:06<00:40, 206.19batch/s]

Training batches on cuda:  14%|█▎        | 1312/9708 [00:06<00:41, 203.40batch/s]

Training batches on cuda:  14%|█▎        | 1333/9708 [00:06<00:41, 201.20batch/s]

Training batches on cuda:  14%|█▍        | 1354/9708 [00:06<00:42, 197.16batch/s]

Training batches on cuda:  14%|█▍        | 1374/9708 [00:06<00:45, 183.00batch/s]

Training batches on cuda:  14%|█▍        | 1396/9708 [00:06<00:43, 192.48batch/s]

Training batches on cuda:  15%|█▍        | 1416/9708 [00:06<00:42, 194.43batch/s]

Training batches on cuda:  15%|█▍        | 1439/9708 [00:06<00:40, 202.50batch/s]

Training batches on cuda:  15%|█▌        | 1462/9708 [00:07<00:39, 208.87batch/s]

Training batches on cuda:  15%|█▌        | 1485/9708 [00:07<00:38, 213.25batch/s]

Training batches on cuda:  16%|█▌        | 1507/9708 [00:07<00:38, 213.66batch/s]

Training batches on cuda:  16%|█▌        | 1529/9708 [00:07<00:38, 211.12batch/s]

Training batches on cuda:  16%|█▌        | 1551/9708 [00:07<00:39, 204.92batch/s]

Training batches on cuda:  16%|█▌        | 1572/9708 [00:07<00:40, 199.04batch/s]

Training batches on cuda:  16%|█▋        | 1596/9708 [00:07<00:39, 207.84batch/s]

Training batches on cuda:  17%|█▋        | 1621/9708 [00:07<00:36, 218.71batch/s]

Training batches on cuda:  17%|█▋        | 1646/9708 [00:07<00:35, 226.43batch/s]

Training batches on cuda:  17%|█▋        | 1669/9708 [00:08<00:36, 219.73batch/s]

Training batches on cuda:  17%|█▋        | 1692/9708 [00:08<00:37, 212.48batch/s]

Training batches on cuda:  18%|█▊        | 1714/9708 [00:08<00:37, 210.74batch/s]

Training batches on cuda:  18%|█▊        | 1736/9708 [00:08<00:38, 207.22batch/s]

Training batches on cuda:  18%|█▊        | 1757/9708 [00:08<00:38, 205.34batch/s]

Training batches on cuda:  18%|█▊        | 1778/9708 [00:08<00:39, 203.12batch/s]

Training batches on cuda:  19%|█▊        | 1799/9708 [00:08<00:39, 200.39batch/s]

Training batches on cuda:  19%|█▊        | 1820/9708 [00:08<00:39, 199.64batch/s]

Training batches on cuda:  19%|█▉        | 1841/9708 [00:08<00:39, 200.19batch/s]

Training batches on cuda:  19%|█▉        | 1862/9708 [00:09<00:40, 196.04batch/s]

Training batches on cuda:  19%|█▉        | 1882/9708 [00:09<00:41, 189.17batch/s]

Training batches on cuda:  20%|█▉        | 1901/9708 [00:09<00:42, 183.38batch/s]

Training batches on cuda:  20%|█▉        | 1920/9708 [00:09<00:42, 184.35batch/s]

Training batches on cuda:  20%|█▉        | 1940/9708 [00:09<00:41, 187.53batch/s]

Training batches on cuda:  20%|██        | 1961/9708 [00:09<00:40, 191.49batch/s]

Training batches on cuda:  20%|██        | 1981/9708 [00:09<00:40, 189.40batch/s]

Training batches on cuda:  21%|██        | 2000/9708 [00:09<00:40, 188.86batch/s]

Training batches on cuda:  21%|██        | 2020/9708 [00:09<00:40, 189.80batch/s]

Training batches on cuda:  21%|██        | 2040/9708 [00:09<00:41, 184.91batch/s]

Training batches on cuda:  21%|██        | 2062/9708 [00:10<00:39, 193.44batch/s]

Training batches on cuda:  21%|██▏       | 2084/9708 [00:10<00:38, 199.67batch/s]

Training batches on cuda:  22%|██▏       | 2107/9708 [00:10<00:36, 206.39batch/s]

Training batches on cuda:  22%|██▏       | 2128/9708 [00:10<00:36, 207.15batch/s]

Training batches on cuda:  22%|██▏       | 2149/9708 [00:10<00:37, 204.08batch/s]

Training batches on cuda:  22%|██▏       | 2170/9708 [00:10<00:36, 204.19batch/s]

Training batches on cuda:  23%|██▎       | 2191/9708 [00:10<00:37, 198.38batch/s]

Training batches on cuda:  23%|██▎       | 2212/9708 [00:10<00:37, 199.29batch/s]

Training batches on cuda:  23%|██▎       | 2232/9708 [00:10<00:37, 198.92batch/s]

Training batches on cuda:  23%|██▎       | 2256/9708 [00:11<00:35, 207.54batch/s]

Training batches on cuda:  23%|██▎       | 2280/9708 [00:11<00:34, 213.54batch/s]

Training batches on cuda:  24%|██▎       | 2303/9708 [00:11<00:34, 217.55batch/s]

Training batches on cuda:  24%|██▍       | 2325/9708 [00:11<00:34, 215.93batch/s]

Training batches on cuda:  24%|██▍       | 2348/9708 [00:11<00:33, 218.57batch/s]

Training batches on cuda:  24%|██▍       | 2370/9708 [00:11<00:34, 215.66batch/s]

Training batches on cuda:  25%|██▍       | 2392/9708 [00:11<00:34, 213.24batch/s]

Training batches on cuda:  25%|██▍       | 2414/9708 [00:11<00:34, 213.50batch/s]

Training batches on cuda:  25%|██▌       | 2436/9708 [00:11<00:33, 214.17batch/s]

Training batches on cuda:  25%|██▌       | 2458/9708 [00:11<00:33, 214.96batch/s]

Training batches on cuda:  26%|██▌       | 2481/9708 [00:12<00:33, 216.60batch/s]

Training batches on cuda:  26%|██▌       | 2505/9708 [00:12<00:32, 220.50batch/s]

Training batches on cuda:  26%|██▌       | 2528/9708 [00:12<00:35, 200.45batch/s]

Training batches on cuda:  26%|██▋       | 2549/9708 [00:12<00:35, 202.61batch/s]

Training batches on cuda:  26%|██▋       | 2572/9708 [00:12<00:34, 207.98batch/s]

Training batches on cuda:  27%|██▋       | 2594/9708 [00:12<00:33, 209.72batch/s]

Training batches on cuda:  27%|██▋       | 2616/9708 [00:12<00:34, 206.99batch/s]

Training batches on cuda:  27%|██▋       | 2637/9708 [00:12<00:34, 206.70batch/s]

Training batches on cuda:  27%|██▋       | 2658/9708 [00:12<00:34, 202.03batch/s]

Training batches on cuda:  28%|██▊       | 2679/9708 [00:13<00:36, 195.04batch/s]

Training batches on cuda:  28%|██▊       | 2699/9708 [00:13<00:35, 194.95batch/s]

Training batches on cuda:  28%|██▊       | 2719/9708 [00:13<00:39, 178.78batch/s]

Training batches on cuda:  28%|██▊       | 2740/9708 [00:13<00:37, 185.46batch/s]

Training batches on cuda:  28%|██▊       | 2761/9708 [00:13<00:36, 191.24batch/s]

Training batches on cuda:  29%|██▊       | 2782/9708 [00:13<00:35, 195.85batch/s]

Training batches on cuda:  29%|██▉       | 2803/9708 [00:13<00:34, 199.00batch/s]

Training batches on cuda:  29%|██▉       | 2824/9708 [00:13<00:34, 201.86batch/s]

Training batches on cuda:  29%|██▉       | 2846/9708 [00:13<00:33, 204.87batch/s]

Training batches on cuda:  30%|██▉       | 2869/9708 [00:13<00:32, 210.45batch/s]

Training batches on cuda:  30%|██▉       | 2892/9708 [00:14<00:31, 213.41batch/s]

Training batches on cuda:  30%|███       | 2915/9708 [00:14<00:31, 216.35batch/s]

Training batches on cuda:  30%|███       | 2937/9708 [00:14<00:31, 217.19batch/s]

Training batches on cuda:  30%|███       | 2960/9708 [00:14<00:30, 220.25batch/s]

Training batches on cuda:  31%|███       | 2983/9708 [00:14<00:30, 221.47batch/s]

Training batches on cuda:  31%|███       | 3006/9708 [00:14<00:30, 218.25batch/s]

Training batches on cuda:  31%|███       | 3028/9708 [00:14<00:30, 217.51batch/s]

Training batches on cuda:  31%|███▏      | 3052/9708 [00:14<00:30, 221.74batch/s]

Training batches on cuda:  32%|███▏      | 3075/9708 [00:14<00:29, 222.98batch/s]

Training batches on cuda:  32%|███▏      | 3100/9708 [00:15<00:28, 228.02batch/s]

Training batches on cuda:  32%|███▏      | 3124/9708 [00:15<00:28, 229.06batch/s]

Training batches on cuda:  32%|███▏      | 3147/9708 [00:15<00:28, 229.16batch/s]

Training batches on cuda:  33%|███▎      | 3170/9708 [00:15<00:28, 228.94batch/s]

Training batches on cuda:  33%|███▎      | 3193/9708 [00:15<00:29, 223.08batch/s]

Training batches on cuda:  33%|███▎      | 3216/9708 [00:15<00:29, 218.62batch/s]

Training batches on cuda:  33%|███▎      | 3238/9708 [00:15<00:31, 207.75batch/s]

Training batches on cuda:  34%|███▎      | 3259/9708 [00:15<00:31, 203.02batch/s]

Training batches on cuda:  34%|███▍      | 3280/9708 [00:15<00:32, 197.13batch/s]

Training batches on cuda:  34%|███▍      | 3300/9708 [00:15<00:33, 190.33batch/s]

Training batches on cuda:  34%|███▍      | 3320/9708 [00:16<00:33, 191.02batch/s]

Training batches on cuda:  34%|███▍      | 3340/9708 [00:16<00:33, 187.44batch/s]

Training batches on cuda:  35%|███▍      | 3360/9708 [00:16<00:33, 190.94batch/s]

Training batches on cuda:  35%|███▍      | 3382/9708 [00:16<00:32, 197.44batch/s]

Training batches on cuda:  35%|███▌      | 3402/9708 [00:16<00:32, 194.99batch/s]

Training batches on cuda:  35%|███▌      | 3422/9708 [00:16<00:33, 190.23batch/s]

Training batches on cuda:  35%|███▌      | 3442/9708 [00:16<00:33, 188.64batch/s]

Training batches on cuda:  36%|███▌      | 3463/9708 [00:16<00:32, 192.38batch/s]

Training batches on cuda:  36%|███▌      | 3483/9708 [00:16<00:31, 194.57batch/s]

Training batches on cuda:  36%|███▌      | 3505/9708 [00:17<00:30, 200.14batch/s]

Training batches on cuda:  36%|███▋      | 3528/9708 [00:17<00:29, 206.25batch/s]

Training batches on cuda:  37%|███▋      | 3553/9708 [00:17<00:28, 215.57batch/s]

Training batches on cuda:  37%|███▋      | 3577/9708 [00:17<00:27, 221.70batch/s]

Training batches on cuda:  37%|███▋      | 3600/9708 [00:17<00:29, 206.83batch/s]

Training batches on cuda:  37%|███▋      | 3622/9708 [00:17<00:31, 195.76batch/s]

Training batches on cuda:  38%|███▊      | 3642/9708 [00:17<00:30, 195.95batch/s]

Training batches on cuda:  38%|███▊      | 3662/9708 [00:17<00:31, 194.86batch/s]

Training batches on cuda:  38%|███▊      | 3682/9708 [00:17<00:30, 194.92batch/s]

Training batches on cuda:  38%|███▊      | 3702/9708 [00:18<00:30, 196.16batch/s]

Training batches on cuda:  38%|███▊      | 3724/9708 [00:18<00:29, 200.33batch/s]

Training batches on cuda:  39%|███▊      | 3745/9708 [00:18<00:30, 193.92batch/s]

Training batches on cuda:  39%|███▉      | 3766/9708 [00:18<00:29, 198.15batch/s]

Training batches on cuda:  39%|███▉      | 3787/9708 [00:18<00:29, 199.44batch/s]

Training batches on cuda:  39%|███▉      | 3811/9708 [00:18<00:28, 208.54batch/s]

Training batches on cuda:  39%|███▉      | 3833/9708 [00:18<00:27, 211.45batch/s]

Training batches on cuda:  40%|███▉      | 3856/9708 [00:18<00:27, 216.22batch/s]

Training batches on cuda:  40%|███▉      | 3880/9708 [00:18<00:26, 220.42batch/s]

Training batches on cuda:  40%|████      | 3903/9708 [00:18<00:26, 219.75batch/s]

Training batches on cuda:  40%|████      | 3927/9708 [00:19<00:25, 224.87batch/s]

Training batches on cuda:  41%|████      | 3953/9708 [00:19<00:24, 233.91batch/s]

Training batches on cuda:  41%|████      | 3977/9708 [00:19<00:24, 233.83batch/s]

Training batches on cuda:  41%|████      | 4001/9708 [00:19<00:24, 231.78batch/s]

Training batches on cuda:  41%|████▏     | 4025/9708 [00:19<00:25, 224.33batch/s]

Training batches on cuda:  42%|████▏     | 4048/9708 [00:19<00:25, 222.76batch/s]

Training batches on cuda:  42%|████▏     | 4071/9708 [00:19<00:26, 216.74batch/s]

Training batches on cuda:  42%|████▏     | 4094/9708 [00:19<00:25, 220.31batch/s]

Training batches on cuda:  42%|████▏     | 4117/9708 [00:19<00:26, 212.42batch/s]

Training batches on cuda:  43%|████▎     | 4139/9708 [00:20<00:28, 194.22batch/s]

Training batches on cuda:  43%|████▎     | 4166/9708 [00:20<00:26, 211.93batch/s]

Training batches on cuda:  43%|████▎     | 4194/9708 [00:20<00:24, 226.67batch/s]

Training batches on cuda:  43%|████▎     | 4222/9708 [00:20<00:23, 238.29batch/s]

Training batches on cuda:  44%|████▍     | 4249/9708 [00:20<00:22, 245.11batch/s]

Training batches on cuda:  44%|████▍     | 4277/9708 [00:20<00:21, 252.41batch/s]

Training batches on cuda:  44%|████▍     | 4305/9708 [00:20<00:20, 257.92batch/s]

Training batches on cuda:  45%|████▍     | 4333/9708 [00:20<00:20, 261.51batch/s]

Training batches on cuda:  45%|████▍     | 4360/9708 [00:20<00:20, 263.21batch/s]

Training batches on cuda:  45%|████▌     | 4387/9708 [00:20<00:20, 265.04batch/s]

Training batches on cuda:  45%|████▌     | 4414/9708 [00:21<00:19, 265.80batch/s]

Training batches on cuda:  46%|████▌     | 4441/9708 [00:21<00:19, 266.61batch/s]

Training batches on cuda:  46%|████▌     | 4469/9708 [00:21<00:19, 268.04batch/s]

Training batches on cuda:  46%|████▋     | 4497/9708 [00:21<00:19, 268.66batch/s]

Training batches on cuda:  47%|████▋     | 4524/9708 [00:21<00:19, 268.56batch/s]

Training batches on cuda:  47%|████▋     | 4552/9708 [00:21<00:19, 269.06batch/s]

Training batches on cuda:  47%|████▋     | 4580/9708 [00:21<00:19, 269.48batch/s]

Training batches on cuda:  47%|████▋     | 4607/9708 [00:21<00:18, 268.89batch/s]

Training batches on cuda:  48%|████▊     | 4634/9708 [00:21<00:18, 269.19batch/s]

Training batches on cuda:  48%|████▊     | 4662/9708 [00:21<00:18, 270.04batch/s]

Training batches on cuda:  48%|████▊     | 4690/9708 [00:22<00:19, 260.80batch/s]

Training batches on cuda:  49%|████▊     | 4717/9708 [00:22<00:20, 244.15batch/s]

Training batches on cuda:  49%|████▉     | 4742/9708 [00:22<00:20, 242.66batch/s]

Training batches on cuda:  49%|████▉     | 4767/9708 [00:22<00:20, 237.17batch/s]

Training batches on cuda:  49%|████▉     | 4791/9708 [00:22<00:20, 236.59batch/s]

Training batches on cuda:  50%|████▉     | 4815/9708 [00:22<00:21, 222.88batch/s]

Training batches on cuda:  50%|████▉     | 4838/9708 [00:22<00:22, 217.96batch/s]

Training batches on cuda:  50%|█████     | 4860/9708 [00:22<00:23, 206.28batch/s]

Training batches on cuda:  50%|█████     | 4881/9708 [00:22<00:23, 206.26batch/s]

Training batches on cuda:  50%|█████     | 4902/9708 [00:23<00:23, 204.39batch/s]

Training batches on cuda:  51%|█████     | 4924/9708 [00:23<00:22, 208.77batch/s]

Training batches on cuda:  51%|█████     | 4945/9708 [00:23<00:25, 184.74batch/s]

Training batches on cuda:  51%|█████     | 4966/9708 [00:23<00:24, 191.24batch/s]

Training batches on cuda:  51%|█████▏    | 4988/9708 [00:23<00:23, 198.24batch/s]

Training batches on cuda:  52%|█████▏    | 5010/9708 [00:23<00:23, 201.83batch/s]

Training batches on cuda:  52%|█████▏    | 5033/9708 [00:23<00:22, 206.99batch/s]

Training batches on cuda:  52%|█████▏    | 5054/9708 [00:23<00:22, 206.32batch/s]

Training batches on cuda:  52%|█████▏    | 5076/9708 [00:23<00:22, 207.97batch/s]

Training batches on cuda:  53%|█████▎    | 5098/9708 [00:24<00:21, 209.72batch/s]

Training batches on cuda:  53%|█████▎    | 5120/9708 [00:24<00:21, 209.57batch/s]

Training batches on cuda:  53%|█████▎    | 5142/9708 [00:24<00:23, 196.11batch/s]

Training batches on cuda:  53%|█████▎    | 5163/9708 [00:24<00:22, 199.62batch/s]

Training batches on cuda:  53%|█████▎    | 5185/9708 [00:24<00:22, 204.06batch/s]

Training batches on cuda:  54%|█████▎    | 5206/9708 [00:24<00:22, 199.22batch/s]

Training batches on cuda:  54%|█████▍    | 5227/9708 [00:24<00:22, 200.88batch/s]

Training batches on cuda:  54%|█████▍    | 5248/9708 [00:24<00:21, 203.31batch/s]

Training batches on cuda:  54%|█████▍    | 5269/9708 [00:24<00:21, 202.61batch/s]

Training batches on cuda:  54%|█████▍    | 5290/9708 [00:25<00:22, 200.43batch/s]

Training batches on cuda:  55%|█████▍    | 5311/9708 [00:25<00:21, 201.27batch/s]

Training batches on cuda:  55%|█████▍    | 5333/9708 [00:25<00:21, 205.52batch/s]

Training batches on cuda:  55%|█████▌    | 5354/9708 [00:25<00:21, 201.12batch/s]

Training batches on cuda:  55%|█████▌    | 5375/9708 [00:25<00:22, 193.34batch/s]

Training batches on cuda:  56%|█████▌    | 5395/9708 [00:25<00:22, 190.53batch/s]

Training batches on cuda:  56%|█████▌    | 5416/9708 [00:25<00:21, 195.39batch/s]

Training batches on cuda:  56%|█████▌    | 5437/9708 [00:25<00:21, 198.71batch/s]

Training batches on cuda:  56%|█████▌    | 5459/9708 [00:25<00:20, 203.98batch/s]

Training batches on cuda:  56%|█████▋    | 5481/9708 [00:25<00:20, 206.04batch/s]

Training batches on cuda:  57%|█████▋    | 5503/9708 [00:26<00:20, 209.33batch/s]

Training batches on cuda:  57%|█████▋    | 5526/9708 [00:26<00:19, 214.21batch/s]

Training batches on cuda:  57%|█████▋    | 5549/9708 [00:26<00:19, 216.72batch/s]

Training batches on cuda:  57%|█████▋    | 5571/9708 [00:26<00:19, 215.20batch/s]

Training batches on cuda:  58%|█████▊    | 5593/9708 [00:26<00:19, 216.04batch/s]

Training batches on cuda:  58%|█████▊    | 5615/9708 [00:26<00:18, 216.15batch/s]

Training batches on cuda:  58%|█████▊    | 5637/9708 [00:26<00:18, 216.15batch/s]

Training batches on cuda:  58%|█████▊    | 5659/9708 [00:26<00:19, 208.38batch/s]

Training batches on cuda:  59%|█████▊    | 5682/9708 [00:26<00:18, 212.21batch/s]

Training batches on cuda:  59%|█████▉    | 5704/9708 [00:27<00:19, 205.14batch/s]

Training batches on cuda:  59%|█████▉    | 5725/9708 [00:27<00:19, 204.92batch/s]

Training batches on cuda:  59%|█████▉    | 5746/9708 [00:27<00:19, 203.93batch/s]

Training batches on cuda:  59%|█████▉    | 5767/9708 [00:27<00:19, 205.58batch/s]

Training batches on cuda:  60%|█████▉    | 5789/9708 [00:27<00:18, 209.14batch/s]

Training batches on cuda:  60%|█████▉    | 5810/9708 [00:27<00:19, 200.59batch/s]

Training batches on cuda:  60%|██████    | 5831/9708 [00:27<00:19, 197.55batch/s]

Training batches on cuda:  60%|██████    | 5851/9708 [00:27<00:19, 193.09batch/s]

Training batches on cuda:  60%|██████    | 5871/9708 [00:27<00:20, 187.67batch/s]

Training batches on cuda:  61%|██████    | 5890/9708 [00:27<00:20, 186.75batch/s]

Training batches on cuda:  61%|██████    | 5911/9708 [00:28<00:19, 190.83batch/s]

Training batches on cuda:  61%|██████    | 5931/9708 [00:28<00:19, 192.09batch/s]

Training batches on cuda:  61%|██████▏   | 5953/9708 [00:28<00:18, 197.87batch/s]

Training batches on cuda:  62%|██████▏   | 5974/9708 [00:28<00:18, 199.78batch/s]

Training batches on cuda:  62%|██████▏   | 5995/9708 [00:28<00:18, 199.08batch/s]

Training batches on cuda:  62%|██████▏   | 6015/9708 [00:28<00:18, 194.39batch/s]

Training batches on cuda:  62%|██████▏   | 6036/9708 [00:28<00:18, 198.04batch/s]

Training batches on cuda:  62%|██████▏   | 6056/9708 [00:28<00:18, 195.10batch/s]

Training batches on cuda:  63%|██████▎   | 6079/9708 [00:28<00:17, 202.65batch/s]

Training batches on cuda:  63%|██████▎   | 6101/9708 [00:29<00:17, 206.45batch/s]

Training batches on cuda:  63%|██████▎   | 6126/9708 [00:29<00:16, 214.62batch/s]

Training batches on cuda:  63%|██████▎   | 6150/9708 [00:29<00:16, 220.37batch/s]

Training batches on cuda:  64%|██████▎   | 6175/9708 [00:29<00:15, 227.51batch/s]

Training batches on cuda:  64%|██████▍   | 6200/9708 [00:29<00:15, 232.36batch/s]

Training batches on cuda:  64%|██████▍   | 6224/9708 [00:29<00:15, 224.04batch/s]

Training batches on cuda:  64%|██████▍   | 6247/9708 [00:29<00:17, 203.00batch/s]

Training batches on cuda:  65%|██████▍   | 6268/9708 [00:29<00:17, 196.36batch/s]

Training batches on cuda:  65%|██████▍   | 6289/9708 [00:29<00:17, 199.89batch/s]

Training batches on cuda:  65%|██████▌   | 6311/9708 [00:30<00:16, 204.89batch/s]

Training batches on cuda:  65%|██████▌   | 6332/9708 [00:30<00:17, 190.96batch/s]

Training batches on cuda:  65%|██████▌   | 6355/9708 [00:30<00:16, 199.07batch/s]

Training batches on cuda:  66%|██████▌   | 6377/9708 [00:30<00:16, 202.86batch/s]

Training batches on cuda:  66%|██████▌   | 6400/9708 [00:30<00:15, 209.26batch/s]

Training batches on cuda:  66%|██████▌   | 6422/9708 [00:30<00:15, 206.65batch/s]

Training batches on cuda:  66%|██████▋   | 6443/9708 [00:30<00:16, 192.27batch/s]

Training batches on cuda:  67%|██████▋   | 6463/9708 [00:30<00:17, 188.53batch/s]

Training batches on cuda:  67%|██████▋   | 6483/9708 [00:30<00:17, 185.53batch/s]

Training batches on cuda:  67%|██████▋   | 6503/9708 [00:31<00:17, 187.97batch/s]

Training batches on cuda:  67%|██████▋   | 6525/9708 [00:31<00:16, 194.45batch/s]

Training batches on cuda:  67%|██████▋   | 6545/9708 [00:31<00:16, 192.99batch/s]

Training batches on cuda:  68%|██████▊   | 6565/9708 [00:31<00:16, 193.91batch/s]

Training batches on cuda:  68%|██████▊   | 6585/9708 [00:31<00:16, 194.67batch/s]

Training batches on cuda:  68%|██████▊   | 6607/9708 [00:31<00:15, 198.88batch/s]

Training batches on cuda:  68%|██████▊   | 6629/9708 [00:31<00:15, 204.38batch/s]

Training batches on cuda:  69%|██████▊   | 6652/9708 [00:31<00:14, 210.64batch/s]

Training batches on cuda:  69%|██████▉   | 6675/9708 [00:31<00:14, 213.98batch/s]

Training batches on cuda:  69%|██████▉   | 6697/9708 [00:31<00:14, 213.64batch/s]

Training batches on cuda:  69%|██████▉   | 6719/9708 [00:32<00:13, 213.96batch/s]

Training batches on cuda:  69%|██████▉   | 6741/9708 [00:32<00:14, 203.71batch/s]

Training batches on cuda:  70%|██████▉   | 6762/9708 [00:32<00:14, 200.31batch/s]

Training batches on cuda:  70%|██████▉   | 6783/9708 [00:32<00:14, 200.82batch/s]

Training batches on cuda:  70%|███████   | 6804/9708 [00:32<00:14, 201.91batch/s]

Training batches on cuda:  70%|███████   | 6825/9708 [00:32<00:14, 203.74batch/s]

Training batches on cuda:  71%|███████   | 6846/9708 [00:32<00:13, 204.82batch/s]

Training batches on cuda:  71%|███████   | 6868/9708 [00:32<00:13, 207.87batch/s]

Training batches on cuda:  71%|███████   | 6889/9708 [00:32<00:13, 206.66batch/s]

Training batches on cuda:  71%|███████   | 6910/9708 [00:32<00:13, 206.87batch/s]

Training batches on cuda:  71%|███████▏  | 6931/9708 [00:33<00:13, 207.39batch/s]

Training batches on cuda:  72%|███████▏  | 6953/9708 [00:33<00:13, 209.53batch/s]

Training batches on cuda:  72%|███████▏  | 6974/9708 [00:33<00:13, 208.44batch/s]

Training batches on cuda:  72%|███████▏  | 6996/9708 [00:33<00:12, 209.47batch/s]

Training batches on cuda:  72%|███████▏  | 7017/9708 [00:33<00:12, 208.07batch/s]

Training batches on cuda:  72%|███████▏  | 7038/9708 [00:33<00:12, 205.88batch/s]

Training batches on cuda:  73%|███████▎  | 7059/9708 [00:33<00:12, 206.64batch/s]

Training batches on cuda:  73%|███████▎  | 7081/9708 [00:33<00:12, 208.81batch/s]

Training batches on cuda:  73%|███████▎  | 7102/9708 [00:33<00:12, 204.75batch/s]

Training batches on cuda:  73%|███████▎  | 7125/9708 [00:34<00:12, 209.28batch/s]

Training batches on cuda:  74%|███████▎  | 7149/9708 [00:34<00:11, 217.04batch/s]

Training batches on cuda:  74%|███████▍  | 7173/9708 [00:34<00:11, 222.91batch/s]

Training batches on cuda:  74%|███████▍  | 7197/9708 [00:34<00:11, 226.06batch/s]

Training batches on cuda:  74%|███████▍  | 7220/9708 [00:34<00:11, 220.82batch/s]

Training batches on cuda:  75%|███████▍  | 7243/9708 [00:34<00:11, 216.08batch/s]

Training batches on cuda:  75%|███████▍  | 7271/9708 [00:34<00:10, 230.46batch/s]

Training batches on cuda:  75%|███████▌  | 7299/9708 [00:34<00:09, 241.57batch/s]

Training batches on cuda:  75%|███████▌  | 7326/9708 [00:34<00:09, 248.35batch/s]

Training batches on cuda:  76%|███████▌  | 7354/9708 [00:34<00:09, 254.63batch/s]

Training batches on cuda:  76%|███████▌  | 7381/9708 [00:35<00:08, 258.70batch/s]

Training batches on cuda:  76%|███████▋  | 7408/9708 [00:35<00:08, 261.94batch/s]

Training batches on cuda:  77%|███████▋  | 7436/9708 [00:35<00:08, 265.44batch/s]

Training batches on cuda:  77%|███████▋  | 7463/9708 [00:35<00:08, 265.22batch/s]

Training batches on cuda:  77%|███████▋  | 7491/9708 [00:35<00:08, 267.37batch/s]

Training batches on cuda:  77%|███████▋  | 7518/9708 [00:35<00:08, 254.87batch/s]

Training batches on cuda:  78%|███████▊  | 7544/9708 [00:35<00:08, 244.95batch/s]

Training batches on cuda:  78%|███████▊  | 7569/9708 [00:35<00:08, 239.79batch/s]

Training batches on cuda:  78%|███████▊  | 7594/9708 [00:35<00:09, 231.97batch/s]

Training batches on cuda:  78%|███████▊  | 7618/9708 [00:36<00:09, 223.95batch/s]

Training batches on cuda:  79%|███████▊  | 7641/9708 [00:36<00:09, 224.98batch/s]

Training batches on cuda:  79%|███████▉  | 7665/9708 [00:36<00:08, 228.28batch/s]

Training batches on cuda:  79%|███████▉  | 7691/9708 [00:36<00:08, 235.70batch/s]

Training batches on cuda:  79%|███████▉  | 7716/9708 [00:36<00:08, 237.92batch/s]

Training batches on cuda:  80%|███████▉  | 7740/9708 [00:36<00:08, 233.80batch/s]

Training batches on cuda:  80%|███████▉  | 7764/9708 [00:36<00:08, 230.45batch/s]

Training batches on cuda:  80%|████████  | 7788/9708 [00:36<00:08, 230.06batch/s]

Training batches on cuda:  80%|████████  | 7812/9708 [00:36<00:08, 229.88batch/s]

Training batches on cuda:  81%|████████  | 7836/9708 [00:36<00:08, 229.65batch/s]

Training batches on cuda:  81%|████████  | 7859/9708 [00:37<00:08, 229.15batch/s]

Training batches on cuda:  81%|████████  | 7883/9708 [00:37<00:07, 229.82batch/s]

Training batches on cuda:  81%|████████▏ | 7906/9708 [00:37<00:07, 228.98batch/s]

Training batches on cuda:  82%|████████▏ | 7929/9708 [00:37<00:07, 229.04batch/s]

Training batches on cuda:  82%|████████▏ | 7952/9708 [00:37<00:07, 228.14batch/s]

Training batches on cuda:  82%|████████▏ | 7975/9708 [00:37<00:07, 227.18batch/s]

Training batches on cuda:  82%|████████▏ | 7998/9708 [00:37<00:07, 227.10batch/s]

Training batches on cuda:  83%|████████▎ | 8021/9708 [00:37<00:07, 227.41batch/s]

Training batches on cuda:  83%|████████▎ | 8044/9708 [00:37<00:07, 220.81batch/s]

Training batches on cuda:  83%|████████▎ | 8067/9708 [00:37<00:07, 222.85batch/s]

Training batches on cuda:  83%|████████▎ | 8090/9708 [00:38<00:07, 223.77batch/s]

Training batches on cuda:  84%|████████▎ | 8113/9708 [00:38<00:07, 221.12batch/s]

Training batches on cuda:  84%|████████▍ | 8136/9708 [00:38<00:07, 214.86batch/s]

Training batches on cuda:  84%|████████▍ | 8158/9708 [00:38<00:07, 207.61batch/s]

Training batches on cuda:  84%|████████▍ | 8179/9708 [00:38<00:07, 207.13batch/s]

Training batches on cuda:  84%|████████▍ | 8201/9708 [00:38<00:07, 208.89batch/s]

Training batches on cuda:  85%|████████▍ | 8224/9708 [00:38<00:06, 214.57batch/s]

Training batches on cuda:  85%|████████▍ | 8247/9708 [00:38<00:06, 218.81batch/s]

Training batches on cuda:  85%|████████▌ | 8272/9708 [00:38<00:06, 224.94batch/s]

Training batches on cuda:  85%|████████▌ | 8295/9708 [00:39<00:06, 215.81batch/s]

Training batches on cuda:  86%|████████▌ | 8317/9708 [00:39<00:06, 211.69batch/s]

Training batches on cuda:  86%|████████▌ | 8339/9708 [00:39<00:06, 210.44batch/s]

Training batches on cuda:  86%|████████▌ | 8362/9708 [00:39<00:06, 213.26batch/s]

Training batches on cuda:  86%|████████▋ | 8384/9708 [00:39<00:06, 209.21batch/s]

Training batches on cuda:  87%|████████▋ | 8405/9708 [00:39<00:06, 207.03batch/s]

Training batches on cuda:  87%|████████▋ | 8426/9708 [00:39<00:06, 205.33batch/s]

Training batches on cuda:  87%|████████▋ | 8447/9708 [00:39<00:06, 202.34batch/s]

Training batches on cuda:  87%|████████▋ | 8469/9708 [00:39<00:06, 206.28batch/s]

Training batches on cuda:  87%|████████▋ | 8491/9708 [00:40<00:05, 207.90batch/s]

Training batches on cuda:  88%|████████▊ | 8513/9708 [00:40<00:05, 210.74batch/s]

Training batches on cuda:  88%|████████▊ | 8536/9708 [00:40<00:05, 215.70batch/s]

Training batches on cuda:  88%|████████▊ | 8559/9708 [00:40<00:05, 219.32batch/s]

Training batches on cuda:  88%|████████▊ | 8582/9708 [00:40<00:05, 221.08batch/s]

Training batches on cuda:  89%|████████▊ | 8606/9708 [00:40<00:04, 223.92batch/s]

Training batches on cuda:  89%|████████▉ | 8629/9708 [00:40<00:04, 225.43batch/s]

Training batches on cuda:  89%|████████▉ | 8652/9708 [00:40<00:04, 226.72batch/s]

Training batches on cuda:  89%|████████▉ | 8675/9708 [00:40<00:04, 225.04batch/s]

Training batches on cuda:  90%|████████▉ | 8698/9708 [00:40<00:04, 226.06batch/s]

Training batches on cuda:  90%|████████▉ | 8721/9708 [00:41<00:04, 225.39batch/s]

Training batches on cuda:  90%|█████████ | 8744/9708 [00:41<00:04, 225.81batch/s]

Training batches on cuda:  90%|█████████ | 8767/9708 [00:41<00:04, 225.51batch/s]

Training batches on cuda:  91%|█████████ | 8790/9708 [00:41<00:04, 218.13batch/s]

Training batches on cuda:  91%|█████████ | 8812/9708 [00:41<00:04, 212.81batch/s]

Training batches on cuda:  91%|█████████ | 8834/9708 [00:41<00:04, 211.19batch/s]

Training batches on cuda:  91%|█████████ | 8856/9708 [00:41<00:04, 211.88batch/s]

Training batches on cuda:  91%|█████████▏| 8878/9708 [00:41<00:03, 211.17batch/s]

Training batches on cuda:  92%|█████████▏| 8900/9708 [00:41<00:03, 212.42batch/s]

Training batches on cuda:  92%|█████████▏| 8922/9708 [00:41<00:03, 209.89batch/s]

Training batches on cuda:  92%|█████████▏| 8944/9708 [00:42<00:03, 205.27batch/s]

Training batches on cuda:  92%|█████████▏| 8966/9708 [00:42<00:03, 208.55batch/s]

Training batches on cuda:  93%|█████████▎| 8988/9708 [00:42<00:03, 209.70batch/s]

Training batches on cuda:  93%|█████████▎| 9009/9708 [00:42<00:03, 208.32batch/s]

Training batches on cuda:  93%|█████████▎| 9031/9708 [00:42<00:03, 210.99batch/s]

Training batches on cuda:  93%|█████████▎| 9053/9708 [00:42<00:03, 211.14batch/s]

Training batches on cuda:  93%|█████████▎| 9075/9708 [00:42<00:03, 209.92batch/s]

Training batches on cuda:  94%|█████████▎| 9097/9708 [00:42<00:02, 203.73batch/s]

Training batches on cuda:  94%|█████████▍| 9118/9708 [00:42<00:03, 195.10batch/s]

Training batches on cuda:  94%|█████████▍| 9138/9708 [00:43<00:03, 183.90batch/s]

Training batches on cuda:  94%|█████████▍| 9158/9708 [00:43<00:02, 184.98batch/s]

Training batches on cuda:  95%|█████████▍| 9178/9708 [00:43<00:02, 189.16batch/s]

Training batches on cuda:  95%|█████████▍| 9199/9708 [00:43<00:02, 193.29batch/s]

Training batches on cuda:  95%|█████████▍| 9220/9708 [00:43<00:02, 196.60batch/s]

Training batches on cuda:  95%|█████████▌| 9241/9708 [00:43<00:02, 198.12batch/s]

Training batches on cuda:  95%|█████████▌| 9265/9708 [00:43<00:02, 207.74batch/s]

Training batches on cuda:  96%|█████████▌| 9288/9708 [00:43<00:01, 212.19batch/s]

Training batches on cuda:  96%|█████████▌| 9312/9708 [00:43<00:01, 218.75batch/s]

Training batches on cuda:  96%|█████████▌| 9335/9708 [00:43<00:01, 221.44batch/s]

Training batches on cuda:  96%|█████████▋| 9358/9708 [00:44<00:01, 221.42batch/s]

Training batches on cuda:  97%|█████████▋| 9382/9708 [00:44<00:01, 225.88batch/s]

Training batches on cuda:  97%|█████████▋| 9405/9708 [00:44<00:01, 225.80batch/s]

Training batches on cuda:  97%|█████████▋| 9428/9708 [00:44<00:01, 224.80batch/s]

Training batches on cuda:  97%|█████████▋| 9451/9708 [00:44<00:01, 224.23batch/s]

Training batches on cuda:  98%|█████████▊| 9474/9708 [00:44<00:01, 224.93batch/s]

Training batches on cuda:  98%|█████████▊| 9497/9708 [00:44<00:00, 221.29batch/s]

Training batches on cuda:  98%|█████████▊| 9520/9708 [00:44<00:00, 218.04batch/s]

Training batches on cuda:  98%|█████████▊| 9542/9708 [00:44<00:00, 216.81batch/s]

Training batches on cuda:  99%|█████████▊| 9564/9708 [00:45<00:00, 213.00batch/s]

Training batches on cuda:  99%|█████████▊| 9586/9708 [00:45<00:00, 213.04batch/s]

Training batches on cuda:  99%|█████████▉| 9608/9708 [00:45<00:00, 213.17batch/s]

Training batches on cuda:  99%|█████████▉| 9630/9708 [00:45<00:00, 212.74batch/s]

Training batches on cuda:  99%|█████████▉| 9652/9708 [00:45<00:00, 212.88batch/s]

Training batches on cuda: 100%|█████████▉| 9675/9708 [00:45<00:00, 217.70batch/s]

Training batches on cuda: 100%|█████████▉| 9698/9708 [00:45<00:00, 220.59batch/s]

                                                                                 
Training epochs on cuda:  40%|████      | 4/10 [03:53<04:41, 46.84s/epoch, loss=0.0416, prev_loss=0.0417]
Training epochs on cuda:  50%|█████     | 5/10 [03:53<03:52, 46.57s/epoch, loss=0.0416, prev_loss=0.0417]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 13/9708 [00:00<01:17, 125.36batch/s]

Training batches on cuda:   0%|          | 37/9708 [00:00<01:06, 145.46batch/s]

Training batches on cuda:   1%|          | 60/9708 [00:00<00:59, 163.44batch/s]

Training batches on cuda:   1%|          | 83/9708 [00:00<00:53, 178.94batch/s]

Training batches on cuda:   1%|          | 107/9708 [00:00<00:49, 192.74batch/s]

Training batches on cuda:   1%|▏         | 131/9708 [00:00<00:47, 202.76batch/s]

Training batches on cuda:   2%|▏         | 155/9708 [00:00<00:45, 210.80batch/s]

Training batches on cuda:   2%|▏         | 179/9708 [00:00<00:44, 216.12batch/s]

Training batches on cuda:   2%|▏         | 203/9708 [00:00<00:43, 220.66batch/s]

Training batches on cuda:   2%|▏         | 226/9708 [00:01<00:42, 221.62batch/s]

Training batches on cuda:   3%|▎         | 249/9708 [00:01<00:42, 223.69batch/s]

Training batches on cuda:   3%|▎         | 273/9708 [00:01<00:41, 226.35batch/s]

Training batches on cuda:   3%|▎         | 297/9708 [00:01<00:41, 227.65batch/s]

Training batches on cuda:   3%|▎         | 320/9708 [00:01<00:42, 222.57batch/s]

Training batches on cuda:   4%|▎         | 343/9708 [00:01<00:45, 206.66batch/s]

Training batches on cuda:   4%|▍         | 365/9708 [00:01<00:44, 208.01batch/s]

Training batches on cuda:   4%|▍         | 388/9708 [00:01<00:43, 213.94batch/s]

Training batches on cuda:   4%|▍         | 411/9708 [00:01<00:43, 215.05batch/s]

Training batches on cuda:   4%|▍         | 433/9708 [00:01<00:43, 212.83batch/s]

Training batches on cuda:   5%|▍         | 455/9708 [00:02<00:44, 206.84batch/s]

Training batches on cuda:   5%|▍         | 476/9708 [00:02<00:44, 207.08batch/s]

Training batches on cuda:   5%|▌         | 497/9708 [00:02<00:44, 207.33batch/s]

Training batches on cuda:   5%|▌         | 518/9708 [00:02<00:44, 207.73batch/s]

Training batches on cuda:   6%|▌         | 539/9708 [00:02<00:50, 182.80batch/s]

Training batches on cuda:   6%|▌         | 561/9708 [00:02<00:47, 190.80batch/s]

Training batches on cuda:   6%|▌         | 583/9708 [00:02<00:46, 196.82batch/s]

Training batches on cuda:   6%|▌         | 604/9708 [00:02<00:45, 200.59batch/s]

Training batches on cuda:   6%|▋         | 626/9708 [00:02<00:44, 203.33batch/s]

Training batches on cuda:   7%|▋         | 648/9708 [00:03<00:43, 207.29batch/s]

Training batches on cuda:   7%|▋         | 670/9708 [00:03<00:43, 208.25batch/s]

Training batches on cuda:   7%|▋         | 692/9708 [00:03<00:42, 210.18batch/s]

Training batches on cuda:   7%|▋         | 714/9708 [00:03<00:42, 212.31batch/s]

Training batches on cuda:   8%|▊         | 736/9708 [00:03<00:42, 212.52batch/s]

Training batches on cuda:   8%|▊         | 758/9708 [00:03<00:41, 214.69batch/s]

Training batches on cuda:   8%|▊         | 781/9708 [00:03<00:41, 216.34batch/s]

Training batches on cuda:   8%|▊         | 805/9708 [00:03<00:40, 221.76batch/s]

Training batches on cuda:   9%|▊         | 829/9708 [00:03<00:39, 224.83batch/s]

Training batches on cuda:   9%|▉         | 853/9708 [00:03<00:39, 225.85batch/s]

Training batches on cuda:   9%|▉         | 876/9708 [00:04<00:40, 219.47batch/s]

Training batches on cuda:   9%|▉         | 899/9708 [00:04<00:40, 218.92batch/s]

Training batches on cuda:   9%|▉         | 921/9708 [00:04<00:41, 213.79batch/s]

Training batches on cuda:  10%|▉         | 943/9708 [00:04<00:41, 211.36batch/s]

Training batches on cuda:  10%|▉         | 965/9708 [00:04<00:41, 209.71batch/s]

Training batches on cuda:  10%|█         | 987/9708 [00:04<00:41, 209.62batch/s]

Training batches on cuda:  10%|█         | 1009/9708 [00:04<00:41, 207.95batch/s]

Training batches on cuda:  11%|█         | 1031/9708 [00:04<00:41, 208.77batch/s]

Training batches on cuda:  11%|█         | 1053/9708 [00:04<00:41, 209.32batch/s]

Training batches on cuda:  11%|█         | 1075/9708 [00:05<00:40, 212.01batch/s]

Training batches on cuda:  11%|█▏        | 1097/9708 [00:05<00:40, 210.37batch/s]

Training batches on cuda:  12%|█▏        | 1120/9708 [00:05<00:40, 213.75batch/s]

Training batches on cuda:  12%|█▏        | 1142/9708 [00:05<00:40, 210.94batch/s]

Training batches on cuda:  12%|█▏        | 1165/9708 [00:05<00:39, 214.28batch/s]

Training batches on cuda:  12%|█▏        | 1187/9708 [00:05<00:40, 208.11batch/s]

Training batches on cuda:  12%|█▏        | 1208/9708 [00:05<00:45, 185.81batch/s]

Training batches on cuda:  13%|█▎        | 1230/9708 [00:05<00:43, 193.13batch/s]

Training batches on cuda:  13%|█▎        | 1252/9708 [00:05<00:42, 198.29batch/s]

Training batches on cuda:  13%|█▎        | 1273/9708 [00:06<00:42, 196.48batch/s]

Training batches on cuda:  13%|█▎        | 1295/9708 [00:06<00:41, 200.76batch/s]

Training batches on cuda:  14%|█▎        | 1318/9708 [00:06<00:40, 207.58batch/s]

Training batches on cuda:  14%|█▍        | 1341/9708 [00:06<00:39, 212.41batch/s]

Training batches on cuda:  14%|█▍        | 1363/9708 [00:06<00:38, 214.49batch/s]

Training batches on cuda:  14%|█▍        | 1387/9708 [00:06<00:37, 221.01batch/s]

Training batches on cuda:  15%|█▍        | 1410/9708 [00:06<00:37, 223.63batch/s]

Training batches on cuda:  15%|█▍        | 1433/9708 [00:06<00:40, 203.64batch/s]

Training batches on cuda:  15%|█▍        | 1454/9708 [00:06<00:42, 195.43batch/s]

Training batches on cuda:  15%|█▌        | 1475/9708 [00:07<00:41, 197.78batch/s]

Training batches on cuda:  15%|█▌        | 1496/9708 [00:07<00:41, 196.75batch/s]

Training batches on cuda:  16%|█▌        | 1516/9708 [00:07<00:41, 195.60batch/s]

Training batches on cuda:  16%|█▌        | 1536/9708 [00:07<00:43, 188.97batch/s]

Training batches on cuda:  16%|█▌        | 1559/9708 [00:07<00:41, 197.00batch/s]

Training batches on cuda:  16%|█▋        | 1579/9708 [00:07<00:46, 175.87batch/s]

Training batches on cuda:  16%|█▋        | 1598/9708 [00:07<00:45, 179.47batch/s]

Training batches on cuda:  17%|█▋        | 1619/9708 [00:07<00:43, 185.23batch/s]

Training batches on cuda:  17%|█▋        | 1639/9708 [00:07<00:42, 189.13batch/s]

Training batches on cuda:  17%|█▋        | 1659/9708 [00:07<00:41, 191.92batch/s]

Training batches on cuda:  17%|█▋        | 1679/9708 [00:08<00:41, 192.22batch/s]

Training batches on cuda:  18%|█▊        | 1700/9708 [00:08<00:41, 194.61batch/s]

Training batches on cuda:  18%|█▊        | 1720/9708 [00:08<00:40, 195.38batch/s]

Training batches on cuda:  18%|█▊        | 1740/9708 [00:08<00:40, 194.97batch/s]

Training batches on cuda:  18%|█▊        | 1760/9708 [00:08<00:40, 194.84batch/s]

Training batches on cuda:  18%|█▊        | 1780/9708 [00:08<00:40, 194.66batch/s]

Training batches on cuda:  19%|█▊        | 1800/9708 [00:08<00:40, 193.33batch/s]

Training batches on cuda:  19%|█▉        | 1821/9708 [00:08<00:40, 196.18batch/s]

Training batches on cuda:  19%|█▉        | 1845/9708 [00:08<00:38, 205.45batch/s]

Training batches on cuda:  19%|█▉        | 1868/9708 [00:09<00:37, 210.19batch/s]

Training batches on cuda:  19%|█▉        | 1891/9708 [00:09<00:36, 215.71batch/s]

Training batches on cuda:  20%|█▉        | 1914/9708 [00:09<00:35, 219.17batch/s]

Training batches on cuda:  20%|█▉        | 1937/9708 [00:09<00:35, 221.72batch/s]

Training batches on cuda:  20%|██        | 1960/9708 [00:09<00:34, 222.88batch/s]

Training batches on cuda:  20%|██        | 1983/9708 [00:09<00:34, 221.61batch/s]

Training batches on cuda:  21%|██        | 2006/9708 [00:09<00:35, 214.83batch/s]

Training batches on cuda:  21%|██        | 2028/9708 [00:09<00:38, 197.35batch/s]

Training batches on cuda:  21%|██        | 2049/9708 [00:09<00:40, 189.94batch/s]

Training batches on cuda:  21%|██▏       | 2070/9708 [00:10<00:39, 193.24batch/s]

Training batches on cuda:  22%|██▏       | 2090/9708 [00:10<00:39, 190.99batch/s]

Training batches on cuda:  22%|██▏       | 2111/9708 [00:10<00:39, 194.62batch/s]

Training batches on cuda:  22%|██▏       | 2131/9708 [00:10<00:39, 193.87batch/s]

Training batches on cuda:  22%|██▏       | 2151/9708 [00:10<00:39, 193.20batch/s]

Training batches on cuda:  22%|██▏       | 2172/9708 [00:10<00:38, 195.65batch/s]

Training batches on cuda:  23%|██▎       | 2193/9708 [00:10<00:38, 196.90batch/s]

Training batches on cuda:  23%|██▎       | 2213/9708 [00:10<00:38, 195.41batch/s]

Training batches on cuda:  23%|██▎       | 2233/9708 [00:10<00:38, 195.18batch/s]

Training batches on cuda:  23%|██▎       | 2253/9708 [00:10<00:37, 196.21batch/s]

Training batches on cuda:  23%|██▎       | 2273/9708 [00:11<00:38, 192.53batch/s]

Training batches on cuda:  24%|██▎       | 2293/9708 [00:11<00:38, 192.05batch/s]

Training batches on cuda:  24%|██▍       | 2313/9708 [00:11<00:38, 191.80batch/s]

Training batches on cuda:  24%|██▍       | 2334/9708 [00:11<00:37, 195.52batch/s]

Training batches on cuda:  24%|██▍       | 2358/9708 [00:11<00:35, 205.18batch/s]

Training batches on cuda:  25%|██▍       | 2381/9708 [00:11<00:34, 211.21batch/s]

Training batches on cuda:  25%|██▍       | 2403/9708 [00:11<00:34, 213.34batch/s]

Training batches on cuda:  25%|██▍       | 2426/9708 [00:11<00:33, 216.53batch/s]

Training batches on cuda:  25%|██▌       | 2448/9708 [00:11<00:35, 203.49batch/s]

Training batches on cuda:  25%|██▌       | 2469/9708 [00:12<00:36, 198.77batch/s]

Training batches on cuda:  26%|██▌       | 2490/9708 [00:12<00:36, 197.06batch/s]

Training batches on cuda:  26%|██▌       | 2511/9708 [00:12<00:35, 200.58batch/s]

Training batches on cuda:  26%|██▌       | 2533/9708 [00:12<00:34, 205.79batch/s]

Training batches on cuda:  26%|██▋       | 2555/9708 [00:12<00:34, 207.77batch/s]

Training batches on cuda:  27%|██▋       | 2577/9708 [00:12<00:33, 211.07batch/s]

Training batches on cuda:  27%|██▋       | 2599/9708 [00:12<00:33, 213.62batch/s]

Training batches on cuda:  27%|██▋       | 2621/9708 [00:12<00:33, 214.18batch/s]

Training batches on cuda:  27%|██▋       | 2643/9708 [00:12<00:33, 214.07batch/s]

Training batches on cuda:  27%|██▋       | 2666/9708 [00:12<00:32, 217.09batch/s]

Training batches on cuda:  28%|██▊       | 2689/9708 [00:13<00:31, 220.39batch/s]

Training batches on cuda:  28%|██▊       | 2712/9708 [00:13<00:31, 222.91batch/s]

Training batches on cuda:  28%|██▊       | 2735/9708 [00:13<00:31, 224.17batch/s]

Training batches on cuda:  28%|██▊       | 2760/9708 [00:13<00:30, 228.67batch/s]

Training batches on cuda:  29%|██▊       | 2784/9708 [00:13<00:30, 230.25batch/s]

Training batches on cuda:  29%|██▉       | 2808/9708 [00:13<00:30, 226.20batch/s]

Training batches on cuda:  29%|██▉       | 2831/9708 [00:13<00:30, 222.79batch/s]

Training batches on cuda:  29%|██▉       | 2854/9708 [00:13<00:30, 224.16batch/s]

Training batches on cuda:  30%|██▉       | 2877/9708 [00:13<00:33, 206.05batch/s]

Training batches on cuda:  30%|██▉       | 2898/9708 [00:14<00:34, 198.64batch/s]

Training batches on cuda:  30%|███       | 2919/9708 [00:14<00:33, 199.92batch/s]

Training batches on cuda:  30%|███       | 2940/9708 [00:14<00:36, 185.50batch/s]

Training batches on cuda:  30%|███       | 2959/9708 [00:14<00:38, 175.31batch/s]

Training batches on cuda:  31%|███       | 2979/9708 [00:14<00:37, 180.34batch/s]

Training batches on cuda:  31%|███       | 3000/9708 [00:14<00:35, 188.26batch/s]

Training batches on cuda:  31%|███       | 3022/9708 [00:14<00:34, 196.19batch/s]

Training batches on cuda:  31%|███▏      | 3042/9708 [00:14<00:33, 196.81batch/s]

Training batches on cuda:  32%|███▏      | 3062/9708 [00:14<00:36, 182.89batch/s]

Training batches on cuda:  32%|███▏      | 3084/9708 [00:14<00:34, 191.73batch/s]

Training batches on cuda:  32%|███▏      | 3105/9708 [00:15<00:33, 195.13batch/s]

Training batches on cuda:  32%|███▏      | 3127/9708 [00:15<00:32, 201.97batch/s]

Training batches on cuda:  32%|███▏      | 3148/9708 [00:15<00:32, 203.38batch/s]

Training batches on cuda:  33%|███▎      | 3169/9708 [00:15<00:33, 194.46batch/s]

Training batches on cuda:  33%|███▎      | 3191/9708 [00:15<00:32, 201.35batch/s]

Training batches on cuda:  33%|███▎      | 3212/9708 [00:15<00:32, 199.52batch/s]

Training batches on cuda:  33%|███▎      | 3233/9708 [00:15<00:32, 196.46batch/s]

Training batches on cuda:  34%|███▎      | 3253/9708 [00:15<00:34, 186.09batch/s]

Training batches on cuda:  34%|███▎      | 3273/9708 [00:15<00:34, 187.80batch/s]

Training batches on cuda:  34%|███▍      | 3292/9708 [00:16<00:34, 186.24batch/s]

Training batches on cuda:  34%|███▍      | 3315/9708 [00:16<00:32, 195.64batch/s]

Training batches on cuda:  34%|███▍      | 3336/9708 [00:16<00:31, 199.16batch/s]

Training batches on cuda:  35%|███▍      | 3358/9708 [00:16<00:31, 204.21batch/s]

Training batches on cuda:  35%|███▍      | 3379/9708 [00:16<00:30, 205.34batch/s]

Training batches on cuda:  35%|███▌      | 3402/9708 [00:16<00:30, 209.87batch/s]

Training batches on cuda:  35%|███▌      | 3424/9708 [00:16<00:30, 207.08batch/s]

Training batches on cuda:  36%|███▌      | 3447/9708 [00:16<00:29, 212.78batch/s]

Training batches on cuda:  36%|███▌      | 3470/9708 [00:16<00:29, 214.82batch/s]

Training batches on cuda:  36%|███▌      | 3494/9708 [00:16<00:28, 219.50batch/s]

Training batches on cuda:  36%|███▌      | 3518/9708 [00:17<00:27, 223.25batch/s]

Training batches on cuda:  36%|███▋      | 3541/9708 [00:17<00:27, 222.65batch/s]

Training batches on cuda:  37%|███▋      | 3565/9708 [00:17<00:27, 224.10batch/s]

Training batches on cuda:  37%|███▋      | 3588/9708 [00:17<00:28, 217.46batch/s]

Training batches on cuda:  37%|███▋      | 3610/9708 [00:17<00:27, 218.16batch/s]

Training batches on cuda:  37%|███▋      | 3632/9708 [00:17<00:31, 190.71batch/s]

Training batches on cuda:  38%|███▊      | 3654/9708 [00:17<00:30, 196.95batch/s]

Training batches on cuda:  38%|███▊      | 3676/9708 [00:17<00:29, 203.19batch/s]

Training batches on cuda:  38%|███▊      | 3698/9708 [00:17<00:29, 206.54batch/s]

Training batches on cuda:  38%|███▊      | 3721/9708 [00:18<00:28, 210.46batch/s]

Training batches on cuda:  39%|███▊      | 3743/9708 [00:18<00:29, 204.89batch/s]

Training batches on cuda:  39%|███▉      | 3764/9708 [00:18<00:28, 205.17batch/s]

Training batches on cuda:  39%|███▉      | 3785/9708 [00:18<00:29, 203.87batch/s]

Training batches on cuda:  39%|███▉      | 3806/9708 [00:18<00:29, 203.38batch/s]

Training batches on cuda:  39%|███▉      | 3827/9708 [00:18<00:29, 200.44batch/s]

Training batches on cuda:  40%|███▉      | 3848/9708 [00:18<00:29, 201.77batch/s]

Training batches on cuda:  40%|███▉      | 3869/9708 [00:18<00:30, 190.52batch/s]

Training batches on cuda:  40%|████      | 3890/9708 [00:18<00:29, 194.28batch/s]

Training batches on cuda:  40%|████      | 3912/9708 [00:19<00:28, 200.03batch/s]

Training batches on cuda:  41%|████      | 3934/9708 [00:19<00:28, 205.10batch/s]

Training batches on cuda:  41%|████      | 3957/9708 [00:19<00:27, 209.43batch/s]

Training batches on cuda:  41%|████      | 3980/9708 [00:19<00:26, 212.95batch/s]

Training batches on cuda:  41%|████      | 4004/9708 [00:19<00:26, 217.84batch/s]

Training batches on cuda:  41%|████▏     | 4027/9708 [00:19<00:25, 220.84batch/s]

Training batches on cuda:  42%|████▏     | 4050/9708 [00:19<00:25, 222.09batch/s]

Training batches on cuda:  42%|████▏     | 4073/9708 [00:19<00:25, 223.03batch/s]

Training batches on cuda:  42%|████▏     | 4096/9708 [00:19<00:25, 224.43batch/s]

Training batches on cuda:  42%|████▏     | 4119/9708 [00:19<00:24, 225.95batch/s]

Training batches on cuda:  43%|████▎     | 4142/9708 [00:20<00:24, 226.72batch/s]

Training batches on cuda:  43%|████▎     | 4165/9708 [00:20<00:24, 226.29batch/s]

Training batches on cuda:  43%|████▎     | 4189/9708 [00:20<00:24, 227.78batch/s]

Training batches on cuda:  43%|████▎     | 4212/9708 [00:20<00:25, 219.75batch/s]

Training batches on cuda:  44%|████▎     | 4235/9708 [00:20<00:24, 220.19batch/s]

Training batches on cuda:  44%|████▍     | 4258/9708 [00:20<00:24, 222.81batch/s]

Training batches on cuda:  44%|████▍     | 4281/9708 [00:20<00:24, 223.48batch/s]

Training batches on cuda:  44%|████▍     | 4305/9708 [00:20<00:23, 225.31batch/s]

Training batches on cuda:  45%|████▍     | 4328/9708 [00:20<00:23, 226.27batch/s]

Training batches on cuda:  45%|████▍     | 4351/9708 [00:21<00:23, 223.30batch/s]

Training batches on cuda:  45%|████▌     | 4374/9708 [00:21<00:24, 216.95batch/s]

Training batches on cuda:  45%|████▌     | 4396/9708 [00:21<00:24, 216.60batch/s]

Training batches on cuda:  46%|████▌     | 4419/9708 [00:21<00:24, 217.62batch/s]

Training batches on cuda:  46%|████▌     | 4441/9708 [00:21<00:24, 217.03batch/s]

Training batches on cuda:  46%|████▌     | 4464/9708 [00:21<00:23, 218.75batch/s]

Training batches on cuda:  46%|████▌     | 4487/9708 [00:21<00:23, 219.47batch/s]

Training batches on cuda:  46%|████▋     | 4509/9708 [00:21<00:23, 218.88batch/s]

Training batches on cuda:  47%|████▋     | 4532/9708 [00:21<00:23, 220.07batch/s]

Training batches on cuda:  47%|████▋     | 4555/9708 [00:21<00:23, 221.36batch/s]

Training batches on cuda:  47%|████▋     | 4578/9708 [00:22<00:23, 219.59batch/s]

Training batches on cuda:  47%|████▋     | 4601/9708 [00:22<00:23, 221.45batch/s]

Training batches on cuda:  48%|████▊     | 4624/9708 [00:22<00:22, 222.12batch/s]

Training batches on cuda:  48%|████▊     | 4647/9708 [00:22<00:22, 220.42batch/s]

Training batches on cuda:  48%|████▊     | 4670/9708 [00:22<00:22, 222.19batch/s]

Training batches on cuda:  48%|████▊     | 4693/9708 [00:22<00:22, 222.28batch/s]

Training batches on cuda:  49%|████▊     | 4716/9708 [00:22<00:24, 200.70batch/s]

Training batches on cuda:  49%|████▉     | 4737/9708 [00:22<00:24, 201.22batch/s]

Training batches on cuda:  49%|████▉     | 4758/9708 [00:22<00:25, 197.46batch/s]

Training batches on cuda:  49%|████▉     | 4778/9708 [00:23<00:25, 196.09batch/s]

Training batches on cuda:  49%|████▉     | 4798/9708 [00:23<00:25, 193.74batch/s]

Training batches on cuda:  50%|████▉     | 4818/9708 [00:23<00:25, 193.25batch/s]

Training batches on cuda:  50%|████▉     | 4839/9708 [00:23<00:24, 197.91batch/s]

Training batches on cuda:  50%|█████     | 4861/9708 [00:23<00:23, 203.96batch/s]

Training batches on cuda:  50%|█████     | 4883/9708 [00:23<00:23, 206.82batch/s]

Training batches on cuda:  51%|█████     | 4904/9708 [00:23<00:23, 205.15batch/s]

Training batches on cuda:  51%|█████     | 4926/9708 [00:23<00:23, 207.78batch/s]

Training batches on cuda:  51%|█████     | 4948/9708 [00:23<00:23, 203.13batch/s]

Training batches on cuda:  51%|█████     | 4971/9708 [00:23<00:22, 210.11batch/s]

Training batches on cuda:  51%|█████▏    | 4993/9708 [00:24<00:22, 212.93batch/s]

Training batches on cuda:  52%|█████▏    | 5017/9708 [00:24<00:21, 217.50batch/s]

Training batches on cuda:  52%|█████▏    | 5040/9708 [00:24<00:21, 219.08batch/s]

Training batches on cuda:  52%|█████▏    | 5063/9708 [00:24<00:21, 220.63batch/s]

Training batches on cuda:  52%|█████▏    | 5087/9708 [00:24<00:20, 223.27batch/s]

Training batches on cuda:  53%|█████▎    | 5110/9708 [00:24<00:20, 221.89batch/s]

Training batches on cuda:  53%|█████▎    | 5133/9708 [00:24<00:20, 222.86batch/s]

Training batches on cuda:  53%|█████▎    | 5156/9708 [00:24<00:20, 222.09batch/s]

Training batches on cuda:  53%|█████▎    | 5179/9708 [00:24<00:20, 220.00batch/s]

Training batches on cuda:  54%|█████▎    | 5202/9708 [00:25<00:21, 207.88batch/s]

Training batches on cuda:  54%|█████▍    | 5224/9708 [00:25<00:21, 209.60batch/s]

Training batches on cuda:  54%|█████▍    | 5246/9708 [00:25<00:21, 207.51batch/s]

Training batches on cuda:  54%|█████▍    | 5269/9708 [00:25<00:20, 212.28batch/s]

Training batches on cuda:  55%|█████▍    | 5292/9708 [00:25<00:20, 215.79batch/s]

Training batches on cuda:  55%|█████▍    | 5314/9708 [00:25<00:20, 216.74batch/s]

Training batches on cuda:  55%|█████▍    | 5336/9708 [00:25<00:20, 216.94batch/s]

Training batches on cuda:  55%|█████▌    | 5358/9708 [00:25<00:20, 212.60batch/s]

Training batches on cuda:  55%|█████▌    | 5380/9708 [00:25<00:20, 208.34batch/s]

Training batches on cuda:  56%|█████▌    | 5402/9708 [00:25<00:20, 209.26batch/s]

Training batches on cuda:  56%|█████▌    | 5424/9708 [00:26<00:20, 210.03batch/s]

Training batches on cuda:  56%|█████▌    | 5446/9708 [00:26<00:20, 207.90batch/s]

Training batches on cuda:  56%|█████▋    | 5467/9708 [00:26<00:20, 206.25batch/s]

Training batches on cuda:  57%|█████▋    | 5488/9708 [00:26<00:20, 206.07batch/s]

Training batches on cuda:  57%|█████▋    | 5512/9708 [00:26<00:19, 214.36batch/s]

Training batches on cuda:  57%|█████▋    | 5537/9708 [00:26<00:18, 222.37batch/s]

Training batches on cuda:  57%|█████▋    | 5562/9708 [00:26<00:18, 228.57batch/s]

Training batches on cuda:  58%|█████▊    | 5587/9708 [00:26<00:17, 233.11batch/s]

Training batches on cuda:  58%|█████▊    | 5612/9708 [00:26<00:17, 237.11batch/s]

Training batches on cuda:  58%|█████▊    | 5639/9708 [00:26<00:16, 243.94batch/s]

Training batches on cuda:  58%|█████▊    | 5667/9708 [00:27<00:16, 251.66batch/s]

Training batches on cuda:  59%|█████▊    | 5693/9708 [00:27<00:17, 235.70batch/s]

Training batches on cuda:  59%|█████▉    | 5717/9708 [00:27<00:17, 225.32batch/s]

Training batches on cuda:  59%|█████▉    | 5740/9708 [00:27<00:17, 224.87batch/s]

Training batches on cuda:  59%|█████▉    | 5763/9708 [00:27<00:17, 224.50batch/s]

Training batches on cuda:  60%|█████▉    | 5786/9708 [00:27<00:17, 225.38batch/s]

Training batches on cuda:  60%|█████▉    | 5810/9708 [00:27<00:17, 227.62batch/s]

Training batches on cuda:  60%|██████    | 5834/9708 [00:27<00:16, 230.09batch/s]

Training batches on cuda:  60%|██████    | 5859/9708 [00:27<00:16, 235.13batch/s]

Training batches on cuda:  61%|██████    | 5884/9708 [00:28<00:16, 238.22batch/s]

Training batches on cuda:  61%|██████    | 5908/9708 [00:28<00:16, 231.24batch/s]

Training batches on cuda:  61%|██████    | 5932/9708 [00:28<00:16, 223.19batch/s]

Training batches on cuda:  61%|██████▏   | 5955/9708 [00:28<00:17, 220.31batch/s]

Training batches on cuda:  62%|██████▏   | 5978/9708 [00:28<00:16, 220.52batch/s]

Training batches on cuda:  62%|██████▏   | 6002/9708 [00:28<00:16, 223.50batch/s]

Training batches on cuda:  62%|██████▏   | 6025/9708 [00:28<00:16, 223.28batch/s]

Training batches on cuda:  62%|██████▏   | 6049/9708 [00:28<00:16, 225.55batch/s]

Training batches on cuda:  63%|██████▎   | 6072/9708 [00:28<00:16, 224.90batch/s]

Training batches on cuda:  63%|██████▎   | 6095/9708 [00:29<00:16, 223.49batch/s]

Training batches on cuda:  63%|██████▎   | 6118/9708 [00:29<00:16, 221.13batch/s]

Training batches on cuda:  63%|██████▎   | 6141/9708 [00:29<00:16, 218.96batch/s]

Training batches on cuda:  63%|██████▎   | 6163/9708 [00:29<00:16, 219.24batch/s]

Training batches on cuda:  64%|██████▎   | 6185/9708 [00:29<00:17, 205.29batch/s]

Training batches on cuda:  64%|██████▍   | 6206/9708 [00:29<00:16, 206.60batch/s]

Training batches on cuda:  64%|██████▍   | 6229/9708 [00:29<00:16, 211.80batch/s]

Training batches on cuda:  64%|██████▍   | 6253/9708 [00:29<00:15, 217.34batch/s]

Training batches on cuda:  65%|██████▍   | 6275/9708 [00:29<00:15, 216.29batch/s]

Training batches on cuda:  65%|██████▍   | 6299/9708 [00:29<00:15, 221.60batch/s]

Training batches on cuda:  65%|██████▌   | 6322/9708 [00:30<00:15, 218.74batch/s]

Training batches on cuda:  65%|██████▌   | 6344/9708 [00:30<00:15, 218.94batch/s]

Training batches on cuda:  66%|██████▌   | 6368/9708 [00:30<00:14, 222.75batch/s]

Training batches on cuda:  66%|██████▌   | 6391/9708 [00:30<00:15, 217.67batch/s]

Training batches on cuda:  66%|██████▌   | 6414/9708 [00:30<00:14, 221.06batch/s]

Training batches on cuda:  66%|██████▋   | 6437/9708 [00:30<00:14, 218.59batch/s]

Training batches on cuda:  67%|██████▋   | 6461/9708 [00:30<00:14, 222.57batch/s]

Training batches on cuda:  67%|██████▋   | 6484/9708 [00:30<00:14, 219.59batch/s]

Training batches on cuda:  67%|██████▋   | 6507/9708 [00:30<00:14, 221.48batch/s]

Training batches on cuda:  67%|██████▋   | 6530/9708 [00:31<00:15, 209.66batch/s]

Training batches on cuda:  68%|██████▊   | 6553/9708 [00:31<00:14, 214.53batch/s]

Training batches on cuda:  68%|██████▊   | 6575/9708 [00:31<00:14, 211.20batch/s]

Training batches on cuda:  68%|██████▊   | 6597/9708 [00:31<00:14, 210.60batch/s]

Training batches on cuda:  68%|██████▊   | 6619/9708 [00:31<00:14, 208.33batch/s]

Training batches on cuda:  68%|██████▊   | 6641/9708 [00:31<00:14, 210.72batch/s]

Training batches on cuda:  69%|██████▊   | 6663/9708 [00:31<00:15, 202.24batch/s]

Training batches on cuda:  69%|██████▉   | 6684/9708 [00:31<00:15, 195.28batch/s]

Training batches on cuda:  69%|██████▉   | 6705/9708 [00:31<00:15, 198.90batch/s]

Training batches on cuda:  69%|██████▉   | 6727/9708 [00:31<00:14, 203.25batch/s]

Training batches on cuda:  70%|██████▉   | 6750/9708 [00:32<00:14, 209.87batch/s]

Training batches on cuda:  70%|██████▉   | 6772/9708 [00:32<00:13, 212.11batch/s]

Training batches on cuda:  70%|██████▉   | 6794/9708 [00:32<00:13, 210.94batch/s]

Training batches on cuda:  70%|███████   | 6816/9708 [00:32<00:13, 208.59batch/s]

Training batches on cuda:  70%|███████   | 6837/9708 [00:32<00:13, 208.70batch/s]

Training batches on cuda:  71%|███████   | 6859/9708 [00:32<00:13, 209.55batch/s]

Training batches on cuda:  71%|███████   | 6880/9708 [00:32<00:14, 200.46batch/s]

Training batches on cuda:  71%|███████   | 6903/9708 [00:32<00:13, 206.95batch/s]

Training batches on cuda:  71%|███████▏  | 6925/9708 [00:32<00:13, 208.58batch/s]

Training batches on cuda:  72%|███████▏  | 6947/9708 [00:33<00:13, 209.54batch/s]

Training batches on cuda:  72%|███████▏  | 6969/9708 [00:33<00:13, 210.22batch/s]

Training batches on cuda:  72%|███████▏  | 6991/9708 [00:33<00:12, 211.69batch/s]

Training batches on cuda:  72%|███████▏  | 7013/9708 [00:33<00:12, 213.97batch/s]

Training batches on cuda:  72%|███████▏  | 7035/9708 [00:33<00:12, 213.96batch/s]

Training batches on cuda:  73%|███████▎  | 7057/9708 [00:33<00:12, 207.75batch/s]

Training batches on cuda:  73%|███████▎  | 7078/9708 [00:33<00:12, 205.91batch/s]

Training batches on cuda:  73%|███████▎  | 7099/9708 [00:33<00:12, 206.15batch/s]

Training batches on cuda:  73%|███████▎  | 7121/9708 [00:33<00:12, 208.07batch/s]

Training batches on cuda:  74%|███████▎  | 7142/9708 [00:33<00:12, 206.21batch/s]

Training batches on cuda:  74%|███████▍  | 7163/9708 [00:34<00:12, 205.39batch/s]

Training batches on cuda:  74%|███████▍  | 7184/9708 [00:34<00:12, 202.36batch/s]

Training batches on cuda:  74%|███████▍  | 7205/9708 [00:34<00:12, 195.22batch/s]

Training batches on cuda:  74%|███████▍  | 7225/9708 [00:34<00:12, 193.17batch/s]

Training batches on cuda:  75%|███████▍  | 7248/9708 [00:34<00:12, 201.88batch/s]

Training batches on cuda:  75%|███████▍  | 7270/9708 [00:34<00:11, 205.53batch/s]

Training batches on cuda:  75%|███████▌  | 7291/9708 [00:34<00:11, 202.53batch/s]

Training batches on cuda:  75%|███████▌  | 7312/9708 [00:34<00:11, 200.97batch/s]

Training batches on cuda:  76%|███████▌  | 7333/9708 [00:34<00:12, 197.26batch/s]

Training batches on cuda:  76%|███████▌  | 7353/9708 [00:35<00:12, 194.82batch/s]

Training batches on cuda:  76%|███████▌  | 7374/9708 [00:35<00:11, 196.91batch/s]

Training batches on cuda:  76%|███████▌  | 7394/9708 [00:35<00:11, 196.70batch/s]

Training batches on cuda:  76%|███████▋  | 7414/9708 [00:35<00:11, 194.65batch/s]

Training batches on cuda:  77%|███████▋  | 7436/9708 [00:35<00:11, 201.07batch/s]

Training batches on cuda:  77%|███████▋  | 7459/9708 [00:35<00:10, 208.92batch/s]

Training batches on cuda:  77%|███████▋  | 7482/9708 [00:35<00:10, 212.81batch/s]

Training batches on cuda:  77%|███████▋  | 7504/9708 [00:35<00:10, 210.00batch/s]

Training batches on cuda:  78%|███████▊  | 7526/9708 [00:35<00:10, 209.35batch/s]

Training batches on cuda:  78%|███████▊  | 7548/9708 [00:35<00:10, 212.41batch/s]

Training batches on cuda:  78%|███████▊  | 7570/9708 [00:36<00:10, 207.90batch/s]

Training batches on cuda:  78%|███████▊  | 7591/9708 [00:36<00:10, 194.29batch/s]

Training batches on cuda:  78%|███████▊  | 7611/9708 [00:36<00:11, 189.37batch/s]

Training batches on cuda:  79%|███████▊  | 7631/9708 [00:36<00:10, 189.27batch/s]

Training batches on cuda:  79%|███████▉  | 7651/9708 [00:36<00:10, 190.43batch/s]

Training batches on cuda:  79%|███████▉  | 7672/9708 [00:36<00:10, 194.99batch/s]

Training batches on cuda:  79%|███████▉  | 7692/9708 [00:36<00:10, 193.81batch/s]

Training batches on cuda:  79%|███████▉  | 7713/9708 [00:36<00:10, 196.18batch/s]

Training batches on cuda:  80%|███████▉  | 7733/9708 [00:36<00:10, 193.27batch/s]

Training batches on cuda:  80%|███████▉  | 7753/9708 [00:37<00:10, 186.29batch/s]

Training batches on cuda:  80%|████████  | 7772/9708 [00:37<00:10, 182.88batch/s]

Training batches on cuda:  80%|████████  | 7791/9708 [00:37<00:10, 179.73batch/s]

Training batches on cuda:  80%|████████  | 7813/9708 [00:37<00:10, 188.35batch/s]

Training batches on cuda:  81%|████████  | 7836/9708 [00:37<00:09, 198.81batch/s]

Training batches on cuda:  81%|████████  | 7857/9708 [00:37<00:09, 197.57batch/s]

Training batches on cuda:  81%|████████  | 7878/9708 [00:37<00:09, 200.57batch/s]

Training batches on cuda:  81%|████████▏ | 7899/9708 [00:37<00:09, 198.88batch/s]

Training batches on cuda:  82%|████████▏ | 7919/9708 [00:37<00:09, 197.91batch/s]

Training batches on cuda:  82%|████████▏ | 7939/9708 [00:38<00:09, 188.45batch/s]

Training batches on cuda:  82%|████████▏ | 7958/9708 [00:38<00:09, 187.16batch/s]

Training batches on cuda:  82%|████████▏ | 7977/9708 [00:38<00:09, 186.27batch/s]

Training batches on cuda:  82%|████████▏ | 7996/9708 [00:38<00:09, 184.57batch/s]

Training batches on cuda:  83%|████████▎ | 8015/9708 [00:38<00:09, 183.89batch/s]

Training batches on cuda:  83%|████████▎ | 8034/9708 [00:38<00:09, 172.44batch/s]

Training batches on cuda:  83%|████████▎ | 8055/9708 [00:38<00:09, 182.10batch/s]

Training batches on cuda:  83%|████████▎ | 8078/9708 [00:38<00:08, 192.61batch/s]

Training batches on cuda:  83%|████████▎ | 8101/9708 [00:38<00:08, 200.51batch/s]

Training batches on cuda:  84%|████████▎ | 8123/9708 [00:38<00:07, 205.06batch/s]

Training batches on cuda:  84%|████████▍ | 8144/9708 [00:39<00:07, 204.18batch/s]

Training batches on cuda:  84%|████████▍ | 8165/9708 [00:39<00:07, 203.53batch/s]

Training batches on cuda:  84%|████████▍ | 8188/9708 [00:39<00:07, 208.68batch/s]

Training batches on cuda:  85%|████████▍ | 8214/9708 [00:39<00:06, 220.26batch/s]

Training batches on cuda:  85%|████████▍ | 8240/9708 [00:39<00:06, 229.62batch/s]

Training batches on cuda:  85%|████████▌ | 8264/9708 [00:39<00:06, 206.97batch/s]

Training batches on cuda:  85%|████████▌ | 8286/9708 [00:39<00:06, 209.18batch/s]

Training batches on cuda:  86%|████████▌ | 8308/9708 [00:39<00:06, 204.43batch/s]

Training batches on cuda:  86%|████████▌ | 8329/9708 [00:39<00:06, 199.37batch/s]

Training batches on cuda:  86%|████████▌ | 8350/9708 [00:40<00:07, 190.94batch/s]

Training batches on cuda:  86%|████████▌ | 8371/9708 [00:40<00:06, 193.94batch/s]

Training batches on cuda:  86%|████████▋ | 8391/9708 [00:40<00:06, 190.03batch/s]

Training batches on cuda:  87%|████████▋ | 8411/9708 [00:40<00:06, 190.20batch/s]

Training batches on cuda:  87%|████████▋ | 8431/9708 [00:40<00:06, 189.26batch/s]

Training batches on cuda:  87%|████████▋ | 8453/9708 [00:40<00:06, 197.36batch/s]

Training batches on cuda:  87%|████████▋ | 8474/9708 [00:40<00:06, 199.72batch/s]

Training batches on cuda:  88%|████████▊ | 8496/9708 [00:40<00:05, 203.98batch/s]

Training batches on cuda:  88%|████████▊ | 8523/9708 [00:40<00:05, 219.99batch/s]

Training batches on cuda:  88%|████████▊ | 8550/9708 [00:41<00:04, 232.57batch/s]

Training batches on cuda:  88%|████████▊ | 8574/9708 [00:41<00:04, 231.53batch/s]

Training batches on cuda:  89%|████████▊ | 8598/9708 [00:41<00:04, 225.33batch/s]

Training batches on cuda:  89%|████████▉ | 8621/9708 [00:41<00:04, 221.19batch/s]

Training batches on cuda:  89%|████████▉ | 8644/9708 [00:41<00:04, 217.38batch/s]

Training batches on cuda:  89%|████████▉ | 8666/9708 [00:41<00:04, 214.31batch/s]

Training batches on cuda:  89%|████████▉ | 8688/9708 [00:41<00:04, 210.43batch/s]

Training batches on cuda:  90%|████████▉ | 8710/9708 [00:41<00:04, 211.22batch/s]

Training batches on cuda:  90%|████████▉ | 8732/9708 [00:41<00:04, 210.96batch/s]

Training batches on cuda:  90%|█████████ | 8754/9708 [00:41<00:04, 210.65batch/s]

Training batches on cuda:  90%|█████████ | 8776/9708 [00:42<00:04, 211.09batch/s]

Training batches on cuda:  91%|█████████ | 8798/9708 [00:42<00:04, 212.83batch/s]

Training batches on cuda:  91%|█████████ | 8820/9708 [00:42<00:04, 213.78batch/s]

Training batches on cuda:  91%|█████████ | 8842/9708 [00:42<00:04, 214.49batch/s]

Training batches on cuda:  91%|█████████▏| 8864/9708 [00:42<00:04, 210.80batch/s]

Training batches on cuda:  92%|█████████▏| 8886/9708 [00:42<00:03, 207.53batch/s]

Training batches on cuda:  92%|█████████▏| 8907/9708 [00:42<00:03, 204.06batch/s]

Training batches on cuda:  92%|█████████▏| 8928/9708 [00:42<00:04, 194.73batch/s]

Training batches on cuda:  92%|█████████▏| 8948/9708 [00:42<00:03, 192.78batch/s]

Training batches on cuda:  92%|█████████▏| 8968/9708 [00:43<00:03, 193.41batch/s]

Training batches on cuda:  93%|█████████▎| 8988/9708 [00:43<00:03, 186.04batch/s]

Training batches on cuda:  93%|█████████▎| 9009/9708 [00:43<00:03, 190.81batch/s]

Training batches on cuda:  93%|█████████▎| 9029/9708 [00:43<00:03, 191.29batch/s]

Training batches on cuda:  93%|█████████▎| 9052/9708 [00:43<00:03, 201.46batch/s]

Training batches on cuda:  93%|█████████▎| 9073/9708 [00:43<00:03, 196.02batch/s]

Training batches on cuda:  94%|█████████▎| 9093/9708 [00:43<00:03, 192.46batch/s]

Training batches on cuda:  94%|█████████▍| 9113/9708 [00:43<00:03, 189.75batch/s]

Training batches on cuda:  94%|█████████▍| 9133/9708 [00:43<00:03, 188.61batch/s]

Training batches on cuda:  94%|█████████▍| 9152/9708 [00:43<00:02, 186.28batch/s]

Training batches on cuda:  94%|█████████▍| 9171/9708 [00:44<00:02, 185.91batch/s]

Training batches on cuda:  95%|█████████▍| 9192/9708 [00:44<00:02, 190.99batch/s]

Training batches on cuda:  95%|█████████▍| 9212/9708 [00:44<00:02, 192.70batch/s]

Training batches on cuda:  95%|█████████▌| 9233/9708 [00:44<00:02, 196.55batch/s]

Training batches on cuda:  95%|█████████▌| 9253/9708 [00:44<00:02, 197.25batch/s]

Training batches on cuda:  96%|█████████▌| 9274/9708 [00:44<00:02, 200.24batch/s]

Training batches on cuda:  96%|█████████▌| 9296/9708 [00:44<00:02, 205.04batch/s]

Training batches on cuda:  96%|█████████▌| 9318/9708 [00:44<00:01, 207.80batch/s]

Training batches on cuda:  96%|█████████▌| 9340/9708 [00:44<00:01, 211.09batch/s]

Training batches on cuda:  96%|█████████▋| 9362/9708 [00:45<00:01, 213.68batch/s]

Training batches on cuda:  97%|█████████▋| 9384/9708 [00:45<00:01, 211.75batch/s]

Training batches on cuda:  97%|█████████▋| 9406/9708 [00:45<00:01, 205.93batch/s]

Training batches on cuda:  97%|█████████▋| 9427/9708 [00:45<00:01, 201.63batch/s]

Training batches on cuda:  97%|█████████▋| 9448/9708 [00:45<00:01, 196.32batch/s]

Training batches on cuda:  98%|█████████▊| 9469/9708 [00:45<00:01, 198.62batch/s]

Training batches on cuda:  98%|█████████▊| 9491/9708 [00:45<00:01, 202.90batch/s]

Training batches on cuda:  98%|█████████▊| 9512/9708 [00:45<00:00, 203.51batch/s]

Training batches on cuda:  98%|█████████▊| 9534/9708 [00:45<00:00, 205.51batch/s]

Training batches on cuda:  98%|█████████▊| 9556/9708 [00:45<00:00, 207.86batch/s]

Training batches on cuda:  99%|█████████▊| 9577/9708 [00:46<00:00, 208.38batch/s]

Training batches on cuda:  99%|█████████▉| 9599/9708 [00:46<00:00, 210.72batch/s]

Training batches on cuda:  99%|█████████▉| 9621/9708 [00:46<00:00, 210.29batch/s]

Training batches on cuda:  99%|█████████▉| 9643/9708 [00:46<00:00, 206.85batch/s]

Training batches on cuda: 100%|█████████▉| 9664/9708 [00:46<00:00, 194.83batch/s]

Training batches on cuda: 100%|█████████▉| 9684/9708 [00:46<00:00, 191.10batch/s]

Training batches on cuda: 100%|█████████▉| 9706/9708 [00:46<00:00, 198.50batch/s]

                                                                                 
Training epochs on cuda:  50%|█████     | 5/10 [04:40<03:52, 46.57s/epoch, loss=0.0416, prev_loss=0.0416]
Training epochs on cuda:  60%|██████    | 6/10 [04:40<03:06, 46.70s/epoch, loss=0.0416, prev_loss=0.0416]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 15/9708 [00:00<01:04, 149.53batch/s]

Training batches on cuda:   0%|          | 37/9708 [00:00<00:58, 164.63batch/s]

Training batches on cuda:   1%|          | 56/9708 [00:00<00:57, 169.24batch/s]

Training batches on cuda:   1%|          | 78/9708 [00:00<00:53, 180.40batch/s]

Training batches on cuda:   1%|          | 98/9708 [00:00<00:52, 184.57batch/s]

Training batches on cuda:   1%|          | 115/9708 [00:00<00:54, 176.55batch/s]

Training batches on cuda:   1%|▏         | 136/9708 [00:00<00:52, 184.06batch/s]

Training batches on cuda:   2%|▏         | 154/9708 [00:00<00:54, 176.87batch/s]

Training batches on cuda:   2%|▏         | 177/9708 [00:00<00:50, 189.45batch/s]

Training batches on cuda:   2%|▏         | 199/9708 [00:01<00:48, 196.76batch/s]

Training batches on cuda:   2%|▏         | 220/9708 [00:01<00:47, 199.25batch/s]

Training batches on cuda:   2%|▏         | 242/9708 [00:01<00:46, 204.79batch/s]

Training batches on cuda:   3%|▎         | 264/9708 [00:01<00:45, 208.92batch/s]

Training batches on cuda:   3%|▎         | 285/9708 [00:01<00:46, 202.91batch/s]

Training batches on cuda:   3%|▎         | 306/9708 [00:01<00:47, 197.48batch/s]

Training batches on cuda:   3%|▎         | 326/9708 [00:01<00:47, 198.07batch/s]

Training batches on cuda:   4%|▎         | 347/9708 [00:01<00:46, 201.29batch/s]

Training batches on cuda:   4%|▍         | 370/9708 [00:01<00:45, 207.17batch/s]

Training batches on cuda:   4%|▍         | 391/9708 [00:01<00:45, 205.29batch/s]

Training batches on cuda:   4%|▍         | 414/9708 [00:02<00:44, 209.87batch/s]

Training batches on cuda:   4%|▍         | 436/9708 [00:02<00:45, 205.93batch/s]

Training batches on cuda:   5%|▍         | 461/9708 [00:02<00:42, 215.34batch/s]

Training batches on cuda:   5%|▍         | 485/9708 [00:02<00:41, 222.01batch/s]

Training batches on cuda:   5%|▌         | 510/9708 [00:02<00:40, 227.61batch/s]

Training batches on cuda:   6%|▌         | 535/9708 [00:02<00:39, 233.48batch/s]

Training batches on cuda:   6%|▌         | 559/9708 [00:02<00:39, 230.27batch/s]

Training batches on cuda:   6%|▌         | 583/9708 [00:02<00:39, 228.44batch/s]

Training batches on cuda:   6%|▌         | 606/9708 [00:02<00:40, 223.09batch/s]

Training batches on cuda:   6%|▋         | 629/9708 [00:03<00:40, 224.94batch/s]

Training batches on cuda:   7%|▋         | 652/9708 [00:03<00:42, 212.74batch/s]

Training batches on cuda:   7%|▋         | 674/9708 [00:03<00:43, 208.91batch/s]

Training batches on cuda:   7%|▋         | 696/9708 [00:03<00:42, 210.30batch/s]

Training batches on cuda:   7%|▋         | 718/9708 [00:03<00:42, 209.35batch/s]

Training batches on cuda:   8%|▊         | 740/9708 [00:03<00:43, 207.71batch/s]

Training batches on cuda:   8%|▊         | 761/9708 [00:03<00:43, 207.32batch/s]

Training batches on cuda:   8%|▊         | 783/9708 [00:03<00:42, 210.00batch/s]

Training batches on cuda:   8%|▊         | 805/9708 [00:03<00:42, 208.41batch/s]

Training batches on cuda:   9%|▊         | 826/9708 [00:03<00:44, 199.10batch/s]

Training batches on cuda:   9%|▊         | 847/9708 [00:04<00:44, 198.19batch/s]

Training batches on cuda:   9%|▉         | 868/9708 [00:04<00:43, 200.93batch/s]

Training batches on cuda:   9%|▉         | 889/9708 [00:04<00:43, 203.10batch/s]

Training batches on cuda:   9%|▉         | 911/9708 [00:04<00:42, 205.67batch/s]

Training batches on cuda:  10%|▉         | 933/9708 [00:04<00:42, 207.20batch/s]

Training batches on cuda:  10%|▉         | 954/9708 [00:04<00:42, 204.88batch/s]

Training batches on cuda:  10%|█         | 975/9708 [00:04<00:42, 206.19batch/s]

Training batches on cuda:  10%|█         | 996/9708 [00:04<00:42, 204.58batch/s]

Training batches on cuda:  10%|█         | 1018/9708 [00:04<00:41, 207.26batch/s]

Training batches on cuda:  11%|█         | 1040/9708 [00:05<00:41, 208.85batch/s]

Training batches on cuda:  11%|█         | 1062/9708 [00:05<00:41, 210.18batch/s]

Training batches on cuda:  11%|█         | 1084/9708 [00:05<00:40, 210.69batch/s]

Training batches on cuda:  11%|█▏        | 1106/9708 [00:05<00:40, 210.20batch/s]

Training batches on cuda:  12%|█▏        | 1128/9708 [00:05<00:41, 206.48batch/s]

Training batches on cuda:  12%|█▏        | 1149/9708 [00:05<00:43, 197.53batch/s]

Training batches on cuda:  12%|█▏        | 1172/9708 [00:05<00:41, 204.29batch/s]

Training batches on cuda:  12%|█▏        | 1193/9708 [00:05<00:41, 202.74batch/s]

Training batches on cuda:  13%|█▎        | 1216/9708 [00:05<00:40, 209.80batch/s]

Training batches on cuda:  13%|█▎        | 1239/9708 [00:05<00:39, 215.22batch/s]

Training batches on cuda:  13%|█▎        | 1262/9708 [00:06<00:38, 218.46batch/s]

Training batches on cuda:  13%|█▎        | 1284/9708 [00:06<00:38, 218.77batch/s]

Training batches on cuda:  13%|█▎        | 1307/9708 [00:06<00:38, 219.85batch/s]

Training batches on cuda:  14%|█▎        | 1330/9708 [00:06<00:37, 220.50batch/s]

Training batches on cuda:  14%|█▍        | 1353/9708 [00:06<00:37, 222.49batch/s]

Training batches on cuda:  14%|█▍        | 1376/9708 [00:06<00:38, 214.80batch/s]

Training batches on cuda:  14%|█▍        | 1398/9708 [00:06<00:39, 212.73batch/s]

Training batches on cuda:  15%|█▍        | 1421/9708 [00:06<00:38, 215.14batch/s]

Training batches on cuda:  15%|█▍        | 1443/9708 [00:06<00:39, 208.15batch/s]

Training batches on cuda:  15%|█▌        | 1464/9708 [00:07<00:40, 205.52batch/s]

Training batches on cuda:  15%|█▌        | 1485/9708 [00:07<00:41, 200.43batch/s]

Training batches on cuda:  16%|█▌        | 1506/9708 [00:07<00:40, 200.15batch/s]

Training batches on cuda:  16%|█▌        | 1527/9708 [00:07<00:40, 202.38batch/s]

Training batches on cuda:  16%|█▌        | 1551/9708 [00:07<00:38, 209.83batch/s]

Training batches on cuda:  16%|█▌        | 1573/9708 [00:07<00:39, 206.77batch/s]

Training batches on cuda:  16%|█▋        | 1595/9708 [00:07<00:38, 208.73batch/s]

Training batches on cuda:  17%|█▋        | 1616/9708 [00:07<00:38, 208.83batch/s]

Training batches on cuda:  17%|█▋        | 1637/9708 [00:07<00:39, 203.39batch/s]

Training batches on cuda:  17%|█▋        | 1659/9708 [00:07<00:39, 206.28batch/s]

Training batches on cuda:  17%|█▋        | 1680/9708 [00:08<00:38, 207.19batch/s]

Training batches on cuda:  18%|█▊        | 1701/9708 [00:08<00:39, 203.63batch/s]

Training batches on cuda:  18%|█▊        | 1723/9708 [00:08<00:38, 206.74batch/s]

Training batches on cuda:  18%|█▊        | 1744/9708 [00:08<00:39, 203.11batch/s]

Training batches on cuda:  18%|█▊        | 1768/9708 [00:08<00:37, 211.98batch/s]

Training batches on cuda:  18%|█▊        | 1790/9708 [00:08<00:36, 214.09batch/s]

Training batches on cuda:  19%|█▊        | 1813/9708 [00:08<00:36, 217.93batch/s]

Training batches on cuda:  19%|█▉        | 1835/9708 [00:08<00:37, 210.39batch/s]

Training batches on cuda:  19%|█▉        | 1857/9708 [00:08<00:37, 208.31batch/s]

Training batches on cuda:  19%|█▉        | 1879/9708 [00:09<00:37, 211.42batch/s]

Training batches on cuda:  20%|█▉        | 1902/9708 [00:09<00:36, 215.38batch/s]

Training batches on cuda:  20%|█▉        | 1924/9708 [00:09<00:36, 215.61batch/s]

Training batches on cuda:  20%|██        | 1948/9708 [00:09<00:35, 220.40batch/s]

Training batches on cuda:  20%|██        | 1971/9708 [00:09<00:36, 212.67batch/s]

Training batches on cuda:  21%|██        | 1993/9708 [00:09<00:37, 208.51batch/s]

Training batches on cuda:  21%|██        | 2014/9708 [00:09<00:37, 205.97batch/s]

Training batches on cuda:  21%|██        | 2036/9708 [00:09<00:36, 207.68batch/s]

Training batches on cuda:  21%|██        | 2058/9708 [00:09<00:36, 208.80batch/s]

Training batches on cuda:  21%|██▏       | 2080/9708 [00:09<00:36, 210.87batch/s]

Training batches on cuda:  22%|██▏       | 2102/9708 [00:10<00:36, 209.24batch/s]

Training batches on cuda:  22%|██▏       | 2123/9708 [00:10<00:36, 209.20batch/s]

Training batches on cuda:  22%|██▏       | 2146/9708 [00:10<00:35, 213.28batch/s]

Training batches on cuda:  22%|██▏       | 2168/9708 [00:10<00:35, 214.43batch/s]

Training batches on cuda:  23%|██▎       | 2190/9708 [00:10<00:35, 210.36batch/s]

Training batches on cuda:  23%|██▎       | 2212/9708 [00:10<00:35, 212.90batch/s]

Training batches on cuda:  23%|██▎       | 2234/9708 [00:10<00:35, 209.95batch/s]

Training batches on cuda:  23%|██▎       | 2256/9708 [00:10<00:35, 207.50batch/s]

Training batches on cuda:  23%|██▎       | 2277/9708 [00:10<00:36, 203.33batch/s]

Training batches on cuda:  24%|██▎       | 2299/9708 [00:11<00:36, 205.49batch/s]

Training batches on cuda:  24%|██▍       | 2320/9708 [00:11<00:35, 205.39batch/s]

Training batches on cuda:  24%|██▍       | 2341/9708 [00:11<00:35, 206.33batch/s]

Training batches on cuda:  24%|██▍       | 2362/9708 [00:11<00:38, 192.05batch/s]

Training batches on cuda:  25%|██▍       | 2382/9708 [00:11<00:39, 185.55batch/s]

Training batches on cuda:  25%|██▍       | 2403/9708 [00:11<00:38, 189.82batch/s]

Training batches on cuda:  25%|██▍       | 2424/9708 [00:11<00:37, 195.10batch/s]

Training batches on cuda:  25%|██▌       | 2447/9708 [00:11<00:35, 203.25batch/s]

Training batches on cuda:  25%|██▌       | 2470/9708 [00:11<00:34, 208.98batch/s]

Training batches on cuda:  26%|██▌       | 2493/9708 [00:11<00:33, 214.60batch/s]

Training batches on cuda:  26%|██▌       | 2515/9708 [00:12<00:33, 214.72batch/s]

Training batches on cuda:  26%|██▌       | 2538/9708 [00:12<00:32, 218.44batch/s]

Training batches on cuda:  26%|██▋       | 2561/9708 [00:12<00:32, 221.68batch/s]

Training batches on cuda:  27%|██▋       | 2585/9708 [00:12<00:31, 224.47batch/s]

Training batches on cuda:  27%|██▋       | 2609/9708 [00:12<00:31, 226.18batch/s]

Training batches on cuda:  27%|██▋       | 2633/9708 [00:12<00:31, 227.26batch/s]

Training batches on cuda:  27%|██▋       | 2656/9708 [00:12<00:30, 227.68batch/s]

Training batches on cuda:  28%|██▊       | 2680/9708 [00:12<00:30, 228.74batch/s]

Training batches on cuda:  28%|██▊       | 2704/9708 [00:12<00:30, 229.15batch/s]

Training batches on cuda:  28%|██▊       | 2728/9708 [00:13<00:30, 230.00batch/s]

Training batches on cuda:  28%|██▊       | 2752/9708 [00:13<00:30, 230.50batch/s]

Training batches on cuda:  29%|██▊       | 2776/9708 [00:13<00:29, 231.45batch/s]

Training batches on cuda:  29%|██▉       | 2800/9708 [00:13<00:29, 230.79batch/s]

Training batches on cuda:  29%|██▉       | 2824/9708 [00:13<00:29, 232.32batch/s]

Training batches on cuda:  29%|██▉       | 2848/9708 [00:13<00:30, 224.44batch/s]

Training batches on cuda:  30%|██▉       | 2871/9708 [00:13<00:31, 220.46batch/s]

Training batches on cuda:  30%|██▉       | 2894/9708 [00:13<00:31, 216.98batch/s]

Training batches on cuda:  30%|███       | 2916/9708 [00:13<00:31, 213.56batch/s]

Training batches on cuda:  30%|███       | 2938/9708 [00:13<00:31, 212.40batch/s]

Training batches on cuda:  30%|███       | 2960/9708 [00:14<00:32, 210.57batch/s]

Training batches on cuda:  31%|███       | 2982/9708 [00:14<00:31, 210.19batch/s]

Training batches on cuda:  31%|███       | 3004/9708 [00:14<00:31, 209.90batch/s]

Training batches on cuda:  31%|███       | 3026/9708 [00:14<00:31, 209.28batch/s]

Training batches on cuda:  31%|███▏      | 3047/9708 [00:14<00:31, 209.07batch/s]

Training batches on cuda:  32%|███▏      | 3068/9708 [00:14<00:31, 208.61batch/s]

Training batches on cuda:  32%|███▏      | 3090/9708 [00:14<00:31, 211.61batch/s]

Training batches on cuda:  32%|███▏      | 3113/9708 [00:14<00:30, 216.64batch/s]

Training batches on cuda:  32%|███▏      | 3136/9708 [00:14<00:29, 220.41batch/s]

Training batches on cuda:  33%|███▎      | 3159/9708 [00:14<00:29, 221.92batch/s]

Training batches on cuda:  33%|███▎      | 3182/9708 [00:15<00:29, 223.67batch/s]

Training batches on cuda:  33%|███▎      | 3205/9708 [00:15<00:28, 224.70batch/s]

Training batches on cuda:  33%|███▎      | 3228/9708 [00:15<00:28, 224.70batch/s]

Training batches on cuda:  33%|███▎      | 3251/9708 [00:15<00:28, 226.16batch/s]

Training batches on cuda:  34%|███▎      | 3275/9708 [00:15<00:28, 227.41batch/s]

Training batches on cuda:  34%|███▍      | 3298/9708 [00:15<00:28, 222.98batch/s]

Training batches on cuda:  34%|███▍      | 3321/9708 [00:15<00:29, 218.22batch/s]

Training batches on cuda:  34%|███▍      | 3343/9708 [00:15<00:29, 217.87batch/s]

Training batches on cuda:  35%|███▍      | 3365/9708 [00:15<00:32, 195.94batch/s]

Training batches on cuda:  35%|███▍      | 3388/9708 [00:16<00:31, 203.70batch/s]

Training batches on cuda:  35%|███▌      | 3412/9708 [00:16<00:29, 212.90batch/s]

Training batches on cuda:  35%|███▌      | 3435/9708 [00:16<00:29, 215.76batch/s]

Training batches on cuda:  36%|███▌      | 3458/9708 [00:16<00:28, 218.40batch/s]

Training batches on cuda:  36%|███▌      | 3481/9708 [00:16<00:28, 220.04batch/s]

Training batches on cuda:  36%|███▌      | 3504/9708 [00:16<00:28, 220.88batch/s]

Training batches on cuda:  36%|███▋      | 3529/9708 [00:16<00:27, 227.17batch/s]

Training batches on cuda:  37%|███▋      | 3552/9708 [00:16<00:27, 226.68batch/s]

Training batches on cuda:  37%|███▋      | 3576/9708 [00:16<00:26, 230.15batch/s]

Training batches on cuda:  37%|███▋      | 3600/9708 [00:16<00:27, 223.47batch/s]

Training batches on cuda:  37%|███▋      | 3623/9708 [00:17<00:27, 223.37batch/s]

Training batches on cuda:  38%|███▊      | 3646/9708 [00:17<00:28, 216.40batch/s]

Training batches on cuda:  38%|███▊      | 3668/9708 [00:17<00:28, 214.15batch/s]

Training batches on cuda:  38%|███▊      | 3690/9708 [00:17<00:28, 207.72batch/s]

Training batches on cuda:  38%|███▊      | 3712/9708 [00:17<00:28, 209.19batch/s]

Training batches on cuda:  38%|███▊      | 3733/9708 [00:17<00:29, 199.61batch/s]

Training batches on cuda:  39%|███▊      | 3755/9708 [00:17<00:29, 202.88batch/s]

Training batches on cuda:  39%|███▉      | 3778/9708 [00:17<00:28, 208.00batch/s]

Training batches on cuda:  39%|███▉      | 3801/9708 [00:17<00:27, 213.75batch/s]

Training batches on cuda:  39%|███▉      | 3825/9708 [00:18<00:26, 218.82batch/s]

Training batches on cuda:  40%|███▉      | 3849/9708 [00:18<00:26, 222.28batch/s]

Training batches on cuda:  40%|███▉      | 3872/9708 [00:18<00:25, 224.53batch/s]

Training batches on cuda:  40%|████      | 3895/9708 [00:18<00:25, 224.55batch/s]

Training batches on cuda:  40%|████      | 3919/9708 [00:18<00:25, 226.76batch/s]

Training batches on cuda:  41%|████      | 3942/9708 [00:18<00:25, 227.09batch/s]

Training batches on cuda:  41%|████      | 3965/9708 [00:18<00:25, 227.19batch/s]

Training batches on cuda:  41%|████      | 3989/9708 [00:18<00:25, 228.35batch/s]

Training batches on cuda:  41%|████▏     | 4012/9708 [00:18<00:24, 228.57batch/s]

Training batches on cuda:  42%|████▏     | 4035/9708 [00:18<00:24, 228.87batch/s]

Training batches on cuda:  42%|████▏     | 4058/9708 [00:19<00:24, 228.75batch/s]

Training batches on cuda:  42%|████▏     | 4082/9708 [00:19<00:24, 229.21batch/s]

Training batches on cuda:  42%|████▏     | 4105/9708 [00:19<00:24, 228.92batch/s]

Training batches on cuda:  43%|████▎     | 4129/9708 [00:19<00:24, 229.37batch/s]

Training batches on cuda:  43%|████▎     | 4153/9708 [00:19<00:24, 229.85batch/s]

Training batches on cuda:  43%|████▎     | 4176/9708 [00:19<00:24, 223.14batch/s]

Training batches on cuda:  43%|████▎     | 4199/9708 [00:19<00:24, 222.80batch/s]

Training batches on cuda:  43%|████▎     | 4222/9708 [00:19<00:24, 222.03batch/s]

Training batches on cuda:  44%|████▎     | 4245/9708 [00:19<00:25, 211.67batch/s]

Training batches on cuda:  44%|████▍     | 4267/9708 [00:20<00:26, 202.10batch/s]

Training batches on cuda:  44%|████▍     | 4288/9708 [00:20<00:26, 203.67batch/s]

Training batches on cuda:  44%|████▍     | 4309/9708 [00:20<00:28, 192.54batch/s]

Training batches on cuda:  45%|████▍     | 4331/9708 [00:20<00:27, 198.61batch/s]

Training batches on cuda:  45%|████▍     | 4352/9708 [00:20<00:26, 201.58batch/s]

Training batches on cuda:  45%|████▌     | 4374/9708 [00:20<00:26, 204.33batch/s]

Training batches on cuda:  45%|████▌     | 4396/9708 [00:20<00:25, 205.90batch/s]

Training batches on cuda:  45%|████▌     | 4417/9708 [00:20<00:25, 206.15batch/s]

Training batches on cuda:  46%|████▌     | 4441/9708 [00:20<00:24, 213.47batch/s]

Training batches on cuda:  46%|████▌     | 4463/9708 [00:21<00:25, 209.72batch/s]

Training batches on cuda:  46%|████▌     | 4489/9708 [00:21<00:23, 221.85batch/s]

Training batches on cuda:  46%|████▋     | 4514/9708 [00:21<00:22, 229.02batch/s]

Training batches on cuda:  47%|████▋     | 4539/9708 [00:21<00:22, 234.20batch/s]

Training batches on cuda:  47%|████▋     | 4564/9708 [00:21<00:21, 238.24batch/s]

Training batches on cuda:  47%|████▋     | 4589/9708 [00:21<00:21, 241.23batch/s]

Training batches on cuda:  48%|████▊     | 4615/9708 [00:21<00:20, 243.91batch/s]

Training batches on cuda:  48%|████▊     | 4640/9708 [00:21<00:20, 242.35batch/s]

Training batches on cuda:  48%|████▊     | 4666/9708 [00:21<00:20, 246.38batch/s]

Training batches on cuda:  48%|████▊     | 4691/9708 [00:21<00:20, 245.11batch/s]

Training batches on cuda:  49%|████▊     | 4716/9708 [00:22<00:20, 246.00batch/s]

Training batches on cuda:  49%|████▉     | 4741/9708 [00:22<00:20, 245.71batch/s]

Training batches on cuda:  49%|████▉     | 4766/9708 [00:22<00:20, 244.38batch/s]

Training batches on cuda:  49%|████▉     | 4791/9708 [00:22<00:20, 238.30batch/s]

Training batches on cuda:  50%|████▉     | 4815/9708 [00:22<00:20, 235.29batch/s]

Training batches on cuda:  50%|████▉     | 4839/9708 [00:22<00:21, 231.77batch/s]

Training batches on cuda:  50%|█████     | 4863/9708 [00:22<00:21, 229.36batch/s]

Training batches on cuda:  50%|█████     | 4886/9708 [00:22<00:21, 228.64batch/s]

Training batches on cuda:  51%|█████     | 4910/9708 [00:22<00:20, 230.17batch/s]

Training batches on cuda:  51%|█████     | 4935/9708 [00:22<00:20, 235.29batch/s]

Training batches on cuda:  51%|█████     | 4961/9708 [00:23<00:19, 240.35batch/s]

Training batches on cuda:  51%|█████▏    | 4986/9708 [00:23<00:19, 238.47batch/s]

Training batches on cuda:  52%|█████▏    | 5010/9708 [00:23<00:19, 236.42batch/s]

Training batches on cuda:  52%|█████▏    | 5034/9708 [00:23<00:19, 234.72batch/s]

Training batches on cuda:  52%|█████▏    | 5059/9708 [00:23<00:19, 237.09batch/s]

Training batches on cuda:  52%|█████▏    | 5085/9708 [00:23<00:19, 240.80batch/s]

Training batches on cuda:  53%|█████▎    | 5110/9708 [00:23<00:19, 239.37batch/s]

Training batches on cuda:  53%|█████▎    | 5136/9708 [00:23<00:18, 244.37batch/s]

Training batches on cuda:  53%|█████▎    | 5161/9708 [00:23<00:18, 242.29batch/s]

Training batches on cuda:  53%|█████▎    | 5187/9708 [00:24<00:18, 246.49batch/s]

Training batches on cuda:  54%|█████▎    | 5212/9708 [00:24<00:18, 244.72batch/s]

Training batches on cuda:  54%|█████▍    | 5238/9708 [00:24<00:17, 248.69batch/s]

Training batches on cuda:  54%|█████▍    | 5263/9708 [00:24<00:18, 242.64batch/s]

Training batches on cuda:  54%|█████▍    | 5288/9708 [00:24<00:18, 239.00batch/s]

Training batches on cuda:  55%|█████▍    | 5312/9708 [00:24<00:18, 233.55batch/s]

Training batches on cuda:  55%|█████▍    | 5336/9708 [00:24<00:18, 232.42batch/s]

Training batches on cuda:  55%|█████▌    | 5360/9708 [00:24<00:18, 233.94batch/s]

Training batches on cuda:  55%|█████▌    | 5384/9708 [00:24<00:19, 224.92batch/s]

Training batches on cuda:  56%|█████▌    | 5408/9708 [00:24<00:18, 227.26batch/s]

Training batches on cuda:  56%|█████▌    | 5433/9708 [00:25<00:18, 231.36batch/s]

Training batches on cuda:  56%|█████▌    | 5459/9708 [00:25<00:17, 237.48batch/s]

Training batches on cuda:  56%|█████▋    | 5483/9708 [00:25<00:17, 237.31batch/s]

Training batches on cuda:  57%|█████▋    | 5509/9708 [00:25<00:17, 242.68batch/s]

Training batches on cuda:  57%|█████▋    | 5534/9708 [00:25<00:18, 228.12batch/s]

Training batches on cuda:  57%|█████▋    | 5558/9708 [00:25<00:18, 224.94batch/s]

Training batches on cuda:  57%|█████▋    | 5581/9708 [00:25<00:18, 222.12batch/s]

Training batches on cuda:  58%|█████▊    | 5604/9708 [00:25<00:18, 219.46batch/s]

Training batches on cuda:  58%|█████▊    | 5627/9708 [00:25<00:18, 217.23batch/s]

Training batches on cuda:  58%|█████▊    | 5650/9708 [00:26<00:18, 219.39batch/s]

Training batches on cuda:  58%|█████▊    | 5672/9708 [00:26<00:18, 217.03batch/s]

Training batches on cuda:  59%|█████▊    | 5694/9708 [00:26<00:19, 210.08batch/s]

Training batches on cuda:  59%|█████▉    | 5716/9708 [00:26<00:18, 210.88batch/s]

Training batches on cuda:  59%|█████▉    | 5738/9708 [00:26<00:19, 205.89batch/s]

Training batches on cuda:  59%|█████▉    | 5759/9708 [00:26<00:19, 205.99batch/s]

Training batches on cuda:  60%|█████▉    | 5780/9708 [00:26<00:19, 206.67batch/s]

Training batches on cuda:  60%|█████▉    | 5801/9708 [00:26<00:18, 206.87batch/s]

Training batches on cuda:  60%|█████▉    | 5822/9708 [00:26<00:18, 206.81batch/s]

Training batches on cuda:  60%|██████    | 5843/9708 [00:26<00:19, 202.63batch/s]

Training batches on cuda:  60%|██████    | 5864/9708 [00:27<00:18, 202.38batch/s]

Training batches on cuda:  61%|██████    | 5885/9708 [00:27<00:18, 204.19batch/s]

Training batches on cuda:  61%|██████    | 5907/9708 [00:27<00:18, 206.29batch/s]

Training batches on cuda:  61%|██████    | 5929/9708 [00:27<00:18, 208.48batch/s]

Training batches on cuda:  61%|██████▏   | 5950/9708 [00:27<00:18, 199.99batch/s]

Training batches on cuda:  62%|██████▏   | 5971/9708 [00:27<00:18, 202.18batch/s]

Training batches on cuda:  62%|██████▏   | 5992/9708 [00:27<00:18, 200.49batch/s]

Training batches on cuda:  62%|██████▏   | 6016/9708 [00:27<00:17, 209.87batch/s]

Training batches on cuda:  62%|██████▏   | 6039/9708 [00:27<00:17, 213.33batch/s]

Training batches on cuda:  62%|██████▏   | 6062/9708 [00:28<00:16, 216.00batch/s]

Training batches on cuda:  63%|██████▎   | 6084/9708 [00:28<00:17, 213.16batch/s]

Training batches on cuda:  63%|██████▎   | 6107/9708 [00:28<00:16, 216.22batch/s]

Training batches on cuda:  63%|██████▎   | 6130/9708 [00:28<00:16, 218.38batch/s]

Training batches on cuda:  63%|██████▎   | 6152/9708 [00:28<00:16, 212.20batch/s]

Training batches on cuda:  64%|██████▎   | 6174/9708 [00:28<00:16, 211.84batch/s]

Training batches on cuda:  64%|██████▍   | 6196/9708 [00:28<00:16, 209.15batch/s]

Training batches on cuda:  64%|██████▍   | 6218/9708 [00:28<00:16, 211.39batch/s]

Training batches on cuda:  64%|██████▍   | 6241/9708 [00:28<00:16, 215.82batch/s]

Training batches on cuda:  65%|██████▍   | 6264/9708 [00:28<00:15, 217.55batch/s]

Training batches on cuda:  65%|██████▍   | 6286/9708 [00:29<00:16, 207.00batch/s]

Training batches on cuda:  65%|██████▍   | 6309/9708 [00:29<00:16, 210.32batch/s]

Training batches on cuda:  65%|██████▌   | 6331/9708 [00:29<00:16, 203.17batch/s]

Training batches on cuda:  65%|██████▌   | 6352/9708 [00:29<00:16, 200.97batch/s]

Training batches on cuda:  66%|██████▌   | 6373/9708 [00:29<00:16, 203.24batch/s]

Training batches on cuda:  66%|██████▌   | 6395/9708 [00:29<00:16, 206.27batch/s]

Training batches on cuda:  66%|██████▌   | 6416/9708 [00:29<00:15, 205.95batch/s]

Training batches on cuda:  66%|██████▋   | 6437/9708 [00:29<00:15, 206.64batch/s]

Training batches on cuda:  67%|██████▋   | 6459/9708 [00:29<00:15, 208.94batch/s]

Training batches on cuda:  67%|██████▋   | 6482/9708 [00:30<00:15, 214.27batch/s]

Training batches on cuda:  67%|██████▋   | 6505/9708 [00:30<00:14, 218.20batch/s]

Training batches on cuda:  67%|██████▋   | 6528/9708 [00:30<00:14, 219.85batch/s]

Training batches on cuda:  67%|██████▋   | 6551/9708 [00:30<00:14, 211.81batch/s]

Training batches on cuda:  68%|██████▊   | 6573/9708 [00:30<00:14, 211.82batch/s]

Training batches on cuda:  68%|██████▊   | 6595/9708 [00:30<00:14, 211.29batch/s]

Training batches on cuda:  68%|██████▊   | 6617/9708 [00:30<00:15, 201.05batch/s]

Training batches on cuda:  68%|██████▊   | 6638/9708 [00:30<00:15, 197.82batch/s]

Training batches on cuda:  69%|██████▊   | 6659/9708 [00:30<00:15, 199.03batch/s]

Training batches on cuda:  69%|██████▉   | 6679/9708 [00:31<00:16, 185.83batch/s]

Training batches on cuda:  69%|██████▉   | 6699/9708 [00:31<00:15, 189.82batch/s]

Training batches on cuda:  69%|██████▉   | 6720/9708 [00:31<00:15, 195.04batch/s]

Training batches on cuda:  69%|██████▉   | 6741/9708 [00:31<00:15, 197.00batch/s]

Training batches on cuda:  70%|██████▉   | 6761/9708 [00:31<00:15, 195.64batch/s]

Training batches on cuda:  70%|██████▉   | 6784/9708 [00:31<00:14, 202.69batch/s]

Training batches on cuda:  70%|███████   | 6805/9708 [00:31<00:14, 203.61batch/s]

Training batches on cuda:  70%|███████   | 6828/9708 [00:31<00:13, 209.33batch/s]

Training batches on cuda:  71%|███████   | 6850/9708 [00:31<00:13, 209.38batch/s]

Training batches on cuda:  71%|███████   | 6872/9708 [00:31<00:13, 208.89batch/s]

Training batches on cuda:  71%|███████   | 6893/9708 [00:32<00:13, 206.96batch/s]

Training batches on cuda:  71%|███████   | 6914/9708 [00:32<00:13, 206.75batch/s]

Training batches on cuda:  71%|███████▏  | 6936/9708 [00:32<00:13, 209.45batch/s]

Training batches on cuda:  72%|███████▏  | 6957/9708 [00:32<00:13, 202.71batch/s]

Training batches on cuda:  72%|███████▏  | 6979/9708 [00:32<00:13, 207.23batch/s]

Training batches on cuda:  72%|███████▏  | 7002/9708 [00:32<00:12, 212.74batch/s]

Training batches on cuda:  72%|███████▏  | 7025/9708 [00:32<00:12, 217.33batch/s]

Training batches on cuda:  73%|███████▎  | 7047/9708 [00:32<00:12, 216.41batch/s]

Training batches on cuda:  73%|███████▎  | 7071/9708 [00:32<00:11, 221.82batch/s]

Training batches on cuda:  73%|███████▎  | 7095/9708 [00:32<00:11, 224.32batch/s]

Training batches on cuda:  73%|███████▎  | 7118/9708 [00:33<00:11, 216.59batch/s]

Training batches on cuda:  74%|███████▎  | 7140/9708 [00:33<00:12, 207.46batch/s]

Training batches on cuda:  74%|███████▍  | 7161/9708 [00:33<00:12, 202.13batch/s]

Training batches on cuda:  74%|███████▍  | 7182/9708 [00:33<00:12, 195.47batch/s]

Training batches on cuda:  74%|███████▍  | 7202/9708 [00:33<00:12, 193.31batch/s]

Training batches on cuda:  74%|███████▍  | 7223/9708 [00:33<00:12, 196.11batch/s]

Training batches on cuda:  75%|███████▍  | 7243/9708 [00:33<00:12, 191.64batch/s]

Training batches on cuda:  75%|███████▍  | 7266/9708 [00:33<00:12, 201.27batch/s]

Training batches on cuda:  75%|███████▌  | 7289/9708 [00:33<00:11, 207.40batch/s]

Training batches on cuda:  75%|███████▌  | 7312/9708 [00:34<00:11, 213.67batch/s]

Training batches on cuda:  76%|███████▌  | 7334/9708 [00:34<00:11, 207.29batch/s]

Training batches on cuda:  76%|███████▌  | 7355/9708 [00:34<00:11, 207.39batch/s]

Training batches on cuda:  76%|███████▌  | 7376/9708 [00:34<00:11, 206.95batch/s]

Training batches on cuda:  76%|███████▌  | 7398/9708 [00:34<00:11, 209.18batch/s]

Training batches on cuda:  76%|███████▋  | 7421/9708 [00:34<00:10, 214.12batch/s]

Training batches on cuda:  77%|███████▋  | 7443/9708 [00:34<00:10, 207.68batch/s]

Training batches on cuda:  77%|███████▋  | 7464/9708 [00:34<00:11, 195.77batch/s]

Training batches on cuda:  77%|███████▋  | 7485/9708 [00:34<00:11, 198.78batch/s]

Training batches on cuda:  77%|███████▋  | 7507/9708 [00:35<00:10, 204.21batch/s]

Training batches on cuda:  78%|███████▊  | 7529/9708 [00:35<00:10, 206.44batch/s]

Training batches on cuda:  78%|███████▊  | 7550/9708 [00:35<00:10, 206.41batch/s]

Training batches on cuda:  78%|███████▊  | 7573/9708 [00:35<00:10, 210.57batch/s]

Training batches on cuda:  78%|███████▊  | 7596/9708 [00:35<00:09, 213.44batch/s]

Training batches on cuda:  78%|███████▊  | 7620/9708 [00:35<00:09, 219.20batch/s]

Training batches on cuda:  79%|███████▊  | 7643/9708 [00:35<00:09, 213.49batch/s]

Training batches on cuda:  79%|███████▉  | 7665/9708 [00:35<00:09, 211.03batch/s]

Training batches on cuda:  79%|███████▉  | 7687/9708 [00:35<00:10, 198.40batch/s]

Training batches on cuda:  79%|███████▉  | 7708/9708 [00:35<00:10, 199.93batch/s]

Training batches on cuda:  80%|███████▉  | 7730/9708 [00:36<00:09, 203.95batch/s]

Training batches on cuda:  80%|███████▉  | 7752/9708 [00:36<00:09, 206.60batch/s]

Training batches on cuda:  80%|████████  | 7773/9708 [00:36<00:09, 206.79batch/s]

Training batches on cuda:  80%|████████  | 7794/9708 [00:36<00:09, 204.92batch/s]

Training batches on cuda:  81%|████████  | 7815/9708 [00:36<00:09, 202.05batch/s]

Training batches on cuda:  81%|████████  | 7837/9708 [00:36<00:09, 204.29batch/s]

Training batches on cuda:  81%|████████  | 7858/9708 [00:36<00:09, 198.74batch/s]

Training batches on cuda:  81%|████████  | 7879/9708 [00:36<00:09, 200.49batch/s]

Training batches on cuda:  81%|████████▏ | 7900/9708 [00:36<00:08, 202.16batch/s]

Training batches on cuda:  82%|████████▏ | 7921/9708 [00:37<00:08, 201.39batch/s]

Training batches on cuda:  82%|████████▏ | 7942/9708 [00:37<00:08, 202.72batch/s]

Training batches on cuda:  82%|████████▏ | 7966/9708 [00:37<00:08, 211.35batch/s]

Training batches on cuda:  82%|████████▏ | 7990/9708 [00:37<00:07, 217.19batch/s]

Training batches on cuda:  83%|████████▎ | 8013/9708 [00:37<00:07, 220.85batch/s]

Training batches on cuda:  83%|████████▎ | 8036/9708 [00:37<00:07, 212.05batch/s]

Training batches on cuda:  83%|████████▎ | 8058/9708 [00:37<00:07, 210.41batch/s]

Training batches on cuda:  83%|████████▎ | 8081/9708 [00:37<00:07, 215.04batch/s]

Training batches on cuda:  83%|████████▎ | 8103/9708 [00:37<00:07, 209.49batch/s]

Training batches on cuda:  84%|████████▎ | 8125/9708 [00:37<00:07, 206.53batch/s]

Training batches on cuda:  84%|████████▍ | 8146/9708 [00:38<00:07, 204.46batch/s]

Training batches on cuda:  84%|████████▍ | 8167/9708 [00:38<00:07, 206.05batch/s]

Training batches on cuda:  84%|████████▍ | 8188/9708 [00:38<00:07, 204.09batch/s]

Training batches on cuda:  85%|████████▍ | 8211/9708 [00:38<00:07, 209.54batch/s]

Training batches on cuda:  85%|████████▍ | 8234/9708 [00:38<00:06, 213.32batch/s]

Training batches on cuda:  85%|████████▌ | 8256/9708 [00:38<00:06, 212.32batch/s]

Training batches on cuda:  85%|████████▌ | 8278/9708 [00:38<00:06, 212.69batch/s]

Training batches on cuda:  85%|████████▌ | 8300/9708 [00:38<00:06, 207.40batch/s]

Training batches on cuda:  86%|████████▌ | 8321/9708 [00:38<00:06, 202.12batch/s]

Training batches on cuda:  86%|████████▌ | 8342/9708 [00:39<00:07, 191.65batch/s]

Training batches on cuda:  86%|████████▌ | 8362/9708 [00:39<00:07, 191.85batch/s]

Training batches on cuda:  86%|████████▋ | 8384/9708 [00:39<00:06, 198.95batch/s]

Training batches on cuda:  87%|████████▋ | 8405/9708 [00:39<00:06, 194.55batch/s]

Training batches on cuda:  87%|████████▋ | 8427/9708 [00:39<00:06, 200.52batch/s]

Training batches on cuda:  87%|████████▋ | 8448/9708 [00:39<00:06, 196.52batch/s]

Training batches on cuda:  87%|████████▋ | 8468/9708 [00:39<00:06, 196.23batch/s]

Training batches on cuda:  87%|████████▋ | 8489/9708 [00:39<00:06, 196.82batch/s]

Training batches on cuda:  88%|████████▊ | 8512/9708 [00:39<00:05, 203.58batch/s]

Training batches on cuda:  88%|████████▊ | 8536/9708 [00:39<00:05, 211.70batch/s]

Training batches on cuda:  88%|████████▊ | 8558/9708 [00:40<00:05, 211.98batch/s]

Training batches on cuda:  88%|████████▊ | 8580/9708 [00:40<00:05, 211.87batch/s]

Training batches on cuda:  89%|████████▊ | 8603/9708 [00:40<00:05, 216.86batch/s]

Training batches on cuda:  89%|████████▉ | 8625/9708 [00:40<00:05, 214.50batch/s]

Training batches on cuda:  89%|████████▉ | 8647/9708 [00:40<00:04, 212.72batch/s]

Training batches on cuda:  89%|████████▉ | 8669/9708 [00:40<00:04, 209.37batch/s]

Training batches on cuda:  90%|████████▉ | 8690/9708 [00:40<00:04, 205.28batch/s]

Training batches on cuda:  90%|████████▉ | 8711/9708 [00:40<00:05, 190.63batch/s]

Training batches on cuda:  90%|████████▉ | 8731/9708 [00:40<00:05, 192.78batch/s]

Training batches on cuda:  90%|█████████ | 8751/9708 [00:41<00:05, 185.17batch/s]

Training batches on cuda:  90%|█████████ | 8772/9708 [00:41<00:04, 191.74batch/s]

Training batches on cuda:  91%|█████████ | 8792/9708 [00:41<00:04, 189.39batch/s]

Training batches on cuda:  91%|█████████ | 8813/9708 [00:41<00:04, 193.05batch/s]

Training batches on cuda:  91%|█████████ | 8834/9708 [00:41<00:04, 196.50batch/s]

Training batches on cuda:  91%|█████████ | 8855/9708 [00:41<00:04, 198.98batch/s]

Training batches on cuda:  91%|█████████▏| 8876/9708 [00:41<00:04, 200.41batch/s]

Training batches on cuda:  92%|█████████▏| 8897/9708 [00:41<00:04, 201.25batch/s]

Training batches on cuda:  92%|█████████▏| 8918/9708 [00:41<00:03, 203.56batch/s]

Training batches on cuda:  92%|█████████▏| 8940/9708 [00:41<00:03, 208.08batch/s]

Training batches on cuda:  92%|█████████▏| 8963/9708 [00:42<00:03, 213.68batch/s]

Training batches on cuda:  93%|█████████▎| 8985/9708 [00:42<00:03, 205.55batch/s]

Training batches on cuda:  93%|█████████▎| 9007/9708 [00:42<00:03, 207.09batch/s]

Training batches on cuda:  93%|█████████▎| 9028/9708 [00:42<00:03, 205.59batch/s]

Training batches on cuda:  93%|█████████▎| 9049/9708 [00:42<00:03, 203.64batch/s]

Training batches on cuda:  93%|█████████▎| 9070/9708 [00:42<00:03, 202.99batch/s]

Training batches on cuda:  94%|█████████▎| 9091/9708 [00:42<00:03, 200.31batch/s]

Training batches on cuda:  94%|█████████▍| 9112/9708 [00:42<00:02, 200.02batch/s]

Training batches on cuda:  94%|█████████▍| 9133/9708 [00:42<00:02, 199.73batch/s]

Training batches on cuda:  94%|█████████▍| 9153/9708 [00:43<00:03, 183.90batch/s]

Training batches on cuda:  94%|█████████▍| 9174/9708 [00:43<00:02, 189.09batch/s]

Training batches on cuda:  95%|█████████▍| 9194/9708 [00:43<00:02, 191.03batch/s]

Training batches on cuda:  95%|█████████▍| 9215/9708 [00:43<00:02, 194.06batch/s]

Training batches on cuda:  95%|█████████▌| 9237/9708 [00:43<00:02, 199.26batch/s]

Training batches on cuda:  95%|█████████▌| 9258/9708 [00:43<00:02, 201.99batch/s]

Training batches on cuda:  96%|█████████▌| 9279/9708 [00:43<00:02, 202.86batch/s]

Training batches on cuda:  96%|█████████▌| 9300/9708 [00:43<00:01, 204.83batch/s]

Training batches on cuda:  96%|█████████▌| 9321/9708 [00:43<00:01, 198.54batch/s]

Training batches on cuda:  96%|█████████▌| 9341/9708 [00:44<00:01, 196.87batch/s]

Training batches on cuda:  96%|█████████▋| 9363/9708 [00:44<00:01, 200.95batch/s]

Training batches on cuda:  97%|█████████▋| 9384/9708 [00:44<00:01, 193.39batch/s]

Training batches on cuda:  97%|█████████▋| 9404/9708 [00:44<00:01, 194.20batch/s]

Training batches on cuda:  97%|█████████▋| 9424/9708 [00:44<00:01, 191.88batch/s]

Training batches on cuda:  97%|█████████▋| 9449/9708 [00:44<00:01, 204.94batch/s]

Training batches on cuda:  98%|█████████▊| 9474/9708 [00:44<00:01, 215.17batch/s]

Training batches on cuda:  98%|█████████▊| 9499/9708 [00:44<00:00, 223.94batch/s]

Training batches on cuda:  98%|█████████▊| 9526/9708 [00:44<00:00, 235.41batch/s]

Training batches on cuda:  98%|█████████▊| 9553/9708 [00:44<00:00, 244.07batch/s]

Training batches on cuda:  99%|█████████▊| 9580/9708 [00:45<00:00, 249.89batch/s]

Training batches on cuda:  99%|█████████▉| 9607/9708 [00:45<00:00, 254.49batch/s]

Training batches on cuda:  99%|█████████▉| 9634/9708 [00:45<00:00, 257.60batch/s]

Training batches on cuda: 100%|█████████▉| 9661/9708 [00:45<00:00, 259.70batch/s]

Training batches on cuda: 100%|█████████▉| 9688/9708 [00:45<00:00, 261.77batch/s]

                                                                                 
Training epochs on cuda:  60%|██████    | 6/10 [05:26<03:06, 46.70s/epoch, loss=0.0417, prev_loss=0.0416]
Training epochs on cuda:  70%|███████   | 7/10 [05:26<02:19, 46.44s/epoch, loss=0.0417, prev_loss=0.0416]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 21/9708 [00:00<00:46, 209.22batch/s]

Training batches on cuda:   0%|          | 48/9708 [00:00<00:43, 223.92batch/s]

Training batches on cuda:   1%|          | 75/9708 [00:00<00:40, 235.04batch/s]

Training batches on cuda:   1%|          | 102/9708 [00:00<00:39, 243.70batch/s]

Training batches on cuda:   1%|▏         | 125/9708 [00:00<00:40, 238.85batch/s]

Training batches on cuda:   2%|▏         | 146/9708 [00:00<00:42, 225.89batch/s]

Training batches on cuda:   2%|▏         | 167/9708 [00:00<00:44, 214.93batch/s]

Training batches on cuda:   2%|▏         | 188/9708 [00:00<00:46, 205.32batch/s]

Training batches on cuda:   2%|▏         | 210/9708 [00:00<00:45, 208.24batch/s]

Training batches on cuda:   2%|▏         | 232/9708 [00:01<00:45, 210.45batch/s]

Training batches on cuda:   3%|▎         | 254/9708 [00:01<00:44, 210.71batch/s]

Training batches on cuda:   3%|▎         | 276/9708 [00:01<00:44, 211.03batch/s]

Training batches on cuda:   3%|▎         | 297/9708 [00:01<00:45, 208.98batch/s]

Training batches on cuda:   3%|▎         | 318/9708 [00:01<00:45, 206.54batch/s]

Training batches on cuda:   3%|▎         | 339/9708 [00:01<00:45, 205.54batch/s]

Training batches on cuda:   4%|▎         | 363/9708 [00:01<00:43, 213.73batch/s]

Training batches on cuda:   4%|▍         | 387/9708 [00:01<00:42, 220.11batch/s]

Training batches on cuda:   4%|▍         | 410/9708 [00:01<00:41, 222.16batch/s]

Training batches on cuda:   4%|▍         | 434/9708 [00:01<00:40, 226.65batch/s]

Training batches on cuda:   5%|▍         | 458/9708 [00:02<00:40, 228.20batch/s]

Training batches on cuda:   5%|▍         | 482/9708 [00:02<00:40, 230.51batch/s]

Training batches on cuda:   5%|▌         | 507/9708 [00:02<00:39, 234.36batch/s]

Training batches on cuda:   5%|▌         | 531/9708 [00:02<00:39, 233.81batch/s]

Training batches on cuda:   6%|▌         | 555/9708 [00:02<00:39, 233.05batch/s]

Training batches on cuda:   6%|▌         | 579/9708 [00:02<00:41, 221.43batch/s]

Training batches on cuda:   6%|▌         | 602/9708 [00:02<00:42, 216.02batch/s]

Training batches on cuda:   6%|▋         | 625/9708 [00:02<00:41, 218.89batch/s]

Training batches on cuda:   7%|▋         | 647/9708 [00:02<00:41, 218.59batch/s]

Training batches on cuda:   7%|▋         | 671/9708 [00:03<00:40, 221.87batch/s]

Training batches on cuda:   7%|▋         | 694/9708 [00:03<00:41, 215.38batch/s]

Training batches on cuda:   7%|▋         | 717/9708 [00:03<00:41, 217.30batch/s]

Training batches on cuda:   8%|▊         | 739/9708 [00:03<00:42, 209.71batch/s]

Training batches on cuda:   8%|▊         | 761/9708 [00:03<00:44, 202.56batch/s]

Training batches on cuda:   8%|▊         | 782/9708 [00:03<00:44, 201.35batch/s]

Training batches on cuda:   8%|▊         | 803/9708 [00:03<00:45, 195.49batch/s]

Training batches on cuda:   8%|▊         | 823/9708 [00:03<00:46, 191.98batch/s]

Training batches on cuda:   9%|▊         | 843/9708 [00:03<00:46, 188.63batch/s]

Training batches on cuda:   9%|▉         | 863/9708 [00:04<00:46, 190.37batch/s]

Training batches on cuda:   9%|▉         | 885/9708 [00:04<00:44, 197.02batch/s]

Training batches on cuda:   9%|▉         | 907/9708 [00:04<00:43, 200.34batch/s]

Training batches on cuda:  10%|▉         | 928/9708 [00:04<00:45, 192.89batch/s]

Training batches on cuda:  10%|▉         | 948/9708 [00:04<00:46, 189.97batch/s]

Training batches on cuda:  10%|▉         | 968/9708 [00:04<00:46, 186.66batch/s]

Training batches on cuda:  10%|█         | 987/9708 [00:04<00:46, 186.91batch/s]

Training batches on cuda:  10%|█         | 1006/9708 [00:04<00:46, 186.36batch/s]

Training batches on cuda:  11%|█         | 1025/9708 [00:04<00:47, 183.72batch/s]

Training batches on cuda:  11%|█         | 1044/9708 [00:04<00:47, 183.67batch/s]

Training batches on cuda:  11%|█         | 1065/9708 [00:05<00:45, 189.38batch/s]

Training batches on cuda:  11%|█         | 1086/9708 [00:05<00:44, 193.86batch/s]

Training batches on cuda:  11%|█▏        | 1107/9708 [00:05<00:43, 195.56batch/s]

Training batches on cuda:  12%|█▏        | 1127/9708 [00:05<00:43, 196.38batch/s]

Training batches on cuda:  12%|█▏        | 1148/9708 [00:05<00:43, 198.29batch/s]

Training batches on cuda:  12%|█▏        | 1168/9708 [00:05<00:44, 194.07batch/s]

Training batches on cuda:  12%|█▏        | 1188/9708 [00:05<00:44, 191.42batch/s]

Training batches on cuda:  12%|█▏        | 1208/9708 [00:05<00:45, 188.66batch/s]

Training batches on cuda:  13%|█▎        | 1228/9708 [00:05<00:44, 188.52batch/s]

Training batches on cuda:  13%|█▎        | 1247/9708 [00:06<00:48, 175.47batch/s]

Training batches on cuda:  13%|█▎        | 1267/9708 [00:06<00:46, 181.54batch/s]

Training batches on cuda:  13%|█▎        | 1288/9708 [00:06<00:44, 188.30batch/s]

Training batches on cuda:  13%|█▎        | 1308/9708 [00:06<00:44, 189.87batch/s]

Training batches on cuda:  14%|█▎        | 1328/9708 [00:06<00:44, 188.29batch/s]

Training batches on cuda:  14%|█▍        | 1348/9708 [00:06<00:43, 190.17batch/s]

Training batches on cuda:  14%|█▍        | 1368/9708 [00:06<00:43, 191.30batch/s]

Training batches on cuda:  14%|█▍        | 1388/9708 [00:06<00:42, 193.80batch/s]

Training batches on cuda:  15%|█▍        | 1411/9708 [00:06<00:41, 201.71batch/s]

Training batches on cuda:  15%|█▍        | 1432/9708 [00:06<00:40, 202.27batch/s]

Training batches on cuda:  15%|█▍        | 1453/9708 [00:07<00:41, 197.11batch/s]

Training batches on cuda:  15%|█▌        | 1477/9708 [00:07<00:39, 206.60batch/s]

Training batches on cuda:  15%|█▌        | 1502/9708 [00:07<00:37, 216.41batch/s]

Training batches on cuda:  16%|█▌        | 1527/9708 [00:07<00:36, 224.17batch/s]

Training batches on cuda:  16%|█▌        | 1554/9708 [00:07<00:34, 236.06batch/s]

Training batches on cuda:  16%|█▋        | 1580/9708 [00:07<00:33, 240.44batch/s]

Training batches on cuda:  17%|█▋        | 1605/9708 [00:07<00:35, 225.41batch/s]

Training batches on cuda:  17%|█▋        | 1628/9708 [00:07<00:36, 220.37batch/s]

Training batches on cuda:  17%|█▋        | 1651/9708 [00:07<00:37, 217.64batch/s]

Training batches on cuda:  17%|█▋        | 1673/9708 [00:08<00:37, 215.89batch/s]

Training batches on cuda:  17%|█▋        | 1695/9708 [00:08<00:37, 214.27batch/s]

Training batches on cuda:  18%|█▊        | 1717/9708 [00:08<00:37, 214.15batch/s]

Training batches on cuda:  18%|█▊        | 1739/9708 [00:08<00:38, 209.68batch/s]

Training batches on cuda:  18%|█▊        | 1761/9708 [00:08<00:38, 207.43batch/s]

Training batches on cuda:  18%|█▊        | 1782/9708 [00:08<00:38, 206.67batch/s]

Training batches on cuda:  19%|█▊        | 1803/9708 [00:08<00:38, 205.88batch/s]

Training batches on cuda:  19%|█▉        | 1824/9708 [00:08<00:38, 203.38batch/s]

Training batches on cuda:  19%|█▉        | 1847/9708 [00:08<00:37, 209.40batch/s]

Training batches on cuda:  19%|█▉        | 1869/9708 [00:08<00:37, 209.40batch/s]

Training batches on cuda:  19%|█▉        | 1890/9708 [00:09<00:37, 208.86batch/s]

Training batches on cuda:  20%|█▉        | 1911/9708 [00:09<00:37, 208.84batch/s]

Training batches on cuda:  20%|█▉        | 1932/9708 [00:09<00:37, 208.54batch/s]

Training batches on cuda:  20%|██        | 1953/9708 [00:09<00:37, 204.86batch/s]

Training batches on cuda:  20%|██        | 1974/9708 [00:09<00:38, 203.31batch/s]

Training batches on cuda:  21%|██        | 1995/9708 [00:09<00:37, 203.48batch/s]

Training batches on cuda:  21%|██        | 2016/9708 [00:09<00:38, 201.62batch/s]

Training batches on cuda:  21%|██        | 2037/9708 [00:09<00:37, 202.37batch/s]

Training batches on cuda:  21%|██        | 2058/9708 [00:09<00:38, 197.80batch/s]

Training batches on cuda:  21%|██▏       | 2078/9708 [00:10<00:38, 198.34batch/s]

Training batches on cuda:  22%|██▏       | 2098/9708 [00:10<00:39, 192.18batch/s]

Training batches on cuda:  22%|██▏       | 2118/9708 [00:10<00:43, 173.55batch/s]

Training batches on cuda:  22%|██▏       | 2138/9708 [00:10<00:41, 180.24batch/s]

Training batches on cuda:  22%|██▏       | 2157/9708 [00:10<00:41, 182.91batch/s]

Training batches on cuda:  22%|██▏       | 2178/9708 [00:10<00:39, 189.83batch/s]

Training batches on cuda:  23%|██▎       | 2199/9708 [00:10<00:38, 194.67batch/s]

Training batches on cuda:  23%|██▎       | 2220/9708 [00:10<00:37, 198.12batch/s]

Training batches on cuda:  23%|██▎       | 2243/9708 [00:10<00:36, 204.35batch/s]

Training batches on cuda:  23%|██▎       | 2265/9708 [00:10<00:35, 208.48batch/s]

Training batches on cuda:  24%|██▎       | 2287/9708 [00:11<00:35, 210.38batch/s]

Training batches on cuda:  24%|██▍       | 2311/9708 [00:11<00:34, 216.24batch/s]

Training batches on cuda:  24%|██▍       | 2334/9708 [00:11<00:33, 219.29batch/s]

Training batches on cuda:  24%|██▍       | 2357/9708 [00:11<00:33, 218.78batch/s]

Training batches on cuda:  25%|██▍       | 2379/9708 [00:11<00:35, 208.10batch/s]

Training batches on cuda:  25%|██▍       | 2400/9708 [00:11<00:37, 197.04batch/s]

Training batches on cuda:  25%|██▍       | 2420/9708 [00:11<00:37, 193.44batch/s]

Training batches on cuda:  25%|██▌       | 2440/9708 [00:11<00:38, 188.26batch/s]

Training batches on cuda:  25%|██▌       | 2459/9708 [00:11<00:38, 188.28batch/s]

Training batches on cuda:  26%|██▌       | 2482/9708 [00:12<00:36, 198.84batch/s]

Training batches on cuda:  26%|██▌       | 2504/9708 [00:12<00:35, 203.59batch/s]

Training batches on cuda:  26%|██▌       | 2527/9708 [00:12<00:34, 209.53batch/s]

Training batches on cuda:  26%|██▋       | 2550/9708 [00:12<00:33, 212.77batch/s]

Training batches on cuda:  27%|██▋       | 2573/9708 [00:12<00:32, 217.38batch/s]

Training batches on cuda:  27%|██▋       | 2595/9708 [00:12<00:33, 212.75batch/s]

Training batches on cuda:  27%|██▋       | 2617/9708 [00:12<00:33, 211.36batch/s]

Training batches on cuda:  27%|██▋       | 2639/9708 [00:12<00:33, 210.57batch/s]

Training batches on cuda:  27%|██▋       | 2661/9708 [00:12<00:33, 211.39batch/s]

Training batches on cuda:  28%|██▊       | 2683/9708 [00:12<00:33, 212.41batch/s]

Training batches on cuda:  28%|██▊       | 2705/9708 [00:13<00:33, 208.67batch/s]

Training batches on cuda:  28%|██▊       | 2726/9708 [00:13<00:34, 203.10batch/s]

Training batches on cuda:  28%|██▊       | 2747/9708 [00:13<00:35, 196.53batch/s]

Training batches on cuda:  29%|██▊       | 2767/9708 [00:13<00:35, 192.94batch/s]

Training batches on cuda:  29%|██▊       | 2787/9708 [00:13<00:37, 183.31batch/s]

Training batches on cuda:  29%|██▉       | 2808/9708 [00:13<00:36, 188.32batch/s]

Training batches on cuda:  29%|██▉       | 2827/9708 [00:13<00:36, 187.94batch/s]

Training batches on cuda:  29%|██▉       | 2847/9708 [00:13<00:36, 189.04batch/s]

Training batches on cuda:  30%|██▉       | 2868/9708 [00:13<00:35, 192.24batch/s]

Training batches on cuda:  30%|██▉       | 2888/9708 [00:14<00:36, 188.35batch/s]

Training batches on cuda:  30%|██▉       | 2907/9708 [00:14<00:36, 187.80batch/s]

Training batches on cuda:  30%|███       | 2926/9708 [00:14<00:36, 186.61batch/s]

Training batches on cuda:  30%|███       | 2945/9708 [00:14<00:36, 187.32batch/s]

Training batches on cuda:  31%|███       | 2964/9708 [00:14<00:36, 187.06batch/s]

Training batches on cuda:  31%|███       | 2985/9708 [00:14<00:35, 191.08batch/s]

Training batches on cuda:  31%|███       | 3006/9708 [00:14<00:34, 195.95batch/s]

Training batches on cuda:  31%|███       | 3026/9708 [00:14<00:34, 191.72batch/s]

Training batches on cuda:  31%|███▏      | 3046/9708 [00:14<00:34, 193.76batch/s]

Training batches on cuda:  32%|███▏      | 3066/9708 [00:15<00:34, 192.41batch/s]

Training batches on cuda:  32%|███▏      | 3086/9708 [00:15<00:34, 193.63batch/s]

Training batches on cuda:  32%|███▏      | 3106/9708 [00:15<00:34, 193.60batch/s]

Training batches on cuda:  32%|███▏      | 3126/9708 [00:15<00:35, 186.07batch/s]

Training batches on cuda:  32%|███▏      | 3146/9708 [00:15<00:34, 189.44batch/s]

Training batches on cuda:  33%|███▎      | 3166/9708 [00:15<00:34, 191.02batch/s]

Training batches on cuda:  33%|███▎      | 3186/9708 [00:15<00:34, 191.27batch/s]

Training batches on cuda:  33%|███▎      | 3206/9708 [00:15<00:33, 193.53batch/s]

Training batches on cuda:  33%|███▎      | 3226/9708 [00:15<00:33, 193.23batch/s]

Training batches on cuda:  33%|███▎      | 3246/9708 [00:15<00:33, 195.04batch/s]

Training batches on cuda:  34%|███▎      | 3266/9708 [00:16<00:33, 194.97batch/s]

Training batches on cuda:  34%|███▍      | 3286/9708 [00:16<00:32, 195.76batch/s]

Training batches on cuda:  34%|███▍      | 3306/9708 [00:16<00:32, 195.89batch/s]

Training batches on cuda:  34%|███▍      | 3328/9708 [00:16<00:31, 202.45batch/s]

Training batches on cuda:  35%|███▍      | 3350/9708 [00:16<00:30, 205.14batch/s]

Training batches on cuda:  35%|███▍      | 3372/9708 [00:16<00:30, 207.13batch/s]

Training batches on cuda:  35%|███▍      | 3393/9708 [00:16<00:30, 207.28batch/s]

Training batches on cuda:  35%|███▌      | 3415/9708 [00:16<00:30, 209.14batch/s]

Training batches on cuda:  35%|███▌      | 3437/9708 [00:16<00:29, 209.20batch/s]

Training batches on cuda:  36%|███▌      | 3459/9708 [00:16<00:29, 210.84batch/s]

Training batches on cuda:  36%|███▌      | 3481/9708 [00:17<00:29, 210.34batch/s]

Training batches on cuda:  36%|███▌      | 3503/9708 [00:17<00:29, 209.06batch/s]

Training batches on cuda:  36%|███▋      | 3525/9708 [00:17<00:29, 209.55batch/s]

Training batches on cuda:  37%|███▋      | 3547/9708 [00:17<00:29, 210.69batch/s]

Training batches on cuda:  37%|███▋      | 3569/9708 [00:17<00:29, 211.12batch/s]

Training batches on cuda:  37%|███▋      | 3591/9708 [00:17<00:28, 213.10batch/s]

Training batches on cuda:  37%|███▋      | 3614/9708 [00:17<00:28, 216.65batch/s]

Training batches on cuda:  37%|███▋      | 3637/9708 [00:17<00:27, 219.31batch/s]

Training batches on cuda:  38%|███▊      | 3661/9708 [00:17<00:27, 223.56batch/s]

Training batches on cuda:  38%|███▊      | 3686/9708 [00:18<00:26, 230.39batch/s]

Training batches on cuda:  38%|███▊      | 3712/9708 [00:18<00:25, 236.65batch/s]

Training batches on cuda:  38%|███▊      | 3737/9708 [00:18<00:25, 237.79batch/s]

Training batches on cuda:  39%|███▉      | 3762/9708 [00:18<00:24, 238.74batch/s]

Training batches on cuda:  39%|███▉      | 3787/9708 [00:18<00:24, 241.33batch/s]

Training batches on cuda:  39%|███▉      | 3812/9708 [00:18<00:24, 239.93batch/s]

Training batches on cuda:  40%|███▉      | 3838/9708 [00:18<00:24, 243.05batch/s]

Training batches on cuda:  40%|███▉      | 3863/9708 [00:18<00:24, 238.61batch/s]

Training batches on cuda:  40%|████      | 3887/9708 [00:18<00:25, 229.45batch/s]

Training batches on cuda:  40%|████      | 3911/9708 [00:18<00:26, 221.80batch/s]

Training batches on cuda:  41%|████      | 3934/9708 [00:19<00:26, 216.90batch/s]

Training batches on cuda:  41%|████      | 3956/9708 [00:19<00:26, 216.85batch/s]

Training batches on cuda:  41%|████      | 3978/9708 [00:19<00:26, 215.93batch/s]

Training batches on cuda:  41%|████      | 4002/9708 [00:19<00:25, 220.37batch/s]

Training batches on cuda:  41%|████▏     | 4025/9708 [00:19<00:25, 223.17batch/s]

Training batches on cuda:  42%|████▏     | 4049/9708 [00:19<00:25, 225.23batch/s]

Training batches on cuda:  42%|████▏     | 4073/9708 [00:19<00:24, 226.98batch/s]

Training batches on cuda:  42%|████▏     | 4096/9708 [00:19<00:25, 221.48batch/s]

Training batches on cuda:  42%|████▏     | 4119/9708 [00:19<00:25, 216.87batch/s]

Training batches on cuda:  43%|████▎     | 4141/9708 [00:20<00:26, 214.07batch/s]

Training batches on cuda:  43%|████▎     | 4163/9708 [00:20<00:26, 212.89batch/s]

Training batches on cuda:  43%|████▎     | 4186/9708 [00:20<00:25, 216.19batch/s]

Training batches on cuda:  43%|████▎     | 4208/9708 [00:20<00:28, 195.25batch/s]

Training batches on cuda:  44%|████▎     | 4231/9708 [00:20<00:26, 202.97batch/s]

Training batches on cuda:  44%|████▍     | 4253/9708 [00:20<00:26, 207.11batch/s]

Training batches on cuda:  44%|████▍     | 4277/9708 [00:20<00:25, 213.57batch/s]

Training batches on cuda:  44%|████▍     | 4299/9708 [00:20<00:25, 214.04batch/s]

Training batches on cuda:  45%|████▍     | 4321/9708 [00:20<00:26, 204.30batch/s]

Training batches on cuda:  45%|████▍     | 4343/9708 [00:20<00:25, 208.73batch/s]

Training batches on cuda:  45%|████▍     | 4366/9708 [00:21<00:24, 214.50batch/s]

Training batches on cuda:  45%|████▌     | 4390/9708 [00:21<00:24, 219.61batch/s]

Training batches on cuda:  45%|████▌     | 4413/9708 [00:21<00:24, 219.40batch/s]

Training batches on cuda:  46%|████▌     | 4436/9708 [00:21<00:24, 215.13batch/s]

Training batches on cuda:  46%|████▌     | 4458/9708 [00:21<00:24, 212.51batch/s]

Training batches on cuda:  46%|████▌     | 4480/9708 [00:21<00:24, 213.20batch/s]

Training batches on cuda:  46%|████▋     | 4503/9708 [00:21<00:24, 216.23batch/s]

Training batches on cuda:  47%|████▋     | 4527/9708 [00:21<00:23, 221.63batch/s]

Training batches on cuda:  47%|████▋     | 4550/9708 [00:21<00:23, 224.00batch/s]

Training batches on cuda:  47%|████▋     | 4573/9708 [00:22<00:22, 225.57batch/s]

Training batches on cuda:  47%|████▋     | 4597/9708 [00:22<00:22, 226.93batch/s]

Training batches on cuda:  48%|████▊     | 4620/9708 [00:22<00:22, 227.77batch/s]

Training batches on cuda:  48%|████▊     | 4644/9708 [00:22<00:22, 228.38batch/s]

Training batches on cuda:  48%|████▊     | 4668/9708 [00:22<00:21, 229.43batch/s]

Training batches on cuda:  48%|████▊     | 4691/9708 [00:22<00:22, 223.17batch/s]

Training batches on cuda:  49%|████▊     | 4714/9708 [00:22<00:22, 218.60batch/s]

Training batches on cuda:  49%|████▉     | 4736/9708 [00:22<00:23, 214.94batch/s]

Training batches on cuda:  49%|████▉     | 4758/9708 [00:22<00:23, 211.44batch/s]

Training batches on cuda:  49%|████▉     | 4780/9708 [00:22<00:23, 208.99batch/s]

Training batches on cuda:  49%|████▉     | 4801/9708 [00:23<00:23, 206.44batch/s]

Training batches on cuda:  50%|████▉     | 4822/9708 [00:23<00:23, 205.26batch/s]

Training batches on cuda:  50%|████▉     | 4843/9708 [00:23<00:23, 205.17batch/s]

Training batches on cuda:  50%|█████     | 4864/9708 [00:23<00:23, 206.12batch/s]

Training batches on cuda:  50%|█████     | 4886/9708 [00:23<00:23, 209.16batch/s]

Training batches on cuda:  51%|█████     | 4908/9708 [00:23<00:22, 210.16batch/s]

Training batches on cuda:  51%|█████     | 4930/9708 [00:23<00:22, 209.59batch/s]

Training batches on cuda:  51%|█████     | 4951/9708 [00:23<00:24, 197.33batch/s]

Training batches on cuda:  51%|█████     | 4971/9708 [00:23<00:24, 196.67batch/s]

Training batches on cuda:  51%|█████▏    | 4991/9708 [00:24<00:24, 196.02batch/s]

Training batches on cuda:  52%|█████▏    | 5011/9708 [00:24<00:23, 195.77batch/s]

Training batches on cuda:  52%|█████▏    | 5031/9708 [00:24<00:23, 196.00batch/s]

Training batches on cuda:  52%|█████▏    | 5051/9708 [00:24<00:23, 194.89batch/s]

Training batches on cuda:  52%|█████▏    | 5071/9708 [00:24<00:23, 196.38batch/s]

Training batches on cuda:  52%|█████▏    | 5091/9708 [00:24<00:23, 196.49batch/s]

Training batches on cuda:  53%|█████▎    | 5111/9708 [00:24<00:23, 194.67batch/s]

Training batches on cuda:  53%|█████▎    | 5131/9708 [00:24<00:23, 192.20batch/s]

Training batches on cuda:  53%|█████▎    | 5151/9708 [00:24<00:23, 192.84batch/s]

Training batches on cuda:  53%|█████▎    | 5171/9708 [00:24<00:23, 194.15batch/s]

Training batches on cuda:  53%|█████▎    | 5191/9708 [00:25<00:23, 193.80batch/s]

Training batches on cuda:  54%|█████▎    | 5211/9708 [00:25<00:23, 193.87batch/s]

Training batches on cuda:  54%|█████▍    | 5231/9708 [00:25<00:22, 194.70batch/s]

Training batches on cuda:  54%|█████▍    | 5251/9708 [00:25<00:24, 180.58batch/s]

Training batches on cuda:  54%|█████▍    | 5271/9708 [00:25<00:23, 185.23batch/s]

Training batches on cuda:  55%|█████▍    | 5293/9708 [00:25<00:22, 192.05batch/s]

Training batches on cuda:  55%|█████▍    | 5314/9708 [00:25<00:22, 196.98batch/s]

Training batches on cuda:  55%|█████▍    | 5336/9708 [00:25<00:21, 201.18batch/s]

Training batches on cuda:  55%|█████▌    | 5357/9708 [00:25<00:21, 203.51batch/s]

Training batches on cuda:  55%|█████▌    | 5378/9708 [00:26<00:22, 195.14batch/s]

Training batches on cuda:  56%|█████▌    | 5398/9708 [00:26<00:22, 189.26batch/s]

Training batches on cuda:  56%|█████▌    | 5418/9708 [00:26<00:23, 184.24batch/s]

Training batches on cuda:  56%|█████▌    | 5439/9708 [00:26<00:22, 189.45batch/s]

Training batches on cuda:  56%|█████▋    | 5462/9708 [00:26<00:21, 198.38batch/s]

Training batches on cuda:  56%|█████▋    | 5483/9708 [00:26<00:21, 200.95batch/s]

Training batches on cuda:  57%|█████▋    | 5504/9708 [00:26<00:21, 197.85batch/s]

Training batches on cuda:  57%|█████▋    | 5524/9708 [00:26<00:21, 194.43batch/s]

Training batches on cuda:  57%|█████▋    | 5544/9708 [00:26<00:21, 194.80batch/s]

Training batches on cuda:  57%|█████▋    | 5565/9708 [00:26<00:21, 196.97batch/s]

Training batches on cuda:  58%|█████▊    | 5586/9708 [00:27<00:20, 199.80batch/s]

Training batches on cuda:  58%|█████▊    | 5607/9708 [00:27<00:20, 199.90batch/s]

Training batches on cuda:  58%|█████▊    | 5628/9708 [00:27<00:20, 196.85batch/s]

Training batches on cuda:  58%|█████▊    | 5648/9708 [00:27<00:20, 196.74batch/s]

Training batches on cuda:  58%|█████▊    | 5668/9708 [00:27<00:20, 196.37batch/s]

Training batches on cuda:  59%|█████▊    | 5689/9708 [00:27<00:20, 198.82batch/s]

Training batches on cuda:  59%|█████▉    | 5711/9708 [00:27<00:19, 202.25batch/s]

Training batches on cuda:  59%|█████▉    | 5733/9708 [00:27<00:19, 205.46batch/s]

Training batches on cuda:  59%|█████▉    | 5755/9708 [00:27<00:19, 207.46batch/s]

Training batches on cuda:  60%|█████▉    | 5777/9708 [00:28<00:18, 210.19batch/s]

Training batches on cuda:  60%|█████▉    | 5800/9708 [00:28<00:18, 215.64batch/s]

Training batches on cuda:  60%|█████▉    | 5822/9708 [00:28<00:18, 208.83batch/s]

Training batches on cuda:  60%|██████    | 5843/9708 [00:28<00:19, 200.08batch/s]

Training batches on cuda:  60%|██████    | 5864/9708 [00:28<00:19, 197.38batch/s]

Training batches on cuda:  61%|██████    | 5886/9708 [00:28<00:18, 201.38batch/s]

Training batches on cuda:  61%|██████    | 5907/9708 [00:28<00:18, 203.14batch/s]

Training batches on cuda:  61%|██████    | 5928/9708 [00:28<00:18, 202.01batch/s]

Training batches on cuda:  61%|██████▏   | 5950/9708 [00:28<00:18, 204.89batch/s]

Training batches on cuda:  62%|██████▏   | 5972/9708 [00:28<00:18, 206.07batch/s]

Training batches on cuda:  62%|██████▏   | 5993/9708 [00:29<00:17, 207.18batch/s]

Training batches on cuda:  62%|██████▏   | 6014/9708 [00:29<00:17, 207.89batch/s]

Training batches on cuda:  62%|██████▏   | 6037/9708 [00:29<00:17, 211.62batch/s]

Training batches on cuda:  62%|██████▏   | 6059/9708 [00:29<00:17, 208.08batch/s]

Training batches on cuda:  63%|██████▎   | 6080/9708 [00:29<00:17, 206.90batch/s]

Training batches on cuda:  63%|██████▎   | 6102/9708 [00:29<00:17, 210.18batch/s]

Training batches on cuda:  63%|██████▎   | 6125/9708 [00:29<00:16, 213.64batch/s]

Training batches on cuda:  63%|██████▎   | 6149/9708 [00:29<00:16, 219.55batch/s]

Training batches on cuda:  64%|██████▎   | 6172/9708 [00:29<00:15, 221.04batch/s]

Training batches on cuda:  64%|██████▍   | 6195/9708 [00:30<00:15, 222.88batch/s]

Training batches on cuda:  64%|██████▍   | 6218/9708 [00:30<00:16, 209.73batch/s]

Training batches on cuda:  64%|██████▍   | 6240/9708 [00:30<00:17, 199.68batch/s]

Training batches on cuda:  64%|██████▍   | 6261/9708 [00:30<00:19, 173.78batch/s]

Training batches on cuda:  65%|██████▍   | 6280/9708 [00:30<00:19, 176.56batch/s]

Training batches on cuda:  65%|██████▍   | 6301/9708 [00:30<00:18, 185.27batch/s]

Training batches on cuda:  65%|██████▌   | 6323/9708 [00:30<00:17, 192.15batch/s]

Training batches on cuda:  65%|██████▌   | 6344/9708 [00:30<00:17, 194.95batch/s]

Training batches on cuda:  66%|██████▌   | 6365/9708 [00:30<00:16, 196.95batch/s]

Training batches on cuda:  66%|██████▌   | 6386/9708 [00:31<00:16, 198.06batch/s]

Training batches on cuda:  66%|██████▌   | 6408/9708 [00:31<00:16, 203.22batch/s]

Training batches on cuda:  66%|██████▌   | 6429/9708 [00:31<00:17, 187.92batch/s]

Training batches on cuda:  66%|██████▋   | 6449/9708 [00:31<00:17, 190.55batch/s]

Training batches on cuda:  67%|██████▋   | 6470/9708 [00:31<00:16, 193.38batch/s]

Training batches on cuda:  67%|██████▋   | 6490/9708 [00:31<00:16, 194.96batch/s]

Training batches on cuda:  67%|██████▋   | 6513/9708 [00:31<00:15, 202.75batch/s]

Training batches on cuda:  67%|██████▋   | 6537/9708 [00:31<00:15, 210.48batch/s]

Training batches on cuda:  68%|██████▊   | 6562/9708 [00:31<00:14, 219.73batch/s]

Training batches on cuda:  68%|██████▊   | 6587/9708 [00:31<00:13, 227.60batch/s]

Training batches on cuda:  68%|██████▊   | 6614/9708 [00:32<00:13, 237.15batch/s]

Training batches on cuda:  68%|██████▊   | 6638/9708 [00:32<00:12, 236.45batch/s]

Training batches on cuda:  69%|██████▊   | 6662/9708 [00:32<00:14, 214.56batch/s]

Training batches on cuda:  69%|██████▉   | 6685/9708 [00:32<00:13, 216.46batch/s]

Training batches on cuda:  69%|██████▉   | 6708/9708 [00:32<00:14, 212.72batch/s]

Training batches on cuda:  69%|██████▉   | 6730/9708 [00:32<00:14, 211.74batch/s]

Training batches on cuda:  70%|██████▉   | 6752/9708 [00:32<00:13, 213.52batch/s]

Training batches on cuda:  70%|██████▉   | 6774/9708 [00:32<00:13, 214.03batch/s]

Training batches on cuda:  70%|███████   | 6797/9708 [00:32<00:13, 217.40batch/s]

Training batches on cuda:  70%|███████   | 6819/9708 [00:33<00:14, 205.21batch/s]

Training batches on cuda:  70%|███████   | 6840/9708 [00:33<00:14, 201.11batch/s]

Training batches on cuda:  71%|███████   | 6863/9708 [00:33<00:13, 206.50batch/s]

Training batches on cuda:  71%|███████   | 6885/9708 [00:33<00:13, 209.41batch/s]

Training batches on cuda:  71%|███████   | 6907/9708 [00:33<00:13, 207.37batch/s]

Training batches on cuda:  71%|███████▏  | 6929/9708 [00:33<00:13, 209.44batch/s]

Training batches on cuda:  72%|███████▏  | 6951/9708 [00:33<00:13, 197.17batch/s]

Training batches on cuda:  72%|███████▏  | 6971/9708 [00:33<00:14, 187.38batch/s]

Training batches on cuda:  72%|███████▏  | 6991/9708 [00:33<00:14, 189.84batch/s]

Training batches on cuda:  72%|███████▏  | 7012/9708 [00:34<00:13, 193.82batch/s]

Training batches on cuda:  72%|███████▏  | 7035/9708 [00:34<00:13, 202.62batch/s]

Training batches on cuda:  73%|███████▎  | 7058/9708 [00:34<00:12, 210.08batch/s]

Training batches on cuda:  73%|███████▎  | 7081/9708 [00:34<00:12, 212.59batch/s]

Training batches on cuda:  73%|███████▎  | 7104/9708 [00:34<00:12, 215.77batch/s]

Training batches on cuda:  73%|███████▎  | 7126/9708 [00:34<00:12, 211.07batch/s]

Training batches on cuda:  74%|███████▎  | 7148/9708 [00:34<00:12, 203.14batch/s]

Training batches on cuda:  74%|███████▍  | 7169/9708 [00:34<00:12, 199.89batch/s]

Training batches on cuda:  74%|███████▍  | 7190/9708 [00:34<00:12, 195.71batch/s]

Training batches on cuda:  74%|███████▍  | 7210/9708 [00:35<00:12, 193.12batch/s]

Training batches on cuda:  74%|███████▍  | 7230/9708 [00:35<00:12, 194.83batch/s]

Training batches on cuda:  75%|███████▍  | 7250/9708 [00:35<00:12, 194.86batch/s]

Training batches on cuda:  75%|███████▍  | 7272/9708 [00:35<00:12, 201.05batch/s]

Training batches on cuda:  75%|███████▌  | 7293/9708 [00:35<00:12, 192.10batch/s]

Training batches on cuda:  75%|███████▌  | 7314/9708 [00:35<00:12, 197.01batch/s]

Training batches on cuda:  76%|███████▌  | 7337/9708 [00:35<00:11, 204.12batch/s]

Training batches on cuda:  76%|███████▌  | 7360/9708 [00:35<00:11, 211.01batch/s]

Training batches on cuda:  76%|███████▌  | 7382/9708 [00:35<00:11, 207.10batch/s]

Training batches on cuda:  76%|███████▋  | 7403/9708 [00:35<00:11, 205.72batch/s]

Training batches on cuda:  76%|███████▋  | 7424/9708 [00:36<00:11, 203.85batch/s]

Training batches on cuda:  77%|███████▋  | 7445/9708 [00:36<00:11, 204.38batch/s]

Training batches on cuda:  77%|███████▋  | 7467/9708 [00:36<00:10, 205.87batch/s]

Training batches on cuda:  77%|███████▋  | 7488/9708 [00:36<00:10, 204.92batch/s]

Training batches on cuda:  77%|███████▋  | 7509/9708 [00:36<00:10, 203.25batch/s]

Training batches on cuda:  78%|███████▊  | 7530/9708 [00:36<00:10, 204.56batch/s]

Training batches on cuda:  78%|███████▊  | 7551/9708 [00:36<00:10, 202.35batch/s]

Training batches on cuda:  78%|███████▊  | 7572/9708 [00:36<00:10, 203.01batch/s]

Training batches on cuda:  78%|███████▊  | 7593/9708 [00:36<00:10, 200.28batch/s]

Training batches on cuda:  78%|███████▊  | 7614/9708 [00:37<00:10, 199.49batch/s]

Training batches on cuda:  79%|███████▊  | 7635/9708 [00:37<00:10, 201.68batch/s]

Training batches on cuda:  79%|███████▉  | 7656/9708 [00:37<00:10, 201.52batch/s]

Training batches on cuda:  79%|███████▉  | 7678/9708 [00:37<00:09, 206.39batch/s]

Training batches on cuda:  79%|███████▉  | 7699/9708 [00:37<00:10, 200.24batch/s]

Training batches on cuda:  80%|███████▉  | 7720/9708 [00:37<00:09, 202.18batch/s]

Training batches on cuda:  80%|███████▉  | 7741/9708 [00:37<00:09, 201.41batch/s]

Training batches on cuda:  80%|███████▉  | 7762/9708 [00:37<00:09, 200.60batch/s]

Training batches on cuda:  80%|████████  | 7783/9708 [00:37<00:09, 196.62batch/s]

Training batches on cuda:  80%|████████  | 7803/9708 [00:37<00:09, 194.76batch/s]

Training batches on cuda:  81%|████████  | 7823/9708 [00:38<00:09, 194.78batch/s]

Training batches on cuda:  81%|████████  | 7843/9708 [00:38<00:09, 193.52batch/s]

Training batches on cuda:  81%|████████  | 7864/9708 [00:38<00:09, 197.92batch/s]

Training batches on cuda:  81%|████████  | 7885/9708 [00:38<00:09, 199.35batch/s]

Training batches on cuda:  81%|████████▏ | 7905/9708 [00:38<00:09, 197.53batch/s]

Training batches on cuda:  82%|████████▏ | 7928/9708 [00:38<00:08, 205.20batch/s]

Training batches on cuda:  82%|████████▏ | 7950/9708 [00:38<00:08, 206.78batch/s]

Training batches on cuda:  82%|████████▏ | 7971/9708 [00:38<00:08, 202.07batch/s]

Training batches on cuda:  82%|████████▏ | 7992/9708 [00:38<00:08, 199.63batch/s]

Training batches on cuda:  83%|████████▎ | 8013/9708 [00:38<00:08, 201.23batch/s]

Training batches on cuda:  83%|████████▎ | 8035/9708 [00:39<00:08, 205.67batch/s]

Training batches on cuda:  83%|████████▎ | 8058/9708 [00:39<00:07, 211.79batch/s]

Training batches on cuda:  83%|████████▎ | 8080/9708 [00:39<00:07, 211.62batch/s]

Training batches on cuda:  83%|████████▎ | 8102/9708 [00:39<00:07, 213.49batch/s]

Training batches on cuda:  84%|████████▎ | 8125/9708 [00:39<00:07, 217.10batch/s]

Training batches on cuda:  84%|████████▍ | 8147/9708 [00:39<00:07, 210.40batch/s]

Training batches on cuda:  84%|████████▍ | 8170/9708 [00:39<00:07, 213.46batch/s]

Training batches on cuda:  84%|████████▍ | 8193/9708 [00:39<00:07, 216.15batch/s]

Training batches on cuda:  85%|████████▍ | 8216/9708 [00:39<00:06, 218.29batch/s]

Training batches on cuda:  85%|████████▍ | 8239/9708 [00:40<00:06, 219.87batch/s]

Training batches on cuda:  85%|████████▌ | 8262/9708 [00:40<00:06, 209.43batch/s]

Training batches on cuda:  85%|████████▌ | 8284/9708 [00:40<00:06, 203.57batch/s]

Training batches on cuda:  86%|████████▌ | 8305/9708 [00:40<00:07, 187.49batch/s]

Training batches on cuda:  86%|████████▌ | 8325/9708 [00:40<00:07, 189.66batch/s]

Training batches on cuda:  86%|████████▌ | 8346/9708 [00:40<00:07, 193.27batch/s]

Training batches on cuda:  86%|████████▌ | 8366/9708 [00:40<00:06, 192.02batch/s]

Training batches on cuda:  86%|████████▋ | 8386/9708 [00:40<00:06, 194.19batch/s]

Training batches on cuda:  87%|████████▋ | 8406/9708 [00:40<00:06, 195.17batch/s]

Training batches on cuda:  87%|████████▋ | 8427/9708 [00:41<00:06, 197.51batch/s]

Training batches on cuda:  87%|████████▋ | 8447/9708 [00:41<00:06, 196.48batch/s]

Training batches on cuda:  87%|████████▋ | 8467/9708 [00:41<00:06, 197.29batch/s]

Training batches on cuda:  87%|████████▋ | 8489/9708 [00:41<00:06, 201.11batch/s]

Training batches on cuda:  88%|████████▊ | 8510/9708 [00:41<00:06, 199.15batch/s]

Training batches on cuda:  88%|████████▊ | 8530/9708 [00:41<00:06, 194.42batch/s]

Training batches on cuda:  88%|████████▊ | 8550/9708 [00:41<00:06, 190.09batch/s]

Training batches on cuda:  88%|████████▊ | 8572/9708 [00:41<00:05, 196.24batch/s]

Training batches on cuda:  89%|████████▊ | 8592/9708 [00:41<00:05, 196.63batch/s]

Training batches on cuda:  89%|████████▊ | 8613/9708 [00:41<00:05, 199.69batch/s]

Training batches on cuda:  89%|████████▉ | 8634/9708 [00:42<00:05, 201.45batch/s]

Training batches on cuda:  89%|████████▉ | 8655/9708 [00:42<00:05, 198.91batch/s]

Training batches on cuda:  89%|████████▉ | 8675/9708 [00:42<00:05, 198.89batch/s]

Training batches on cuda:  90%|████████▉ | 8695/9708 [00:42<00:05, 197.98batch/s]

Training batches on cuda:  90%|████████▉ | 8715/9708 [00:42<00:05, 197.49batch/s]

Training batches on cuda:  90%|████████▉ | 8735/9708 [00:42<00:04, 195.81batch/s]

Training batches on cuda:  90%|█████████ | 8756/9708 [00:42<00:04, 198.56batch/s]

Training batches on cuda:  90%|█████████ | 8777/9708 [00:42<00:04, 201.76batch/s]

Training batches on cuda:  91%|█████████ | 8798/9708 [00:42<00:04, 202.76batch/s]

Training batches on cuda:  91%|█████████ | 8819/9708 [00:42<00:04, 203.06batch/s]

Training batches on cuda:  91%|█████████ | 8842/9708 [00:43<00:04, 209.38batch/s]

Training batches on cuda:  91%|█████████▏| 8863/9708 [00:43<00:04, 204.95batch/s]

Training batches on cuda:  92%|█████████▏| 8884/9708 [00:43<00:04, 202.84batch/s]

Training batches on cuda:  92%|█████████▏| 8905/9708 [00:43<00:04, 197.29batch/s]

Training batches on cuda:  92%|█████████▏| 8925/9708 [00:43<00:04, 188.93batch/s]

Training batches on cuda:  92%|█████████▏| 8945/9708 [00:43<00:04, 183.54batch/s]

Training batches on cuda:  92%|█████████▏| 8965/9708 [00:43<00:03, 187.11batch/s]

Training batches on cuda:  93%|█████████▎| 8985/9708 [00:43<00:03, 188.21batch/s]

Training batches on cuda:  93%|█████████▎| 9006/9708 [00:43<00:03, 193.41batch/s]

Training batches on cuda:  93%|█████████▎| 9026/9708 [00:44<00:03, 191.00batch/s]

Training batches on cuda:  93%|█████████▎| 9047/9708 [00:44<00:03, 196.08batch/s]

Training batches on cuda:  93%|█████████▎| 9068/9708 [00:44<00:03, 199.85batch/s]

Training batches on cuda:  94%|█████████▎| 9089/9708 [00:44<00:03, 198.75batch/s]

Training batches on cuda:  94%|█████████▍| 9109/9708 [00:44<00:03, 198.32batch/s]

Training batches on cuda:  94%|█████████▍| 9129/9708 [00:44<00:02, 197.67batch/s]

Training batches on cuda:  94%|█████████▍| 9151/9708 [00:44<00:02, 201.48batch/s]

Training batches on cuda:  94%|█████████▍| 9173/9708 [00:44<00:02, 205.70batch/s]

Training batches on cuda:  95%|█████████▍| 9196/9708 [00:44<00:02, 210.50batch/s]

Training batches on cuda:  95%|█████████▍| 9218/9708 [00:44<00:02, 209.93batch/s]

Training batches on cuda:  95%|█████████▌| 9240/9708 [00:45<00:02, 204.78batch/s]

Training batches on cuda:  95%|█████████▌| 9262/9708 [00:45<00:02, 207.92batch/s]

Training batches on cuda:  96%|█████████▌| 9283/9708 [00:45<00:02, 202.12batch/s]

Training batches on cuda:  96%|█████████▌| 9308/9708 [00:45<00:01, 213.27batch/s]

Training batches on cuda:  96%|█████████▌| 9336/9708 [00:45<00:01, 227.86batch/s]

Training batches on cuda:  96%|█████████▋| 9364/9708 [00:45<00:01, 239.99batch/s]

Training batches on cuda:  97%|█████████▋| 9389/9708 [00:45<00:01, 241.55batch/s]

Training batches on cuda:  97%|█████████▋| 9414/9708 [00:45<00:01, 233.78batch/s]

Training batches on cuda:  97%|█████████▋| 9438/9708 [00:45<00:01, 226.28batch/s]

Training batches on cuda:  97%|█████████▋| 9461/9708 [00:46<00:01, 222.64batch/s]

Training batches on cuda:  98%|█████████▊| 9485/9708 [00:46<00:00, 227.17batch/s]

Training batches on cuda:  98%|█████████▊| 9509/9708 [00:46<00:00, 228.43batch/s]

Training batches on cuda:  98%|█████████▊| 9532/9708 [00:46<00:00, 226.95batch/s]

Training batches on cuda:  98%|█████████▊| 9555/9708 [00:46<00:00, 219.45batch/s]

Training batches on cuda:  99%|█████████▊| 9578/9708 [00:46<00:00, 207.65batch/s]

Training batches on cuda:  99%|█████████▉| 9599/9708 [00:46<00:00, 207.38batch/s]

Training batches on cuda:  99%|█████████▉| 9620/9708 [00:46<00:00, 206.28batch/s]

Training batches on cuda:  99%|█████████▉| 9641/9708 [00:46<00:00, 207.33batch/s]

Training batches on cuda: 100%|█████████▉| 9662/9708 [00:47<00:00, 206.95batch/s]

Training batches on cuda: 100%|█████████▉| 9683/9708 [00:47<00:00, 206.87batch/s]

Training batches on cuda: 100%|█████████▉| 9705/9708 [00:47<00:00, 207.87batch/s]

                                                                                 
Training epochs on cuda:  70%|███████   | 7/10 [06:13<02:19, 46.44s/epoch, loss=0.0417, prev_loss=0.0417]
Training epochs on cuda:  80%|████████  | 8/10 [06:13<01:33, 46.77s/epoch, loss=0.0417, prev_loss=0.0417]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 11/9708 [00:00<01:30, 107.01batch/s]

Training batches on cuda:   0%|          | 33/9708 [00:00<01:16, 125.79batch/s]

Training batches on cuda:   1%|          | 55/9708 [00:00<01:07, 143.59batch/s]

Training batches on cuda:   1%|          | 77/9708 [00:00<01:00, 158.85batch/s]

Training batches on cuda:   1%|          | 97/9708 [00:00<00:57, 167.88batch/s]

Training batches on cuda:   1%|          | 118/9708 [00:00<00:54, 176.71batch/s]

Training batches on cuda:   1%|▏         | 140/9708 [00:00<00:51, 187.18batch/s]

Training batches on cuda:   2%|▏         | 160/9708 [00:00<00:50, 188.70batch/s]

Training batches on cuda:   2%|▏         | 179/9708 [00:00<00:50, 187.32batch/s]

Training batches on cuda:   2%|▏         | 198/9708 [00:01<00:52, 181.54batch/s]

Training batches on cuda:   2%|▏         | 221/9708 [00:01<00:49, 192.77batch/s]

Training batches on cuda:   3%|▎         | 244/9708 [00:01<00:47, 200.58batch/s]

Training batches on cuda:   3%|▎         | 267/9708 [00:01<00:45, 206.76batch/s]

Training batches on cuda:   3%|▎         | 288/9708 [00:01<00:45, 206.40batch/s]

Training batches on cuda:   3%|▎         | 309/9708 [00:01<00:45, 204.67batch/s]

Training batches on cuda:   3%|▎         | 331/9708 [00:01<00:45, 206.53batch/s]

Training batches on cuda:   4%|▎         | 353/9708 [00:01<00:44, 207.95batch/s]

Training batches on cuda:   4%|▍         | 376/9708 [00:01<00:43, 213.04batch/s]

Training batches on cuda:   4%|▍         | 398/9708 [00:01<00:43, 215.08batch/s]

Training batches on cuda:   4%|▍         | 420/9708 [00:02<00:43, 214.11batch/s]

Training batches on cuda:   5%|▍         | 443/9708 [00:02<00:42, 218.20batch/s]

Training batches on cuda:   5%|▍         | 465/9708 [00:02<00:42, 217.42batch/s]

Training batches on cuda:   5%|▌         | 488/9708 [00:02<00:41, 219.58batch/s]

Training batches on cuda:   5%|▌         | 510/9708 [00:02<00:42, 216.70batch/s]

Training batches on cuda:   5%|▌         | 533/9708 [00:02<00:42, 217.79batch/s]

Training batches on cuda:   6%|▌         | 555/9708 [00:02<00:45, 199.82batch/s]

Training batches on cuda:   6%|▌         | 576/9708 [00:02<00:49, 184.79batch/s]

Training batches on cuda:   6%|▌         | 595/9708 [00:02<00:50, 180.69batch/s]

Training batches on cuda:   6%|▋         | 614/9708 [00:03<00:49, 182.87batch/s]

Training batches on cuda:   7%|▋         | 636/9708 [00:03<00:47, 192.20batch/s]

Training batches on cuda:   7%|▋         | 659/9708 [00:03<00:45, 200.51batch/s]

Training batches on cuda:   7%|▋         | 681/9708 [00:03<00:44, 203.20batch/s]

Training batches on cuda:   7%|▋         | 702/9708 [00:03<00:48, 184.28batch/s]

Training batches on cuda:   7%|▋         | 721/9708 [00:03<00:49, 180.96batch/s]

Training batches on cuda:   8%|▊         | 740/9708 [00:03<00:52, 170.39batch/s]

Training batches on cuda:   8%|▊         | 761/9708 [00:03<00:49, 179.55batch/s]

Training batches on cuda:   8%|▊         | 782/9708 [00:03<00:47, 187.07batch/s]

Training batches on cuda:   8%|▊         | 802/9708 [00:04<00:47, 187.21batch/s]

Training batches on cuda:   8%|▊         | 822/9708 [00:04<00:47, 188.95batch/s]

Training batches on cuda:   9%|▊         | 842/9708 [00:04<00:47, 184.75batch/s]

Training batches on cuda:   9%|▉         | 862/9708 [00:04<00:47, 187.67batch/s]

Training batches on cuda:   9%|▉         | 881/9708 [00:04<00:47, 187.19batch/s]

Training batches on cuda:   9%|▉         | 901/9708 [00:04<00:46, 190.03batch/s]

Training batches on cuda:   9%|▉         | 922/9708 [00:04<00:45, 194.47batch/s]

Training batches on cuda:  10%|▉         | 944/9708 [00:04<00:43, 199.33batch/s]

Training batches on cuda:  10%|▉         | 966/9708 [00:04<00:43, 202.36batch/s]

Training batches on cuda:  10%|█         | 987/9708 [00:04<00:43, 199.68batch/s]

Training batches on cuda:  10%|█         | 1008/9708 [00:05<00:43, 198.47batch/s]

Training batches on cuda:  11%|█         | 1030/9708 [00:05<00:42, 203.12batch/s]

Training batches on cuda:  11%|█         | 1051/9708 [00:05<00:42, 202.43batch/s]

Training batches on cuda:  11%|█         | 1072/9708 [00:05<00:43, 199.16batch/s]

Training batches on cuda:  11%|█▏        | 1093/9708 [00:05<00:42, 202.22batch/s]

Training batches on cuda:  11%|█▏        | 1116/9708 [00:05<00:41, 209.35batch/s]

Training batches on cuda:  12%|█▏        | 1139/9708 [00:05<00:39, 215.14batch/s]

Training batches on cuda:  12%|█▏        | 1163/9708 [00:05<00:38, 219.53batch/s]

Training batches on cuda:  12%|█▏        | 1186/9708 [00:05<00:38, 222.47batch/s]

Training batches on cuda:  12%|█▏        | 1211/9708 [00:06<00:37, 227.60batch/s]

Training batches on cuda:  13%|█▎        | 1234/9708 [00:06<00:37, 225.79batch/s]

Training batches on cuda:  13%|█▎        | 1257/9708 [00:06<00:37, 223.46batch/s]

Training batches on cuda:  13%|█▎        | 1280/9708 [00:06<00:37, 223.13batch/s]

Training batches on cuda:  13%|█▎        | 1303/9708 [00:06<00:37, 223.60batch/s]

Training batches on cuda:  14%|█▎        | 1327/9708 [00:06<00:36, 227.37batch/s]

Training batches on cuda:  14%|█▍        | 1350/9708 [00:06<00:36, 226.56batch/s]

Training batches on cuda:  14%|█▍        | 1373/9708 [00:06<00:38, 219.08batch/s]

Training batches on cuda:  14%|█▍        | 1395/9708 [00:06<00:38, 214.40batch/s]

Training batches on cuda:  15%|█▍        | 1417/9708 [00:06<00:39, 208.90batch/s]

Training batches on cuda:  15%|█▍        | 1440/9708 [00:07<00:38, 212.78batch/s]

Training batches on cuda:  15%|█▌        | 1463/9708 [00:07<00:38, 215.15batch/s]

Training batches on cuda:  15%|█▌        | 1486/9708 [00:07<00:37, 216.72batch/s]

Training batches on cuda:  16%|█▌        | 1508/9708 [00:07<00:38, 214.52batch/s]

Training batches on cuda:  16%|█▌        | 1532/9708 [00:07<00:37, 219.90batch/s]

Training batches on cuda:  16%|█▌        | 1556/9708 [00:07<00:36, 224.78batch/s]

Training batches on cuda:  16%|█▋        | 1580/9708 [00:07<00:35, 228.58batch/s]

Training batches on cuda:  17%|█▋        | 1603/9708 [00:07<00:35, 227.86batch/s]

Training batches on cuda:  17%|█▋        | 1626/9708 [00:07<00:35, 227.77batch/s]

Training batches on cuda:  17%|█▋        | 1649/9708 [00:08<00:35, 224.87batch/s]

Training batches on cuda:  17%|█▋        | 1672/9708 [00:08<00:35, 224.20batch/s]

Training batches on cuda:  17%|█▋        | 1695/9708 [00:08<00:36, 220.13batch/s]

Training batches on cuda:  18%|█▊        | 1718/9708 [00:08<00:36, 218.81batch/s]

Training batches on cuda:  18%|█▊        | 1741/9708 [00:08<00:36, 219.76batch/s]

Training batches on cuda:  18%|█▊        | 1763/9708 [00:08<00:36, 217.62batch/s]

Training batches on cuda:  18%|█▊        | 1787/9708 [00:08<00:35, 221.58batch/s]

Training batches on cuda:  19%|█▊        | 1811/9708 [00:08<00:35, 225.15batch/s]

Training batches on cuda:  19%|█▉        | 1835/9708 [00:08<00:34, 226.52batch/s]

Training batches on cuda:  19%|█▉        | 1859/9708 [00:08<00:34, 228.54batch/s]

Training batches on cuda:  19%|█▉        | 1883/9708 [00:09<00:33, 230.26batch/s]

Training batches on cuda:  20%|█▉        | 1907/9708 [00:09<00:33, 230.07batch/s]

Training batches on cuda:  20%|█▉        | 1931/9708 [00:09<00:33, 230.64batch/s]

Training batches on cuda:  20%|██        | 1955/9708 [00:09<00:33, 230.57batch/s]

Training batches on cuda:  20%|██        | 1979/9708 [00:09<00:33, 228.89batch/s]

Training batches on cuda:  21%|██        | 2003/9708 [00:09<00:33, 229.63batch/s]

Training batches on cuda:  21%|██        | 2027/9708 [00:09<00:33, 230.78batch/s]

Training batches on cuda:  21%|██        | 2051/9708 [00:09<00:34, 225.18batch/s]

Training batches on cuda:  21%|██▏       | 2074/9708 [00:09<00:34, 223.04batch/s]

Training batches on cuda:  22%|██▏       | 2098/9708 [00:10<00:33, 225.01batch/s]

Training batches on cuda:  22%|██▏       | 2123/9708 [00:10<00:33, 229.55batch/s]

Training batches on cuda:  22%|██▏       | 2147/9708 [00:10<00:32, 230.11batch/s]

Training batches on cuda:  22%|██▏       | 2172/9708 [00:10<00:32, 235.17batch/s]

Training batches on cuda:  23%|██▎       | 2196/9708 [00:10<00:32, 233.59batch/s]

Training batches on cuda:  23%|██▎       | 2221/9708 [00:10<00:31, 237.80batch/s]

Training batches on cuda:  23%|██▎       | 2245/9708 [00:10<00:31, 234.50batch/s]

Training batches on cuda:  23%|██▎       | 2270/9708 [00:10<00:31, 238.08batch/s]

Training batches on cuda:  24%|██▎       | 2294/9708 [00:10<00:31, 235.44batch/s]

Training batches on cuda:  24%|██▍       | 2318/9708 [00:10<00:31, 235.18batch/s]

Training batches on cuda:  24%|██▍       | 2342/9708 [00:11<00:33, 220.28batch/s]

Training batches on cuda:  24%|██▍       | 2365/9708 [00:11<00:33, 220.98batch/s]

Training batches on cuda:  25%|██▍       | 2390/9708 [00:11<00:32, 227.56batch/s]

Training batches on cuda:  25%|██▍       | 2413/9708 [00:11<00:32, 223.97batch/s]

Training batches on cuda:  25%|██▌       | 2436/9708 [00:11<00:32, 222.02batch/s]

Training batches on cuda:  25%|██▌       | 2459/9708 [00:11<00:32, 221.34batch/s]

Training batches on cuda:  26%|██▌       | 2483/9708 [00:11<00:32, 223.72batch/s]

Training batches on cuda:  26%|██▌       | 2506/9708 [00:11<00:32, 222.82batch/s]

Training batches on cuda:  26%|██▌       | 2529/9708 [00:11<00:34, 207.22batch/s]

Training batches on cuda:  26%|██▋       | 2550/9708 [00:12<00:36, 195.09batch/s]

Training batches on cuda:  26%|██▋       | 2572/9708 [00:12<00:35, 199.12batch/s]

Training batches on cuda:  27%|██▋       | 2595/9708 [00:12<00:34, 205.93batch/s]

Training batches on cuda:  27%|██▋       | 2617/9708 [00:12<00:34, 207.54batch/s]

Training batches on cuda:  27%|██▋       | 2638/9708 [00:12<00:35, 196.92batch/s]

Training batches on cuda:  27%|██▋       | 2658/9708 [00:12<00:36, 195.07batch/s]

Training batches on cuda:  28%|██▊       | 2679/9708 [00:12<00:35, 197.58batch/s]

Training batches on cuda:  28%|██▊       | 2699/9708 [00:12<00:35, 196.53batch/s]

Training batches on cuda:  28%|██▊       | 2721/9708 [00:12<00:34, 200.16batch/s]

Training batches on cuda:  28%|██▊       | 2742/9708 [00:13<00:36, 191.98batch/s]

Training batches on cuda:  28%|██▊       | 2763/9708 [00:13<00:35, 196.19batch/s]

Training batches on cuda:  29%|██▊       | 2783/9708 [00:13<00:35, 195.43batch/s]

Training batches on cuda:  29%|██▉       | 2804/9708 [00:13<00:34, 198.76batch/s]

Training batches on cuda:  29%|██▉       | 2827/9708 [00:13<00:33, 205.04batch/s]

Training batches on cuda:  29%|██▉       | 2848/9708 [00:13<00:33, 204.69batch/s]

Training batches on cuda:  30%|██▉       | 2874/9708 [00:13<00:31, 217.20batch/s]

Training batches on cuda:  30%|██▉       | 2898/9708 [00:13<00:30, 221.54batch/s]

Training batches on cuda:  30%|███       | 2921/9708 [00:13<00:31, 215.93batch/s]

Training batches on cuda:  30%|███       | 2943/9708 [00:13<00:32, 211.21batch/s]

Training batches on cuda:  31%|███       | 2965/9708 [00:14<00:32, 208.48batch/s]

Training batches on cuda:  31%|███       | 2986/9708 [00:14<00:32, 205.79batch/s]

Training batches on cuda:  31%|███       | 3007/9708 [00:14<00:33, 200.92batch/s]

Training batches on cuda:  31%|███       | 3028/9708 [00:14<00:33, 197.40batch/s]

Training batches on cuda:  31%|███▏      | 3048/9708 [00:14<00:36, 183.68batch/s]

Training batches on cuda:  32%|███▏      | 3068/9708 [00:14<00:35, 186.27batch/s]

Training batches on cuda:  32%|███▏      | 3089/9708 [00:14<00:34, 192.57batch/s]

Training batches on cuda:  32%|███▏      | 3109/9708 [00:14<00:34, 192.04batch/s]

Training batches on cuda:  32%|███▏      | 3130/9708 [00:14<00:33, 196.36batch/s]

Training batches on cuda:  32%|███▏      | 3150/9708 [00:15<00:33, 195.37batch/s]

Training batches on cuda:  33%|███▎      | 3171/9708 [00:15<00:33, 197.80batch/s]

Training batches on cuda:  33%|███▎      | 3193/9708 [00:15<00:32, 202.85batch/s]

Training batches on cuda:  33%|███▎      | 3215/9708 [00:15<00:31, 205.81batch/s]

Training batches on cuda:  33%|███▎      | 3237/9708 [00:15<00:30, 209.70batch/s]

Training batches on cuda:  34%|███▎      | 3260/9708 [00:15<00:30, 214.14batch/s]

Training batches on cuda:  34%|███▍      | 3283/9708 [00:15<00:29, 217.02batch/s]

Training batches on cuda:  34%|███▍      | 3307/9708 [00:15<00:28, 222.42batch/s]

Training batches on cuda:  34%|███▍      | 3331/9708 [00:15<00:28, 225.31batch/s]

Training batches on cuda:  35%|███▍      | 3355/9708 [00:15<00:27, 228.21batch/s]

Training batches on cuda:  35%|███▍      | 3379/9708 [00:16<00:27, 229.32batch/s]

Training batches on cuda:  35%|███▌      | 3403/9708 [00:16<00:27, 231.22batch/s]

Training batches on cuda:  35%|███▌      | 3427/9708 [00:16<00:27, 228.63batch/s]

Training batches on cuda:  36%|███▌      | 3450/9708 [00:16<00:28, 216.99batch/s]

Training batches on cuda:  36%|███▌      | 3473/9708 [00:16<00:28, 219.99batch/s]

Training batches on cuda:  36%|███▌      | 3496/9708 [00:16<00:28, 221.49batch/s]

Training batches on cuda:  36%|███▋      | 3520/9708 [00:16<00:27, 224.63batch/s]

Training batches on cuda:  36%|███▋      | 3543/9708 [00:16<00:27, 226.17batch/s]

Training batches on cuda:  37%|███▋      | 3566/9708 [00:16<00:27, 224.46batch/s]

Training batches on cuda:  37%|███▋      | 3589/9708 [00:16<00:27, 219.90batch/s]

Training batches on cuda:  37%|███▋      | 3612/9708 [00:17<00:27, 218.29batch/s]

Training batches on cuda:  37%|███▋      | 3634/9708 [00:17<00:28, 214.72batch/s]

Training batches on cuda:  38%|███▊      | 3656/9708 [00:17<00:28, 214.35batch/s]

Training batches on cuda:  38%|███▊      | 3678/9708 [00:17<00:29, 207.54batch/s]

Training batches on cuda:  38%|███▊      | 3699/9708 [00:17<00:29, 206.54batch/s]

Training batches on cuda:  38%|███▊      | 3721/9708 [00:17<00:28, 210.31batch/s]

Training batches on cuda:  39%|███▊      | 3745/9708 [00:17<00:27, 216.18batch/s]

Training batches on cuda:  39%|███▉      | 3768/9708 [00:17<00:27, 217.92batch/s]

Training batches on cuda:  39%|███▉      | 3790/9708 [00:17<00:29, 198.05batch/s]

Training batches on cuda:  39%|███▉      | 3812/9708 [00:18<00:29, 202.52batch/s]

Training batches on cuda:  39%|███▉      | 3833/9708 [00:18<00:28, 203.59batch/s]

Training batches on cuda:  40%|███▉      | 3855/9708 [00:18<00:28, 205.46batch/s]

Training batches on cuda:  40%|███▉      | 3877/9708 [00:18<00:27, 208.96batch/s]

Training batches on cuda:  40%|████      | 3899/9708 [00:18<00:27, 208.80batch/s]

Training batches on cuda:  40%|████      | 3921/9708 [00:18<00:27, 209.87batch/s]

Training batches on cuda:  41%|████      | 3943/9708 [00:18<00:27, 208.50batch/s]

Training batches on cuda:  41%|████      | 3964/9708 [00:18<00:27, 207.73batch/s]

Training batches on cuda:  41%|████      | 3985/9708 [00:18<00:27, 204.45batch/s]

Training batches on cuda:  41%|████▏     | 4006/9708 [00:19<00:28, 200.71batch/s]

Training batches on cuda:  41%|████▏     | 4027/9708 [00:19<00:29, 195.05batch/s]

Training batches on cuda:  42%|████▏     | 4047/9708 [00:19<00:29, 191.31batch/s]

Training batches on cuda:  42%|████▏     | 4067/9708 [00:19<00:29, 192.99batch/s]

Training batches on cuda:  42%|████▏     | 4089/9708 [00:19<00:28, 199.40batch/s]

Training batches on cuda:  42%|████▏     | 4110/9708 [00:19<00:27, 202.10batch/s]

Training batches on cuda:  43%|████▎     | 4131/9708 [00:19<00:27, 201.05batch/s]

Training batches on cuda:  43%|████▎     | 4152/9708 [00:19<00:27, 200.66batch/s]

Training batches on cuda:  43%|████▎     | 4174/9708 [00:19<00:27, 203.65batch/s]

Training batches on cuda:  43%|████▎     | 4195/9708 [00:19<00:27, 201.70batch/s]

Training batches on cuda:  43%|████▎     | 4217/9708 [00:20<00:26, 205.52batch/s]

Training batches on cuda:  44%|████▎     | 4239/9708 [00:20<00:26, 207.06batch/s]

Training batches on cuda:  44%|████▍     | 4260/9708 [00:20<00:26, 202.32batch/s]

Training batches on cuda:  44%|████▍     | 4283/9708 [00:20<00:26, 206.90batch/s]

Training batches on cuda:  44%|████▍     | 4304/9708 [00:20<00:26, 207.39batch/s]

Training batches on cuda:  45%|████▍     | 4325/9708 [00:20<00:26, 204.95batch/s]

Training batches on cuda:  45%|████▍     | 4347/9708 [00:20<00:25, 207.56batch/s]

Training batches on cuda:  45%|████▍     | 4368/9708 [00:20<00:26, 201.14batch/s]

Training batches on cuda:  45%|████▌     | 4389/9708 [00:20<00:28, 189.69batch/s]

Training batches on cuda:  45%|████▌     | 4416/9708 [00:21<00:25, 207.71batch/s]

Training batches on cuda:  46%|████▌     | 4442/9708 [00:21<00:23, 219.87batch/s]

Training batches on cuda:  46%|████▌     | 4465/9708 [00:21<00:24, 217.76batch/s]

Training batches on cuda:  46%|████▌     | 4488/9708 [00:21<00:24, 213.96batch/s]

Training batches on cuda:  46%|████▋     | 4510/9708 [00:21<00:24, 213.59batch/s]

Training batches on cuda:  47%|████▋     | 4532/9708 [00:21<00:24, 208.64batch/s]

Training batches on cuda:  47%|████▋     | 4557/9708 [00:21<00:23, 218.56batch/s]

Training batches on cuda:  47%|████▋     | 4582/9708 [00:21<00:22, 226.77batch/s]

Training batches on cuda:  47%|████▋     | 4605/9708 [00:21<00:23, 219.33batch/s]

Training batches on cuda:  48%|████▊     | 4628/9708 [00:21<00:23, 214.50batch/s]

Training batches on cuda:  48%|████▊     | 4650/9708 [00:22<00:23, 210.80batch/s]

Training batches on cuda:  48%|████▊     | 4672/9708 [00:22<00:23, 210.22batch/s]

Training batches on cuda:  48%|████▊     | 4694/9708 [00:22<00:25, 197.46batch/s]

Training batches on cuda:  49%|████▊     | 4714/9708 [00:22<00:25, 196.40batch/s]

Training batches on cuda:  49%|████▉     | 4736/9708 [00:22<00:24, 202.00batch/s]

Training batches on cuda:  49%|████▉     | 4758/9708 [00:22<00:24, 206.09batch/s]

Training batches on cuda:  49%|████▉     | 4779/9708 [00:22<00:24, 204.32batch/s]

Training batches on cuda:  49%|████▉     | 4801/9708 [00:22<00:23, 207.05batch/s]

Training batches on cuda:  50%|████▉     | 4822/9708 [00:22<00:25, 192.62batch/s]

Training batches on cuda:  50%|████▉     | 4843/9708 [00:23<00:24, 197.28batch/s]

Training batches on cuda:  50%|█████     | 4866/9708 [00:23<00:23, 204.26batch/s]

Training batches on cuda:  50%|█████     | 4888/9708 [00:23<00:23, 207.22batch/s]

Training batches on cuda:  51%|█████     | 4910/9708 [00:23<00:22, 210.82batch/s]

Training batches on cuda:  51%|█████     | 4932/9708 [00:23<00:23, 206.01batch/s]

Training batches on cuda:  51%|█████     | 4953/9708 [00:23<00:23, 205.24batch/s]

Training batches on cuda:  51%|█████     | 4974/9708 [00:23<00:24, 192.20batch/s]

Training batches on cuda:  51%|█████▏    | 4994/9708 [00:23<00:24, 190.81batch/s]

Training batches on cuda:  52%|█████▏    | 5014/9708 [00:23<00:25, 181.84batch/s]

Training batches on cuda:  52%|█████▏    | 5033/9708 [00:24<00:26, 178.22batch/s]

Training batches on cuda:  52%|█████▏    | 5051/9708 [00:24<00:26, 178.69batch/s]

Training batches on cuda:  52%|█████▏    | 5071/9708 [00:24<00:25, 184.21batch/s]

Training batches on cuda:  52%|█████▏    | 5092/9708 [00:24<00:24, 189.01batch/s]

Training batches on cuda:  53%|█████▎    | 5112/9708 [00:24<00:24, 190.20batch/s]

Training batches on cuda:  53%|█████▎    | 5136/9708 [00:24<00:22, 200.93batch/s]

Training batches on cuda:  53%|█████▎    | 5160/9708 [00:24<00:21, 209.75batch/s]

Training batches on cuda:  53%|█████▎    | 5184/9708 [00:24<00:20, 216.08batch/s]

Training batches on cuda:  54%|█████▎    | 5209/9708 [00:24<00:20, 224.48batch/s]

Training batches on cuda:  54%|█████▍    | 5233/9708 [00:24<00:19, 227.13batch/s]

Training batches on cuda:  54%|█████▍    | 5256/9708 [00:25<00:20, 212.22batch/s]

Training batches on cuda:  54%|█████▍    | 5278/9708 [00:25<00:21, 201.70batch/s]

Training batches on cuda:  55%|█████▍    | 5299/9708 [00:25<00:23, 188.59batch/s]

Training batches on cuda:  55%|█████▍    | 5319/9708 [00:25<00:24, 181.31batch/s]

Training batches on cuda:  55%|█████▍    | 5338/9708 [00:25<00:23, 182.53batch/s]

Training batches on cuda:  55%|█████▌    | 5357/9708 [00:25<00:23, 183.27batch/s]

Training batches on cuda:  55%|█████▌    | 5377/9708 [00:25<00:23, 186.52batch/s]

Training batches on cuda:  56%|█████▌    | 5396/9708 [00:25<00:23, 187.47batch/s]

Training batches on cuda:  56%|█████▌    | 5417/9708 [00:25<00:22, 192.67batch/s]

Training batches on cuda:  56%|█████▌    | 5438/9708 [00:26<00:21, 195.35batch/s]

Training batches on cuda:  56%|█████▌    | 5458/9708 [00:26<00:23, 184.32batch/s]

Training batches on cuda:  56%|█████▋    | 5477/9708 [00:26<00:22, 185.02batch/s]

Training batches on cuda:  57%|█████▋    | 5496/9708 [00:26<00:22, 185.56batch/s]

Training batches on cuda:  57%|█████▋    | 5517/9708 [00:26<00:21, 191.68batch/s]

Training batches on cuda:  57%|█████▋    | 5538/9708 [00:26<00:21, 195.58batch/s]

Training batches on cuda:  57%|█████▋    | 5560/9708 [00:26<00:20, 201.14batch/s]

Training batches on cuda:  57%|█████▋    | 5582/9708 [00:26<00:20, 204.68batch/s]

Training batches on cuda:  58%|█████▊    | 5603/9708 [00:26<00:20, 204.96batch/s]

Training batches on cuda:  58%|█████▊    | 5624/9708 [00:27<00:20, 202.57batch/s]

Training batches on cuda:  58%|█████▊    | 5645/9708 [00:27<00:20, 194.71batch/s]

Training batches on cuda:  58%|█████▊    | 5665/9708 [00:27<00:21, 189.64batch/s]

Training batches on cuda:  59%|█████▊    | 5685/9708 [00:27<00:21, 187.77batch/s]

Training batches on cuda:  59%|█████▉    | 5704/9708 [00:27<00:21, 183.32batch/s]

Training batches on cuda:  59%|█████▉    | 5724/9708 [00:27<00:21, 187.86batch/s]

Training batches on cuda:  59%|█████▉    | 5745/9708 [00:27<00:20, 192.56batch/s]

Training batches on cuda:  59%|█████▉    | 5766/9708 [00:27<00:20, 196.30batch/s]

Training batches on cuda:  60%|█████▉    | 5787/9708 [00:27<00:19, 198.34batch/s]

Training batches on cuda:  60%|█████▉    | 5807/9708 [00:28<00:22, 175.04batch/s]

Training batches on cuda:  60%|██████    | 5826/9708 [00:28<00:21, 177.28batch/s]

Training batches on cuda:  60%|██████    | 5845/9708 [00:28<00:21, 176.27batch/s]

Training batches on cuda:  60%|██████    | 5865/9708 [00:28<00:21, 181.66batch/s]

Training batches on cuda:  61%|██████    | 5887/9708 [00:28<00:20, 190.03batch/s]

Training batches on cuda:  61%|██████    | 5908/9708 [00:28<00:19, 195.25batch/s]

Training batches on cuda:  61%|██████    | 5928/9708 [00:28<00:19, 194.29batch/s]

Training batches on cuda:  61%|██████▏   | 5948/9708 [00:28<00:19, 189.88batch/s]

Training batches on cuda:  61%|██████▏   | 5968/9708 [00:28<00:19, 190.79batch/s]

Training batches on cuda:  62%|██████▏   | 5988/9708 [00:28<00:19, 193.20batch/s]

Training batches on cuda:  62%|██████▏   | 6010/9708 [00:29<00:18, 199.13batch/s]

Training batches on cuda:  62%|██████▏   | 6033/9708 [00:29<00:17, 206.96batch/s]

Training batches on cuda:  62%|██████▏   | 6054/9708 [00:29<00:17, 205.85batch/s]

Training batches on cuda:  63%|██████▎   | 6075/9708 [00:29<00:17, 202.09batch/s]

Training batches on cuda:  63%|██████▎   | 6098/9708 [00:29<00:17, 207.47batch/s]

Training batches on cuda:  63%|██████▎   | 6119/9708 [00:29<00:17, 206.47batch/s]

Training batches on cuda:  63%|██████▎   | 6140/9708 [00:29<00:17, 205.27batch/s]

Training batches on cuda:  63%|██████▎   | 6161/9708 [00:29<00:17, 199.28batch/s]

Training batches on cuda:  64%|██████▎   | 6182/9708 [00:29<00:18, 189.49batch/s]

Training batches on cuda:  64%|██████▍   | 6202/9708 [00:30<00:19, 181.86batch/s]

Training batches on cuda:  64%|██████▍   | 6222/9708 [00:30<00:18, 186.51batch/s]

Training batches on cuda:  64%|██████▍   | 6244/9708 [00:30<00:17, 193.55batch/s]

Training batches on cuda:  65%|██████▍   | 6268/9708 [00:30<00:16, 204.94batch/s]

Training batches on cuda:  65%|██████▍   | 6293/9708 [00:30<00:15, 215.89batch/s]

Training batches on cuda:  65%|██████▌   | 6315/9708 [00:30<00:15, 216.48batch/s]

Training batches on cuda:  65%|██████▌   | 6339/9708 [00:30<00:15, 220.97batch/s]

Training batches on cuda:  66%|██████▌   | 6362/9708 [00:30<00:15, 222.86batch/s]

Training batches on cuda:  66%|██████▌   | 6386/9708 [00:30<00:14, 226.05batch/s]

Training batches on cuda:  66%|██████▌   | 6409/9708 [00:30<00:14, 221.48batch/s]

Training batches on cuda:  66%|██████▋   | 6432/9708 [00:31<00:15, 211.58batch/s]

Training batches on cuda:  66%|██████▋   | 6454/9708 [00:31<00:15, 210.78batch/s]

Training batches on cuda:  67%|██████▋   | 6476/9708 [00:31<00:16, 201.75batch/s]

Training batches on cuda:  67%|██████▋   | 6500/9708 [00:31<00:15, 211.87batch/s]

Training batches on cuda:  67%|██████▋   | 6523/9708 [00:31<00:14, 215.45batch/s]

Training batches on cuda:  67%|██████▋   | 6546/9708 [00:31<00:14, 217.02batch/s]

Training batches on cuda:  68%|██████▊   | 6568/9708 [00:31<00:15, 204.16batch/s]

Training batches on cuda:  68%|██████▊   | 6589/9708 [00:31<00:15, 195.05batch/s]

Training batches on cuda:  68%|██████▊   | 6609/9708 [00:31<00:15, 194.16batch/s]

Training batches on cuda:  68%|██████▊   | 6629/9708 [00:32<00:15, 193.95batch/s]

Training batches on cuda:  68%|██████▊   | 6649/9708 [00:32<00:16, 189.64batch/s]

Training batches on cuda:  69%|██████▊   | 6669/9708 [00:32<00:16, 189.45batch/s]

Training batches on cuda:  69%|██████▉   | 6691/9708 [00:32<00:15, 197.21batch/s]

Training batches on cuda:  69%|██████▉   | 6713/9708 [00:32<00:14, 201.00batch/s]

Training batches on cuda:  69%|██████▉   | 6735/9708 [00:32<00:14, 204.31batch/s]

Training batches on cuda:  70%|██████▉   | 6756/9708 [00:32<00:15, 188.43batch/s]

Training batches on cuda:  70%|██████▉   | 6777/9708 [00:32<00:15, 194.05batch/s]

Training batches on cuda:  70%|███████   | 6798/9708 [00:32<00:14, 196.15batch/s]

Training batches on cuda:  70%|███████   | 6818/9708 [00:33<00:16, 178.05batch/s]

Training batches on cuda:  70%|███████   | 6839/9708 [00:33<00:15, 184.78batch/s]

Training batches on cuda:  71%|███████   | 6858/9708 [00:33<00:15, 185.21batch/s]

Training batches on cuda:  71%|███████   | 6877/9708 [00:33<00:15, 186.33batch/s]

Training batches on cuda:  71%|███████   | 6897/9708 [00:33<00:14, 188.91batch/s]

Training batches on cuda:  71%|███████▏  | 6917/9708 [00:33<00:14, 188.35batch/s]

Training batches on cuda:  71%|███████▏  | 6937/9708 [00:33<00:14, 188.97batch/s]

Training batches on cuda:  72%|███████▏  | 6956/9708 [00:33<00:14, 188.67batch/s]

Training batches on cuda:  72%|███████▏  | 6975/9708 [00:33<00:14, 185.90batch/s]

Training batches on cuda:  72%|███████▏  | 6994/9708 [00:34<00:14, 185.54batch/s]

Training batches on cuda:  72%|███████▏  | 7013/9708 [00:34<00:14, 184.85batch/s]

Training batches on cuda:  72%|███████▏  | 7036/9708 [00:34<00:13, 195.47batch/s]

Training batches on cuda:  73%|███████▎  | 7061/9708 [00:34<00:12, 207.60batch/s]

Training batches on cuda:  73%|███████▎  | 7083/9708 [00:34<00:12, 209.40batch/s]

Training batches on cuda:  73%|███████▎  | 7105/9708 [00:34<00:12, 208.16batch/s]

Training batches on cuda:  73%|███████▎  | 7127/9708 [00:34<00:12, 209.23batch/s]

Training batches on cuda:  74%|███████▎  | 7149/9708 [00:34<00:12, 209.73batch/s]

Training batches on cuda:  74%|███████▍  | 7171/9708 [00:34<00:11, 211.55batch/s]

Training batches on cuda:  74%|███████▍  | 7193/9708 [00:34<00:11, 211.60batch/s]

Training batches on cuda:  74%|███████▍  | 7215/9708 [00:35<00:11, 211.86batch/s]

Training batches on cuda:  75%|███████▍  | 7237/9708 [00:35<00:11, 209.23batch/s]

Training batches on cuda:  75%|███████▍  | 7259/9708 [00:35<00:11, 210.80batch/s]

Training batches on cuda:  75%|███████▌  | 7281/9708 [00:35<00:11, 210.80batch/s]

Training batches on cuda:  75%|███████▌  | 7303/9708 [00:35<00:11, 211.29batch/s]

Training batches on cuda:  75%|███████▌  | 7326/9708 [00:35<00:11, 214.92batch/s]

Training batches on cuda:  76%|███████▌  | 7349/9708 [00:35<00:10, 218.91batch/s]

Training batches on cuda:  76%|███████▌  | 7371/9708 [00:35<00:10, 214.31batch/s]

Training batches on cuda:  76%|███████▌  | 7394/9708 [00:35<00:10, 216.49batch/s]

Training batches on cuda:  76%|███████▋  | 7416/9708 [00:35<00:10, 215.25batch/s]

Training batches on cuda:  77%|███████▋  | 7438/9708 [00:36<00:10, 211.67batch/s]

Training batches on cuda:  77%|███████▋  | 7460/9708 [00:36<00:10, 212.97batch/s]

Training batches on cuda:  77%|███████▋  | 7483/9708 [00:36<00:10, 215.12batch/s]

Training batches on cuda:  77%|███████▋  | 7505/9708 [00:36<00:10, 213.39batch/s]

Training batches on cuda:  78%|███████▊  | 7528/9708 [00:36<00:10, 215.90batch/s]

Training batches on cuda:  78%|███████▊  | 7550/9708 [00:36<00:10, 213.38batch/s]

Training batches on cuda:  78%|███████▊  | 7572/9708 [00:36<00:10, 206.86batch/s]

Training batches on cuda:  78%|███████▊  | 7593/9708 [00:36<00:10, 199.47batch/s]

Training batches on cuda:  78%|███████▊  | 7614/9708 [00:36<00:10, 201.27batch/s]

Training batches on cuda:  79%|███████▊  | 7635/9708 [00:37<00:10, 202.80batch/s]

Training batches on cuda:  79%|███████▉  | 7658/9708 [00:37<00:09, 209.66batch/s]

Training batches on cuda:  79%|███████▉  | 7681/9708 [00:37<00:09, 212.84batch/s]

Training batches on cuda:  79%|███████▉  | 7703/9708 [00:37<00:09, 213.06batch/s]

Training batches on cuda:  80%|███████▉  | 7725/9708 [00:37<00:09, 213.75batch/s]

Training batches on cuda:  80%|███████▉  | 7747/9708 [00:37<00:09, 211.84batch/s]

Training batches on cuda:  80%|████████  | 7769/9708 [00:37<00:09, 210.95batch/s]

Training batches on cuda:  80%|████████  | 7791/9708 [00:37<00:09, 210.36batch/s]

Training batches on cuda:  80%|████████  | 7813/9708 [00:37<00:09, 208.61batch/s]

Training batches on cuda:  81%|████████  | 7834/9708 [00:38<00:09, 203.09batch/s]

Training batches on cuda:  81%|████████  | 7855/9708 [00:38<00:09, 198.92batch/s]

Training batches on cuda:  81%|████████  | 7876/9708 [00:38<00:09, 201.16batch/s]

Training batches on cuda:  81%|████████▏ | 7898/9708 [00:38<00:08, 204.73batch/s]

Training batches on cuda:  82%|████████▏ | 7919/9708 [00:38<00:08, 205.99batch/s]

Training batches on cuda:  82%|████████▏ | 7940/9708 [00:38<00:08, 207.11batch/s]

Training batches on cuda:  82%|████████▏ | 7961/9708 [00:38<00:08, 207.41batch/s]

Training batches on cuda:  82%|████████▏ | 7984/9708 [00:38<00:08, 211.58batch/s]

Training batches on cuda:  82%|████████▏ | 8007/9708 [00:38<00:07, 214.06batch/s]

Training batches on cuda:  83%|████████▎ | 8029/9708 [00:38<00:07, 214.44batch/s]

Training batches on cuda:  83%|████████▎ | 8052/9708 [00:39<00:07, 216.39batch/s]

Training batches on cuda:  83%|████████▎ | 8074/9708 [00:39<00:07, 217.35batch/s]

Training batches on cuda:  83%|████████▎ | 8098/9708 [00:39<00:07, 222.63batch/s]

Training batches on cuda:  84%|████████▎ | 8125/9708 [00:39<00:06, 233.26batch/s]

Training batches on cuda:  84%|████████▍ | 8152/9708 [00:39<00:06, 241.88batch/s]

Training batches on cuda:  84%|████████▍ | 8177/9708 [00:39<00:06, 236.49batch/s]

Training batches on cuda:  84%|████████▍ | 8201/9708 [00:39<00:06, 226.51batch/s]

Training batches on cuda:  85%|████████▍ | 8224/9708 [00:39<00:06, 223.79batch/s]

Training batches on cuda:  85%|████████▍ | 8247/9708 [00:39<00:06, 215.63batch/s]

Training batches on cuda:  85%|████████▌ | 8269/9708 [00:39<00:06, 211.98batch/s]

Training batches on cuda:  85%|████████▌ | 8291/9708 [00:40<00:06, 206.29batch/s]

Training batches on cuda:  86%|████████▌ | 8312/9708 [00:40<00:06, 205.08batch/s]

Training batches on cuda:  86%|████████▌ | 8333/9708 [00:40<00:06, 202.64batch/s]

Training batches on cuda:  86%|████████▌ | 8354/9708 [00:40<00:06, 202.40batch/s]

Training batches on cuda:  86%|████████▋ | 8377/9708 [00:40<00:06, 209.17batch/s]

Training batches on cuda:  87%|████████▋ | 8401/9708 [00:40<00:06, 217.44batch/s]

Training batches on cuda:  87%|████████▋ | 8423/9708 [00:40<00:05, 217.70batch/s]

Training batches on cuda:  87%|████████▋ | 8446/9708 [00:40<00:05, 220.43batch/s]

Training batches on cuda:  87%|████████▋ | 8469/9708 [00:40<00:05, 219.94batch/s]

Training batches on cuda:  87%|████████▋ | 8492/9708 [00:41<00:05, 218.19batch/s]

Training batches on cuda:  88%|████████▊ | 8514/9708 [00:41<00:05, 216.88batch/s]

Training batches on cuda:  88%|████████▊ | 8536/9708 [00:41<00:05, 215.89batch/s]

Training batches on cuda:  88%|████████▊ | 8558/9708 [00:41<00:05, 213.06batch/s]

Training batches on cuda:  88%|████████▊ | 8580/9708 [00:41<00:05, 208.09batch/s]

Training batches on cuda:  89%|████████▊ | 8601/9708 [00:41<00:05, 199.65batch/s]

Training batches on cuda:  89%|████████▉ | 8622/9708 [00:41<00:05, 197.39batch/s]

Training batches on cuda:  89%|████████▉ | 8642/9708 [00:41<00:05, 187.14batch/s]

Training batches on cuda:  89%|████████▉ | 8661/9708 [00:41<00:05, 181.91batch/s]

Training batches on cuda:  89%|████████▉ | 8682/9708 [00:42<00:05, 186.85batch/s]

Training batches on cuda:  90%|████████▉ | 8703/9708 [00:42<00:05, 191.28batch/s]

Training batches on cuda:  90%|████████▉ | 8723/9708 [00:42<00:05, 191.64batch/s]

Training batches on cuda:  90%|█████████ | 8743/9708 [00:42<00:05, 187.93batch/s]

Training batches on cuda:  90%|█████████ | 8762/9708 [00:42<00:05, 187.80batch/s]

Training batches on cuda:  90%|█████████ | 8783/9708 [00:42<00:04, 192.12batch/s]

Training batches on cuda:  91%|█████████ | 8804/9708 [00:42<00:04, 195.66batch/s]

Training batches on cuda:  91%|█████████ | 8826/9708 [00:42<00:04, 200.54batch/s]

Training batches on cuda:  91%|█████████ | 8847/9708 [00:42<00:04, 202.68batch/s]

Training batches on cuda:  91%|█████████▏| 8869/9708 [00:42<00:04, 205.04batch/s]

Training batches on cuda:  92%|█████████▏| 8891/9708 [00:43<00:03, 206.95batch/s]

Training batches on cuda:  92%|█████████▏| 8913/9708 [00:43<00:03, 208.55batch/s]

Training batches on cuda:  92%|█████████▏| 8935/9708 [00:43<00:03, 210.15batch/s]

Training batches on cuda:  92%|█████████▏| 8957/9708 [00:43<00:03, 210.80batch/s]

Training batches on cuda:  92%|█████████▏| 8979/9708 [00:43<00:03, 209.38batch/s]

Training batches on cuda:  93%|█████████▎| 9001/9708 [00:43<00:03, 209.62batch/s]

Training batches on cuda:  93%|█████████▎| 9022/9708 [00:43<00:03, 209.13batch/s]

Training batches on cuda:  93%|█████████▎| 9044/9708 [00:43<00:03, 210.27batch/s]

Training batches on cuda:  93%|█████████▎| 9067/9708 [00:43<00:02, 215.46batch/s]

Training batches on cuda:  94%|█████████▎| 9090/9708 [00:43<00:02, 217.50batch/s]

Training batches on cuda:  94%|█████████▍| 9114/9708 [00:44<00:02, 221.07batch/s]

Training batches on cuda:  94%|█████████▍| 9137/9708 [00:44<00:02, 214.45batch/s]

Training batches on cuda:  94%|█████████▍| 9159/9708 [00:44<00:02, 213.64batch/s]

Training batches on cuda:  95%|█████████▍| 9181/9708 [00:44<00:02, 213.56batch/s]

Training batches on cuda:  95%|█████████▍| 9203/9708 [00:44<00:02, 211.91batch/s]

Training batches on cuda:  95%|█████████▌| 9225/9708 [00:44<00:02, 208.65batch/s]

Training batches on cuda:  95%|█████████▌| 9247/9708 [00:44<00:02, 209.42batch/s]

Training batches on cuda:  95%|█████████▌| 9268/9708 [00:44<00:02, 205.02batch/s]

Training batches on cuda:  96%|█████████▌| 9289/9708 [00:44<00:02, 201.75batch/s]

Training batches on cuda:  96%|█████████▌| 9310/9708 [00:45<00:01, 201.74batch/s]

Training batches on cuda:  96%|█████████▌| 9331/9708 [00:45<00:01, 202.51batch/s]

Training batches on cuda:  96%|█████████▋| 9352/9708 [00:45<00:01, 191.32batch/s]

Training batches on cuda:  97%|█████████▋| 9372/9708 [00:45<00:01, 192.02batch/s]

Training batches on cuda:  97%|█████████▋| 9392/9708 [00:45<00:01, 194.20batch/s]

Training batches on cuda:  97%|█████████▋| 9412/9708 [00:45<00:01, 188.04batch/s]

Training batches on cuda:  97%|█████████▋| 9434/9708 [00:45<00:01, 194.97batch/s]

Training batches on cuda:  97%|█████████▋| 9454/9708 [00:45<00:01, 192.44batch/s]

Training batches on cuda:  98%|█████████▊| 9475/9708 [00:45<00:01, 197.30batch/s]

Training batches on cuda:  98%|█████████▊| 9497/9708 [00:46<00:01, 201.42batch/s]

Training batches on cuda:  98%|█████████▊| 9518/9708 [00:46<00:00, 202.31batch/s]

Training batches on cuda:  98%|█████████▊| 9540/9708 [00:46<00:00, 206.02batch/s]

Training batches on cuda:  99%|█████████▊| 9563/9708 [00:46<00:00, 211.24batch/s]

Training batches on cuda:  99%|█████████▊| 9585/9708 [00:46<00:00, 212.21batch/s]

Training batches on cuda:  99%|█████████▉| 9609/9708 [00:46<00:00, 217.45batch/s]

Training batches on cuda:  99%|█████████▉| 9631/9708 [00:46<00:00, 215.18batch/s]

Training batches on cuda:  99%|█████████▉| 9653/9708 [00:46<00:00, 212.39batch/s]

Training batches on cuda: 100%|█████████▉| 9675/9708 [00:46<00:00, 212.60batch/s]

Training batches on cuda: 100%|█████████▉| 9697/9708 [00:46<00:00, 211.28batch/s]

                                                                                 
Training epochs on cuda:  80%|████████  | 8/10 [07:00<01:33, 46.77s/epoch, loss=0.0417, prev_loss=0.0417]
Training epochs on cuda:  90%|█████████ | 9/10 [07:00<00:46, 46.91s/epoch, loss=0.0417, prev_loss=0.0417]

Training batches on cuda:   0%|          | 0/9708 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 9/9708 [00:00<01:49, 88.40batch/s]

Training batches on cuda:   0%|          | 30/9708 [00:00<01:30, 106.96batch/s]

Training batches on cuda:   1%|          | 52/9708 [00:00<01:16, 126.18batch/s]

Training batches on cuda:   1%|          | 74/9708 [00:00<01:06, 143.91batch/s]

Training batches on cuda:   1%|          | 97/9708 [00:00<00:59, 160.86batch/s]

Training batches on cuda:   1%|          | 119/9708 [00:00<00:55, 173.37batch/s]

Training batches on cuda:   1%|▏         | 143/9708 [00:00<00:50, 188.78batch/s]

Training batches on cuda:   2%|▏         | 167/9708 [00:00<00:47, 200.81batch/s]

Training batches on cuda:   2%|▏         | 193/9708 [00:00<00:44, 213.78batch/s]

Training batches on cuda:   2%|▏         | 216/9708 [00:01<00:43, 218.34batch/s]

Training batches on cuda:   2%|▏         | 239/9708 [00:01<00:42, 221.00batch/s]

Training batches on cuda:   3%|▎         | 262/9708 [00:01<00:42, 221.98batch/s]

Training batches on cuda:   3%|▎         | 285/9708 [00:01<00:42, 221.13batch/s]

Training batches on cuda:   3%|▎         | 308/9708 [00:01<00:42, 220.35batch/s]

Training batches on cuda:   3%|▎         | 331/9708 [00:01<00:42, 219.98batch/s]

Training batches on cuda:   4%|▎         | 354/9708 [00:01<00:43, 217.39batch/s]

Training batches on cuda:   4%|▍         | 376/9708 [00:01<00:46, 202.68batch/s]

Training batches on cuda:   4%|▍         | 399/9708 [00:01<00:44, 208.06batch/s]

Training batches on cuda:   4%|▍         | 422/9708 [00:01<00:43, 211.83batch/s]

Training batches on cuda:   5%|▍         | 446/9708 [00:02<00:42, 218.00batch/s]

Training batches on cuda:   5%|▍         | 469/9708 [00:02<00:42, 219.38batch/s]

Training batches on cuda:   5%|▌         | 492/9708 [00:02<00:41, 220.43batch/s]

Training batches on cuda:   5%|▌         | 515/9708 [00:02<00:41, 222.77batch/s]

Training batches on cuda:   6%|▌         | 538/9708 [00:02<00:40, 224.71batch/s]

Training batches on cuda:   6%|▌         | 561/9708 [00:02<00:40, 226.09batch/s]

Training batches on cuda:   6%|▌         | 584/9708 [00:02<00:40, 226.00batch/s]

Training batches on cuda:   6%|▋         | 607/9708 [00:02<00:40, 226.15batch/s]

Training batches on cuda:   6%|▋         | 630/9708 [00:02<00:40, 226.02batch/s]

Training batches on cuda:   7%|▋         | 653/9708 [00:02<00:40, 225.89batch/s]

Training batches on cuda:   7%|▋         | 676/9708 [00:03<00:39, 226.24batch/s]

Training batches on cuda:   7%|▋         | 699/9708 [00:03<00:39, 225.79batch/s]

Training batches on cuda:   7%|▋         | 723/9708 [00:03<00:39, 227.92batch/s]

Training batches on cuda:   8%|▊         | 746/9708 [00:03<00:39, 225.65batch/s]

Training batches on cuda:   8%|▊         | 769/9708 [00:03<00:39, 226.66batch/s]

Training batches on cuda:   8%|▊         | 792/9708 [00:03<00:40, 222.87batch/s]

Training batches on cuda:   8%|▊         | 816/9708 [00:03<00:39, 226.06batch/s]

Training batches on cuda:   9%|▊         | 839/9708 [00:03<00:39, 226.28batch/s]

Training batches on cuda:   9%|▉         | 863/9708 [00:03<00:38, 228.56batch/s]

Training batches on cuda:   9%|▉         | 886/9708 [00:04<00:38, 228.14batch/s]

Training batches on cuda:   9%|▉         | 910/9708 [00:04<00:38, 230.22batch/s]

Training batches on cuda:  10%|▉         | 934/9708 [00:04<00:38, 230.55batch/s]

Training batches on cuda:  10%|▉         | 958/9708 [00:04<00:38, 228.50batch/s]

Training batches on cuda:  10%|█         | 982/9708 [00:04<00:37, 229.67batch/s]

Training batches on cuda:  10%|█         | 1005/9708 [00:04<00:37, 229.65batch/s]

Training batches on cuda:  11%|█         | 1028/9708 [00:04<00:37, 229.58batch/s]

Training batches on cuda:  11%|█         | 1051/9708 [00:04<00:38, 223.13batch/s]

Training batches on cuda:  11%|█         | 1074/9708 [00:04<00:39, 217.42batch/s]

Training batches on cuda:  11%|█▏        | 1096/9708 [00:04<00:39, 217.11batch/s]

Training batches on cuda:  12%|█▏        | 1118/9708 [00:05<00:39, 216.38batch/s]

Training batches on cuda:  12%|█▏        | 1140/9708 [00:05<00:40, 213.30batch/s]

Training batches on cuda:  12%|█▏        | 1162/9708 [00:05<00:40, 209.44batch/s]

Training batches on cuda:  12%|█▏        | 1183/9708 [00:05<00:41, 206.39batch/s]

Training batches on cuda:  12%|█▏        | 1204/9708 [00:05<00:42, 201.05batch/s]

Training batches on cuda:  13%|█▎        | 1225/9708 [00:05<00:41, 202.03batch/s]

Training batches on cuda:  13%|█▎        | 1247/9708 [00:05<00:40, 206.64batch/s]

Training batches on cuda:  13%|█▎        | 1269/9708 [00:05<00:40, 208.57batch/s]

Training batches on cuda:  13%|█▎        | 1290/9708 [00:05<00:40, 206.38batch/s]

Training batches on cuda:  14%|█▎        | 1312/9708 [00:06<00:40, 208.53batch/s]

Training batches on cuda:  14%|█▎        | 1334/9708 [00:06<00:39, 211.60batch/s]

Training batches on cuda:  14%|█▍        | 1357/9708 [00:06<00:38, 216.44batch/s]

Training batches on cuda:  14%|█▍        | 1380/9708 [00:06<00:37, 219.67batch/s]

Training batches on cuda:  14%|█▍        | 1403/9708 [00:06<00:37, 221.60batch/s]

Training batches on cuda:  15%|█▍        | 1427/9708 [00:06<00:36, 224.16batch/s]

Training batches on cuda:  15%|█▍        | 1451/9708 [00:06<00:36, 226.48batch/s]

Training batches on cuda:  15%|█▌        | 1474/9708 [00:06<00:36, 226.75batch/s]

Training batches on cuda:  15%|█▌        | 1497/9708 [00:06<00:36, 224.01batch/s]

Training batches on cuda:  16%|█▌        | 1521/9708 [00:06<00:36, 226.68batch/s]

Training batches on cuda:  16%|█▌        | 1545/9708 [00:07<00:35, 228.05batch/s]

Training batches on cuda:  16%|█▌        | 1569/9708 [00:07<00:35, 228.73batch/s]

Training batches on cuda:  16%|█▋        | 1593/9708 [00:07<00:35, 230.22batch/s]

Training batches on cuda:  17%|█▋        | 1617/9708 [00:07<00:35, 230.34batch/s]

Training batches on cuda:  17%|█▋        | 1641/9708 [00:07<00:34, 231.12batch/s]

Training batches on cuda:  17%|█▋        | 1665/9708 [00:07<00:35, 228.93batch/s]

Training batches on cuda:  17%|█▋        | 1688/9708 [00:07<00:35, 226.17batch/s]

Training batches on cuda:  18%|█▊        | 1711/9708 [00:07<00:35, 225.28batch/s]

Training batches on cuda:  18%|█▊        | 1734/9708 [00:07<00:35, 225.33batch/s]

Training batches on cuda:  18%|█▊        | 1758/9708 [00:07<00:35, 226.06batch/s]

Training batches on cuda:  18%|█▊        | 1781/9708 [00:08<00:35, 223.14batch/s]

Training batches on cuda:  19%|█▊        | 1804/9708 [00:08<00:36, 217.58batch/s]

Training batches on cuda:  19%|█▉        | 1826/9708 [00:08<00:36, 218.00batch/s]

Training batches on cuda:  19%|█▉        | 1848/9708 [00:08<00:36, 218.24batch/s]

Training batches on cuda:  19%|█▉        | 1870/9708 [00:08<00:35, 218.63batch/s]

Training batches on cuda:  19%|█▉        | 1892/9708 [00:08<00:35, 217.29batch/s]

Training batches on cuda:  20%|█▉        | 1914/9708 [00:08<00:36, 216.49batch/s]

Training batches on cuda:  20%|█▉        | 1936/9708 [00:08<00:35, 217.40batch/s]

Training batches on cuda:  20%|██        | 1959/9708 [00:08<00:35, 218.71batch/s]

Training batches on cuda:  20%|██        | 1983/9708 [00:09<00:34, 222.14batch/s]

Training batches on cuda:  21%|██        | 2007/9708 [00:09<00:34, 224.75batch/s]

Training batches on cuda:  21%|██        | 2030/9708 [00:09<00:33, 225.88batch/s]

Training batches on cuda:  21%|██        | 2054/9708 [00:09<00:33, 227.54batch/s]

Training batches on cuda:  21%|██▏       | 2077/9708 [00:09<00:33, 227.33batch/s]

Training batches on cuda:  22%|██▏       | 2100/9708 [00:09<00:34, 217.66batch/s]

Training batches on cuda:  22%|██▏       | 2122/9708 [00:09<00:35, 215.58batch/s]

Training batches on cuda:  22%|██▏       | 2144/9708 [00:09<00:36, 210.08batch/s]

Training batches on cuda:  22%|██▏       | 2166/9708 [00:09<00:35, 211.08batch/s]

Training batches on cuda:  23%|██▎       | 2188/9708 [00:09<00:35, 212.87batch/s]

Training batches on cuda:  23%|██▎       | 2210/9708 [00:10<00:35, 210.10batch/s]

Training batches on cuda:  23%|██▎       | 2232/9708 [00:10<00:37, 201.81batch/s]

Training batches on cuda:  23%|██▎       | 2253/9708 [00:10<00:37, 196.48batch/s]

Training batches on cuda:  23%|██▎       | 2273/9708 [00:10<00:38, 191.89batch/s]

Training batches on cuda:  24%|██▎       | 2294/9708 [00:10<00:38, 194.33batch/s]

Training batches on cuda:  24%|██▍       | 2314/9708 [00:10<00:37, 195.96batch/s]

Training batches on cuda:  24%|██▍       | 2335/9708 [00:10<00:37, 197.27batch/s]

Training batches on cuda:  24%|██▍       | 2356/9708 [00:10<00:36, 198.78batch/s]

Training batches on cuda:  24%|██▍       | 2376/9708 [00:10<00:40, 179.20batch/s]

Training batches on cuda:  25%|██▍       | 2398/9708 [00:11<00:38, 189.04batch/s]

Training batches on cuda:  25%|██▍       | 2419/9708 [00:11<00:37, 192.79batch/s]

Training batches on cuda:  25%|██▌       | 2441/9708 [00:11<00:36, 198.26batch/s]

Training batches on cuda:  25%|██▌       | 2463/9708 [00:11<00:35, 202.87batch/s]

Training batches on cuda:  26%|██▌       | 2485/9708 [00:11<00:34, 207.35batch/s]

Training batches on cuda:  26%|██▌       | 2509/9708 [00:11<00:33, 213.70batch/s]

Training batches on cuda:  26%|██▌       | 2532/9708 [00:11<00:32, 218.29batch/s]

Training batches on cuda:  26%|██▋       | 2555/9708 [00:11<00:32, 221.36batch/s]

Training batches on cuda:  27%|██▋       | 2578/9708 [00:11<00:32, 222.61batch/s]

Training batches on cuda:  27%|██▋       | 2601/9708 [00:12<00:35, 201.18batch/s]

Training batches on cuda:  27%|██▋       | 2623/9708 [00:12<00:34, 206.16batch/s]

Training batches on cuda:  27%|██▋       | 2644/9708 [00:12<00:34, 203.81batch/s]

Training batches on cuda:  27%|██▋       | 2665/9708 [00:12<00:35, 200.32batch/s]

Training batches on cuda:  28%|██▊       | 2686/9708 [00:12<00:34, 201.26batch/s]

Training batches on cuda:  28%|██▊       | 2707/9708 [00:12<00:34, 200.24batch/s]

Training batches on cuda:  28%|██▊       | 2728/9708 [00:12<00:34, 199.59batch/s]

Training batches on cuda:  28%|██▊       | 2749/9708 [00:12<00:35, 196.71batch/s]

Training batches on cuda:  29%|██▊       | 2770/9708 [00:12<00:35, 198.21batch/s]

Training batches on cuda:  29%|██▊       | 2790/9708 [00:12<00:34, 198.51batch/s]

Training batches on cuda:  29%|██▉       | 2812/9708 [00:13<00:33, 203.48batch/s]

Training batches on cuda:  29%|██▉       | 2834/9708 [00:13<00:33, 205.76batch/s]

Training batches on cuda:  29%|██▉       | 2858/9708 [00:13<00:31, 214.76batch/s]

Training batches on cuda:  30%|██▉       | 2881/9708 [00:13<00:31, 218.62batch/s]

Training batches on cuda:  30%|██▉       | 2905/9708 [00:13<00:30, 223.68batch/s]

Training batches on cuda:  30%|███       | 2928/9708 [00:13<00:30, 225.19batch/s]

Training batches on cuda:  30%|███       | 2951/9708 [00:13<00:31, 214.48batch/s]

Training batches on cuda:  31%|███       | 2973/9708 [00:13<00:32, 208.40batch/s]

Training batches on cuda:  31%|███       | 2994/9708 [00:13<00:32, 208.48batch/s]

Training batches on cuda:  31%|███       | 3016/9708 [00:14<00:32, 208.85batch/s]

Training batches on cuda:  31%|███▏      | 3038/9708 [00:14<00:31, 210.87batch/s]

Training batches on cuda:  32%|███▏      | 3061/9708 [00:14<00:30, 214.57batch/s]

Training batches on cuda:  32%|███▏      | 3084/9708 [00:14<00:30, 216.16batch/s]

Training batches on cuda:  32%|███▏      | 3106/9708 [00:14<00:30, 214.56batch/s]

Training batches on cuda:  32%|███▏      | 3128/9708 [00:14<00:30, 216.13batch/s]

Training batches on cuda:  32%|███▏      | 3150/9708 [00:14<00:30, 215.44batch/s]

Training batches on cuda:  33%|███▎      | 3172/9708 [00:14<00:30, 215.42batch/s]

Training batches on cuda:  33%|███▎      | 3194/9708 [00:14<00:30, 215.09batch/s]

Training batches on cuda:  33%|███▎      | 3216/9708 [00:14<00:30, 212.31batch/s]

Training batches on cuda:  33%|███▎      | 3238/9708 [00:15<00:30, 212.10batch/s]

Training batches on cuda:  34%|███▎      | 3261/9708 [00:15<00:30, 214.30batch/s]

Training batches on cuda:  34%|███▍      | 3283/9708 [00:15<00:30, 213.20batch/s]

Training batches on cuda:  34%|███▍      | 3305/9708 [00:15<00:30, 209.83batch/s]

Training batches on cuda:  34%|███▍      | 3327/9708 [00:15<00:30, 211.41batch/s]

Training batches on cuda:  34%|███▍      | 3349/9708 [00:15<00:30, 211.78batch/s]

Training batches on cuda:  35%|███▍      | 3371/9708 [00:15<00:29, 211.38batch/s]

Training batches on cuda:  35%|███▍      | 3393/9708 [00:15<00:29, 211.29batch/s]

Training batches on cuda:  35%|███▌      | 3415/9708 [00:15<00:30, 207.50batch/s]

Training batches on cuda:  35%|███▌      | 3436/9708 [00:15<00:31, 198.13batch/s]

Training batches on cuda:  36%|███▌      | 3457/9708 [00:16<00:31, 199.61batch/s]

Training batches on cuda:  36%|███▌      | 3479/9708 [00:16<00:30, 202.77batch/s]

Training batches on cuda:  36%|███▌      | 3500/9708 [00:16<00:30, 202.22batch/s]

Training batches on cuda:  36%|███▋      | 3521/9708 [00:16<00:30, 204.19batch/s]

Training batches on cuda:  36%|███▋      | 3542/9708 [00:16<00:30, 201.73batch/s]

Training batches on cuda:  37%|███▋      | 3563/9708 [00:16<00:30, 199.17batch/s]

Training batches on cuda:  37%|███▋      | 3584/9708 [00:16<00:30, 202.14batch/s]

Training batches on cuda:  37%|███▋      | 3606/9708 [00:16<00:29, 204.60batch/s]

Training batches on cuda:  37%|███▋      | 3627/9708 [00:16<00:30, 201.76batch/s]

Training batches on cuda:  38%|███▊      | 3648/9708 [00:17<00:29, 203.52batch/s]

Training batches on cuda:  38%|███▊      | 3669/9708 [00:17<00:30, 198.75batch/s]

Training batches on cuda:  38%|███▊      | 3689/9708 [00:17<00:30, 198.83batch/s]

Training batches on cuda:  38%|███▊      | 3709/9708 [00:17<00:30, 195.43batch/s]

Training batches on cuda:  38%|███▊      | 3729/9708 [00:17<00:30, 194.34batch/s]

Training batches on cuda:  39%|███▊      | 3749/9708 [00:17<00:30, 192.55batch/s]

Training batches on cuda:  39%|███▉      | 3769/9708 [00:17<00:31, 191.33batch/s]

Training batches on cuda:  39%|███▉      | 3789/9708 [00:17<00:31, 189.81batch/s]

Training batches on cuda:  39%|███▉      | 3809/9708 [00:17<00:30, 192.32batch/s]

Training batches on cuda:  39%|███▉      | 3829/9708 [00:17<00:31, 186.25batch/s]

Training batches on cuda:  40%|███▉      | 3848/9708 [00:18<00:31, 185.98batch/s]

Training batches on cuda:  40%|███▉      | 3868/9708 [00:18<00:31, 187.28batch/s]

Training batches on cuda:  40%|████      | 3888/9708 [00:18<00:30, 189.33batch/s]

Training batches on cuda:  40%|████      | 3907/9708 [00:18<00:31, 185.39batch/s]

Training batches on cuda:  40%|████      | 3926/9708 [00:18<00:31, 182.89batch/s]

Training batches on cuda:  41%|████      | 3945/9708 [00:18<00:31, 183.15batch/s]

Training batches on cuda:  41%|████      | 3965/9708 [00:18<00:30, 187.24batch/s]

Training batches on cuda:  41%|████      | 3986/9708 [00:18<00:29, 192.30batch/s]

Training batches on cuda:  41%|████▏     | 4010/9708 [00:18<00:28, 202.90batch/s]

Training batches on cuda:  42%|████▏     | 4032/9708 [00:19<00:27, 206.17batch/s]

Training batches on cuda:  42%|████▏     | 4054/9708 [00:19<00:27, 207.39batch/s]

Training batches on cuda:  42%|████▏     | 4076/9708 [00:19<00:26, 208.66batch/s]

Training batches on cuda:  42%|████▏     | 4097/9708 [00:19<00:26, 208.80batch/s]

Training batches on cuda:  42%|████▏     | 4118/9708 [00:19<00:26, 208.36batch/s]

Training batches on cuda:  43%|████▎     | 4140/9708 [00:19<00:26, 209.43batch/s]

Training batches on cuda:  43%|████▎     | 4161/9708 [00:19<00:26, 207.31batch/s]

Training batches on cuda:  43%|████▎     | 4182/9708 [00:19<00:27, 202.11batch/s]

Training batches on cuda:  43%|████▎     | 4203/9708 [00:19<00:27, 203.54batch/s]

Training batches on cuda:  44%|████▎     | 4225/9708 [00:19<00:26, 206.24batch/s]

Training batches on cuda:  44%|████▎     | 4246/9708 [00:20<00:26, 205.79batch/s]

Training batches on cuda:  44%|████▍     | 4267/9708 [00:20<00:26, 206.34batch/s]

Training batches on cuda:  44%|████▍     | 4289/9708 [00:20<00:26, 207.81batch/s]

Training batches on cuda:  44%|████▍     | 4310/9708 [00:20<00:26, 205.04batch/s]

Training batches on cuda:  45%|████▍     | 4331/9708 [00:20<00:26, 205.03batch/s]

Training batches on cuda:  45%|████▍     | 4353/9708 [00:20<00:25, 206.60batch/s]

Training batches on cuda:  45%|████▌     | 4374/9708 [00:20<00:26, 201.52batch/s]

Training batches on cuda:  45%|████▌     | 4395/9708 [00:20<00:26, 197.45batch/s]

Training batches on cuda:  45%|████▌     | 4415/9708 [00:20<00:27, 195.70batch/s]

Training batches on cuda:  46%|████▌     | 4435/9708 [00:21<00:29, 179.00batch/s]

Training batches on cuda:  46%|████▌     | 4454/9708 [00:21<00:29, 178.67batch/s]

Training batches on cuda:  46%|████▌     | 4475/9708 [00:21<00:28, 184.95batch/s]

Training batches on cuda:  46%|████▋     | 4495/9708 [00:21<00:27, 188.18batch/s]

Training batches on cuda:  47%|████▋     | 4515/9708 [00:21<00:27, 189.60batch/s]

Training batches on cuda:  47%|████▋     | 4537/9708 [00:21<00:26, 195.92batch/s]

Training batches on cuda:  47%|████▋     | 4560/9708 [00:21<00:25, 203.12batch/s]

Training batches on cuda:  47%|████▋     | 4585/9708 [00:21<00:23, 213.56batch/s]

Training batches on cuda:  47%|████▋     | 4610/9708 [00:21<00:22, 221.80batch/s]

Training batches on cuda:  48%|████▊     | 4633/9708 [00:22<00:24, 207.81batch/s]

Training batches on cuda:  48%|████▊     | 4655/9708 [00:22<00:24, 205.75batch/s]

Training batches on cuda:  48%|████▊     | 4676/9708 [00:22<00:24, 202.06batch/s]

Training batches on cuda:  48%|████▊     | 4699/9708 [00:22<00:24, 208.52batch/s]

Training batches on cuda:  49%|████▊     | 4724/9708 [00:22<00:22, 218.53batch/s]

Training batches on cuda:  49%|████▉     | 4749/9708 [00:22<00:22, 225.37batch/s]

Training batches on cuda:  49%|████▉     | 4772/9708 [00:22<00:22, 216.80batch/s]

Training batches on cuda:  49%|████▉     | 4794/9708 [00:22<00:23, 213.22batch/s]

Training batches on cuda:  50%|████▉     | 4816/9708 [00:22<00:23, 211.91batch/s]

Training batches on cuda:  50%|████▉     | 4838/9708 [00:22<00:24, 201.35batch/s]

Training batches on cuda:  50%|█████     | 4859/9708 [00:23<00:24, 195.25batch/s]

Training batches on cuda:  50%|█████     | 4879/9708 [00:23<00:25, 192.08batch/s]

Training batches on cuda:  50%|█████     | 4899/9708 [00:23<00:25, 189.24batch/s]

Training batches on cuda:  51%|█████     | 4919/9708 [00:23<00:25, 187.64batch/s]

Training batches on cuda:  51%|█████     | 4938/9708 [00:23<00:25, 186.53batch/s]

Training batches on cuda:  51%|█████     | 4959/9708 [00:23<00:24, 192.72batch/s]

Training batches on cuda:  51%|█████▏    | 4981/9708 [00:23<00:23, 198.70batch/s]

Training batches on cuda:  52%|█████▏    | 5002/9708 [00:23<00:23, 201.79batch/s]

Training batches on cuda:  52%|█████▏    | 5023/9708 [00:23<00:23, 203.63batch/s]

Training batches on cuda:  52%|█████▏    | 5044/9708 [00:24<00:22, 205.11batch/s]

Training batches on cuda:  52%|█████▏    | 5066/9708 [00:24<00:22, 206.61batch/s]

Training batches on cuda:  52%|█████▏    | 5087/9708 [00:24<00:22, 205.41batch/s]

Training batches on cuda:  53%|█████▎    | 5108/9708 [00:24<00:22, 200.79batch/s]

Training batches on cuda:  53%|█████▎    | 5129/9708 [00:24<00:23, 195.38batch/s]

Training batches on cuda:  53%|█████▎    | 5149/9708 [00:24<00:23, 190.62batch/s]

Training batches on cuda:  53%|█████▎    | 5169/9708 [00:24<00:23, 190.81batch/s]

Training batches on cuda:  53%|█████▎    | 5190/9708 [00:24<00:23, 195.49batch/s]

Training batches on cuda:  54%|█████▎    | 5211/9708 [00:24<00:22, 198.80batch/s]

Training batches on cuda:  54%|█████▍    | 5234/9708 [00:24<00:21, 206.35batch/s]

Training batches on cuda:  54%|█████▍    | 5257/9708 [00:25<00:20, 212.45batch/s]

Training batches on cuda:  54%|█████▍    | 5281/9708 [00:25<00:20, 218.42batch/s]

Training batches on cuda:  55%|█████▍    | 5304/9708 [00:25<00:19, 221.76batch/s]

Training batches on cuda:  55%|█████▍    | 5330/9708 [00:25<00:19, 230.07batch/s]

Training batches on cuda:  55%|█████▌    | 5354/9708 [00:25<00:19, 229.07batch/s]

Training batches on cuda:  55%|█████▌    | 5380/9708 [00:25<00:18, 236.25batch/s]

Training batches on cuda:  56%|█████▌    | 5404/9708 [00:25<00:18, 237.20batch/s]

Training batches on cuda:  56%|█████▌    | 5428/9708 [00:25<00:18, 234.09batch/s]

Training batches on cuda:  56%|█████▌    | 5452/9708 [00:25<00:18, 232.54batch/s]

Training batches on cuda:  56%|█████▋    | 5476/9708 [00:26<00:18, 224.82batch/s]

Training batches on cuda:  57%|█████▋    | 5499/9708 [00:26<00:19, 210.63batch/s]

Training batches on cuda:  57%|█████▋    | 5521/9708 [00:26<00:20, 200.57batch/s]

Training batches on cuda:  57%|█████▋    | 5542/9708 [00:26<00:21, 194.38batch/s]

Training batches on cuda:  57%|█████▋    | 5562/9708 [00:26<00:22, 186.85batch/s]

Training batches on cuda:  58%|█████▊    | 5584/9708 [00:26<00:21, 193.26batch/s]

Training batches on cuda:  58%|█████▊    | 5605/9708 [00:26<00:20, 195.78batch/s]

Training batches on cuda:  58%|█████▊    | 5627/9708 [00:26<00:20, 199.96batch/s]

Training batches on cuda:  58%|█████▊    | 5648/9708 [00:26<00:20, 195.97batch/s]

Training batches on cuda:  58%|█████▊    | 5670/9708 [00:27<00:20, 200.85batch/s]

Training batches on cuda:  59%|█████▊    | 5691/9708 [00:27<00:20, 200.65batch/s]

Training batches on cuda:  59%|█████▉    | 5712/9708 [00:27<00:20, 196.09batch/s]

Training batches on cuda:  59%|█████▉    | 5734/9708 [00:27<00:19, 200.37batch/s]

Training batches on cuda:  59%|█████▉    | 5756/9708 [00:27<00:19, 204.09batch/s]

Training batches on cuda:  60%|█████▉    | 5777/9708 [00:27<00:19, 205.22batch/s]

Training batches on cuda:  60%|█████▉    | 5798/9708 [00:27<00:18, 206.14batch/s]

Training batches on cuda:  60%|█████▉    | 5821/9708 [00:27<00:18, 211.62batch/s]

Training batches on cuda:  60%|██████    | 5844/9708 [00:27<00:17, 214.89batch/s]

Training batches on cuda:  60%|██████    | 5867/9708 [00:27<00:17, 217.93batch/s]

Training batches on cuda:  61%|██████    | 5890/9708 [00:28<00:17, 220.55batch/s]

Training batches on cuda:  61%|██████    | 5913/9708 [00:28<00:17, 219.97batch/s]

Training batches on cuda:  61%|██████    | 5936/9708 [00:28<00:17, 212.11batch/s]

Training batches on cuda:  61%|██████▏   | 5958/9708 [00:28<00:17, 212.45batch/s]

Training batches on cuda:  62%|██████▏   | 5980/9708 [00:28<00:17, 211.15batch/s]

Training batches on cuda:  62%|██████▏   | 6003/9708 [00:28<00:17, 214.15batch/s]

Training batches on cuda:  62%|██████▏   | 6026/9708 [00:28<00:17, 216.08batch/s]

Training batches on cuda:  62%|██████▏   | 6049/9708 [00:28<00:16, 218.89batch/s]

Training batches on cuda:  63%|██████▎   | 6074/9708 [00:28<00:15, 227.14batch/s]

Training batches on cuda:  63%|██████▎   | 6098/9708 [00:29<00:15, 230.17batch/s]

Training batches on cuda:  63%|██████▎   | 6122/9708 [00:29<00:15, 226.50batch/s]

Training batches on cuda:  63%|██████▎   | 6145/9708 [00:29<00:16, 210.63batch/s]

Training batches on cuda:  64%|██████▎   | 6167/9708 [00:29<00:16, 211.85batch/s]

Training batches on cuda:  64%|██████▍   | 6189/9708 [00:29<00:16, 211.68batch/s]

Training batches on cuda:  64%|██████▍   | 6212/9708 [00:29<00:16, 215.58batch/s]

Training batches on cuda:  64%|██████▍   | 6235/9708 [00:29<00:15, 217.64batch/s]

Training batches on cuda:  64%|██████▍   | 6257/9708 [00:29<00:16, 214.56batch/s]

Training batches on cuda:  65%|██████▍   | 6280/9708 [00:29<00:15, 217.48batch/s]

Training batches on cuda:  65%|██████▍   | 6302/9708 [00:29<00:15, 216.11batch/s]

Training batches on cuda:  65%|██████▌   | 6324/9708 [00:30<00:16, 209.35batch/s]

Training batches on cuda:  65%|██████▌   | 6346/9708 [00:30<00:16, 200.89batch/s]

Training batches on cuda:  66%|██████▌   | 6367/9708 [00:30<00:16, 199.04batch/s]

Training batches on cuda:  66%|██████▌   | 6388/9708 [00:30<00:16, 201.16batch/s]

Training batches on cuda:  66%|██████▌   | 6409/9708 [00:30<00:16, 198.57batch/s]

Training batches on cuda:  66%|██████▌   | 6429/9708 [00:30<00:17, 190.81batch/s]

Training batches on cuda:  66%|██████▋   | 6449/9708 [00:30<00:17, 189.77batch/s]

Training batches on cuda:  67%|██████▋   | 6469/9708 [00:30<00:17, 189.45batch/s]

Training batches on cuda:  67%|██████▋   | 6490/9708 [00:30<00:16, 194.63batch/s]

Training batches on cuda:  67%|██████▋   | 6510/9708 [00:31<00:18, 176.96batch/s]

Training batches on cuda:  67%|██████▋   | 6532/9708 [00:31<00:16, 187.40batch/s]

Training batches on cuda:  68%|██████▊   | 6553/9708 [00:31<00:16, 192.21batch/s]

Training batches on cuda:  68%|██████▊   | 6577/9708 [00:31<00:15, 202.38batch/s]

Training batches on cuda:  68%|██████▊   | 6600/9708 [00:31<00:14, 209.79batch/s]

Training batches on cuda:  68%|██████▊   | 6623/9708 [00:31<00:14, 212.82batch/s]

Training batches on cuda:  68%|██████▊   | 6646/9708 [00:31<00:14, 216.50batch/s]

Training batches on cuda:  69%|██████▊   | 6669/9708 [00:31<00:13, 218.12batch/s]

Training batches on cuda:  69%|██████▉   | 6692/9708 [00:31<00:13, 219.80batch/s]

Training batches on cuda:  69%|██████▉   | 6715/9708 [00:32<00:13, 218.88batch/s]

Training batches on cuda:  69%|██████▉   | 6739/9708 [00:32<00:13, 222.86batch/s]

Training batches on cuda:  70%|██████▉   | 6762/9708 [00:32<00:13, 219.55batch/s]

Training batches on cuda:  70%|██████▉   | 6785/9708 [00:32<00:13, 221.61batch/s]

Training batches on cuda:  70%|███████   | 6808/9708 [00:32<00:12, 223.50batch/s]

Training batches on cuda:  70%|███████   | 6831/9708 [00:32<00:12, 224.81batch/s]

Training batches on cuda:  71%|███████   | 6854/9708 [00:32<00:13, 215.09batch/s]

Training batches on cuda:  71%|███████   | 6876/9708 [00:32<00:14, 194.78batch/s]

Training batches on cuda:  71%|███████   | 6896/9708 [00:32<00:14, 188.91batch/s]

Training batches on cuda:  71%|███████   | 6916/9708 [00:32<00:14, 188.87batch/s]

Training batches on cuda:  71%|███████▏  | 6937/9708 [00:33<00:14, 193.70batch/s]

Training batches on cuda:  72%|███████▏  | 6957/9708 [00:33<00:14, 191.19batch/s]

Training batches on cuda:  72%|███████▏  | 6979/9708 [00:33<00:13, 195.87batch/s]

Training batches on cuda:  72%|███████▏  | 7000/9708 [00:33<00:13, 197.44batch/s]

Training batches on cuda:  72%|███████▏  | 7022/9708 [00:33<00:13, 201.18batch/s]

Training batches on cuda:  73%|███████▎  | 7043/9708 [00:33<00:13, 202.79batch/s]

Training batches on cuda:  73%|███████▎  | 7064/9708 [00:33<00:13, 200.07batch/s]

Training batches on cuda:  73%|███████▎  | 7085/9708 [00:33<00:13, 193.67batch/s]

Training batches on cuda:  73%|███████▎  | 7107/9708 [00:33<00:13, 197.80batch/s]

Training batches on cuda:  73%|███████▎  | 7129/9708 [00:34<00:12, 201.71batch/s]

Training batches on cuda:  74%|███████▎  | 7150/9708 [00:34<00:12, 203.20batch/s]

Training batches on cuda:  74%|███████▍  | 7172/9708 [00:34<00:12, 205.77batch/s]

Training batches on cuda:  74%|███████▍  | 7194/9708 [00:34<00:12, 208.81batch/s]

Training batches on cuda:  74%|███████▍  | 7215/9708 [00:34<00:12, 207.44batch/s]

Training batches on cuda:  75%|███████▍  | 7237/9708 [00:34<00:11, 208.14batch/s]

Training batches on cuda:  75%|███████▍  | 7258/9708 [00:34<00:11, 208.58batch/s]

Training batches on cuda:  75%|███████▌  | 7281/9708 [00:34<00:11, 213.16batch/s]

Training batches on cuda:  75%|███████▌  | 7304/9708 [00:34<00:11, 216.40batch/s]

Training batches on cuda:  75%|███████▌  | 7326/9708 [00:34<00:10, 217.18batch/s]

Training batches on cuda:  76%|███████▌  | 7349/9708 [00:35<00:10, 218.35batch/s]

Training batches on cuda:  76%|███████▌  | 7371/9708 [00:35<00:10, 214.83batch/s]

Training batches on cuda:  76%|███████▌  | 7393/9708 [00:35<00:10, 213.28batch/s]

Training batches on cuda:  76%|███████▋  | 7415/9708 [00:35<00:10, 213.37batch/s]

Training batches on cuda:  77%|███████▋  | 7437/9708 [00:35<00:10, 212.35batch/s]

Training batches on cuda:  77%|███████▋  | 7459/9708 [00:35<00:10, 210.46batch/s]

Training batches on cuda:  77%|███████▋  | 7481/9708 [00:35<00:10, 210.36batch/s]

Training batches on cuda:  77%|███████▋  | 7503/9708 [00:35<00:10, 209.39batch/s]

Training batches on cuda:  78%|███████▊  | 7524/9708 [00:35<00:10, 206.59batch/s]

Training batches on cuda:  78%|███████▊  | 7545/9708 [00:36<00:10, 200.82batch/s]

Training batches on cuda:  78%|███████▊  | 7566/9708 [00:36<00:10, 201.18batch/s]

Training batches on cuda:  78%|███████▊  | 7588/9708 [00:36<00:10, 203.81batch/s]

Training batches on cuda:  78%|███████▊  | 7610/9708 [00:36<00:10, 206.34batch/s]

Training batches on cuda:  79%|███████▊  | 7634/9708 [00:36<00:09, 214.49batch/s]

Training batches on cuda:  79%|███████▉  | 7656/9708 [00:36<00:09, 210.71batch/s]

Training batches on cuda:  79%|███████▉  | 7678/9708 [00:36<00:09, 204.26batch/s]

Training batches on cuda:  79%|███████▉  | 7699/9708 [00:36<00:09, 202.28batch/s]

Training batches on cuda:  80%|███████▉  | 7720/9708 [00:36<00:09, 201.11batch/s]

Training batches on cuda:  80%|███████▉  | 7741/9708 [00:36<00:09, 198.90batch/s]

Training batches on cuda:  80%|███████▉  | 7761/9708 [00:37<00:09, 196.30batch/s]

Training batches on cuda:  80%|████████  | 7783/9708 [00:37<00:09, 201.58batch/s]

Training batches on cuda:  80%|████████  | 7804/9708 [00:37<00:10, 190.02batch/s]

Training batches on cuda:  81%|████████  | 7824/9708 [00:37<00:09, 192.60batch/s]

Training batches on cuda:  81%|████████  | 7844/9708 [00:37<00:09, 190.11batch/s]

Training batches on cuda:  81%|████████  | 7864/9708 [00:37<00:10, 183.54batch/s]

Training batches on cuda:  81%|████████  | 7883/9708 [00:37<00:09, 184.37batch/s]

Training batches on cuda:  81%|████████▏ | 7904/9708 [00:37<00:09, 190.27batch/s]

Training batches on cuda:  82%|████████▏ | 7924/9708 [00:37<00:09, 190.70batch/s]

Training batches on cuda:  82%|████████▏ | 7945/9708 [00:38<00:09, 195.05batch/s]

Training batches on cuda:  82%|████████▏ | 7966/9708 [00:38<00:08, 198.33batch/s]

Training batches on cuda:  82%|████████▏ | 7987/9708 [00:38<00:08, 201.61batch/s]

Training batches on cuda:  82%|████████▏ | 8008/9708 [00:38<00:08, 194.78batch/s]

Training batches on cuda:  83%|████████▎ | 8030/9708 [00:38<00:08, 199.00batch/s]

Training batches on cuda:  83%|████████▎ | 8051/9708 [00:38<00:08, 200.22batch/s]

Training batches on cuda:  83%|████████▎ | 8073/9708 [00:38<00:08, 203.36batch/s]

Training batches on cuda:  83%|████████▎ | 8094/9708 [00:38<00:07, 205.01batch/s]

Training batches on cuda:  84%|████████▎ | 8115/9708 [00:38<00:07, 205.83batch/s]

Training batches on cuda:  84%|████████▍ | 8136/9708 [00:38<00:07, 205.05batch/s]

Training batches on cuda:  84%|████████▍ | 8157/9708 [00:39<00:07, 204.81batch/s]

Training batches on cuda:  84%|████████▍ | 8178/9708 [00:39<00:07, 204.13batch/s]

Training batches on cuda:  84%|████████▍ | 8199/9708 [00:39<00:07, 205.23batch/s]

Training batches on cuda:  85%|████████▍ | 8220/9708 [00:39<00:07, 203.21batch/s]

Training batches on cuda:  85%|████████▍ | 8241/9708 [00:39<00:07, 200.47batch/s]

Training batches on cuda:  85%|████████▌ | 8262/9708 [00:39<00:07, 201.35batch/s]

Training batches on cuda:  85%|████████▌ | 8283/9708 [00:39<00:07, 192.68batch/s]

Training batches on cuda:  86%|████████▌ | 8305/9708 [00:39<00:07, 196.47batch/s]

Training batches on cuda:  86%|████████▌ | 8326/9708 [00:39<00:06, 197.87batch/s]

Training batches on cuda:  86%|████████▌ | 8347/9708 [00:40<00:06, 200.69batch/s]

Training batches on cuda:  86%|████████▌ | 8368/9708 [00:40<00:06, 202.63batch/s]

Training batches on cuda:  86%|████████▋ | 8389/9708 [00:40<00:06, 204.15batch/s]

Training batches on cuda:  87%|████████▋ | 8410/9708 [00:40<00:06, 205.41batch/s]

Training batches on cuda:  87%|████████▋ | 8431/9708 [00:40<00:06, 203.69batch/s]

Training batches on cuda:  87%|████████▋ | 8452/9708 [00:40<00:06, 205.52batch/s]

Training batches on cuda:  87%|████████▋ | 8474/9708 [00:40<00:05, 207.06batch/s]

Training batches on cuda:  88%|████████▊ | 8495/9708 [00:40<00:05, 207.68batch/s]

Training batches on cuda:  88%|████████▊ | 8517/9708 [00:40<00:05, 209.05batch/s]

Training batches on cuda:  88%|████████▊ | 8538/9708 [00:40<00:05, 208.69batch/s]

Training batches on cuda:  88%|████████▊ | 8560/9708 [00:41<00:05, 209.13batch/s]

Training batches on cuda:  88%|████████▊ | 8581/9708 [00:41<00:06, 183.74batch/s]

Training batches on cuda:  89%|████████▊ | 8601/9708 [00:41<00:05, 188.08batch/s]

Training batches on cuda:  89%|████████▉ | 8621/9708 [00:41<00:05, 188.83batch/s]

Training batches on cuda:  89%|████████▉ | 8641/9708 [00:41<00:05, 191.93batch/s]

Training batches on cuda:  89%|████████▉ | 8661/9708 [00:41<00:05, 189.24batch/s]

Training batches on cuda:  89%|████████▉ | 8681/9708 [00:41<00:05, 189.79batch/s]

Training batches on cuda:  90%|████████▉ | 8702/9708 [00:41<00:05, 192.77batch/s]

Training batches on cuda:  90%|████████▉ | 8722/9708 [00:41<00:05, 188.89batch/s]

Training batches on cuda:  90%|█████████ | 8741/9708 [00:42<00:05, 187.92batch/s]

Training batches on cuda:  90%|█████████ | 8760/9708 [00:42<00:05, 186.13batch/s]

Training batches on cuda:  90%|█████████ | 8779/9708 [00:42<00:05, 182.29batch/s]

Training batches on cuda:  91%|█████████ | 8798/9708 [00:42<00:05, 176.64batch/s]

Training batches on cuda:  91%|█████████ | 8816/9708 [00:42<00:05, 176.64batch/s]

Training batches on cuda:  91%|█████████ | 8836/9708 [00:42<00:04, 181.02batch/s]

Training batches on cuda:  91%|█████████ | 8855/9708 [00:42<00:04, 174.08batch/s]

Training batches on cuda:  91%|█████████▏| 8876/9708 [00:42<00:04, 181.34batch/s]

Training batches on cuda:  92%|█████████▏| 8895/9708 [00:42<00:04, 175.56batch/s]

Training batches on cuda:  92%|█████████▏| 8916/9708 [00:43<00:04, 183.58batch/s]

Training batches on cuda:  92%|█████████▏| 8937/9708 [00:43<00:04, 188.63batch/s]

Training batches on cuda:  92%|█████████▏| 8957/9708 [00:43<00:03, 188.58batch/s]

Training batches on cuda:  92%|█████████▏| 8977/9708 [00:43<00:03, 189.52batch/s]

Training batches on cuda:  93%|█████████▎| 8999/9708 [00:43<00:03, 195.81batch/s]

Training batches on cuda:  93%|█████████▎| 9020/9708 [00:43<00:03, 197.61batch/s]

Training batches on cuda:  93%|█████████▎| 9040/9708 [00:43<00:03, 191.11batch/s]

Training batches on cuda:  93%|█████████▎| 9061/9708 [00:43<00:03, 193.87batch/s]

Training batches on cuda:  94%|█████████▎| 9081/9708 [00:43<00:03, 191.34batch/s]

Training batches on cuda:  94%|█████████▍| 9102/9708 [00:43<00:03, 194.26batch/s]

Training batches on cuda:  94%|█████████▍| 9122/9708 [00:44<00:03, 193.72batch/s]

Training batches on cuda:  94%|█████████▍| 9142/9708 [00:44<00:02, 190.28batch/s]

Training batches on cuda:  94%|█████████▍| 9162/9708 [00:44<00:02, 189.38batch/s]

Training batches on cuda:  95%|█████████▍| 9182/9708 [00:44<00:02, 190.70batch/s]

Training batches on cuda:  95%|█████████▍| 9202/9708 [00:44<00:02, 190.34batch/s]

Training batches on cuda:  95%|█████████▌| 9224/9708 [00:44<00:02, 196.37batch/s]

Training batches on cuda:  95%|█████████▌| 9245/9708 [00:44<00:02, 199.27batch/s]

Training batches on cuda:  95%|█████████▌| 9266/9708 [00:44<00:02, 200.15batch/s]

Training batches on cuda:  96%|█████████▌| 9287/9708 [00:44<00:02, 202.19batch/s]

Training batches on cuda:  96%|█████████▌| 9310/9708 [00:45<00:01, 207.35batch/s]

Training batches on cuda:  96%|█████████▌| 9331/9708 [00:45<00:01, 206.86batch/s]

Training batches on cuda:  96%|█████████▋| 9353/9708 [00:45<00:01, 209.50batch/s]

Training batches on cuda:  97%|█████████▋| 9374/9708 [00:45<00:01, 201.36batch/s]

Training batches on cuda:  97%|█████████▋| 9395/9708 [00:45<00:01, 201.41batch/s]

Training batches on cuda:  97%|█████████▋| 9417/9708 [00:45<00:01, 205.68batch/s]

Training batches on cuda:  97%|█████████▋| 9440/9708 [00:45<00:01, 211.10batch/s]

Training batches on cuda:  97%|█████████▋| 9462/9708 [00:45<00:01, 211.72batch/s]

Training batches on cuda:  98%|█████████▊| 9484/9708 [00:45<00:01, 207.43batch/s]

Training batches on cuda:  98%|█████████▊| 9505/9708 [00:45<00:00, 203.94batch/s]

Training batches on cuda:  98%|█████████▊| 9526/9708 [00:46<00:00, 201.37batch/s]

Training batches on cuda:  98%|█████████▊| 9547/9708 [00:46<00:00, 180.77batch/s]

Training batches on cuda:  99%|█████████▊| 9569/9708 [00:46<00:00, 189.77batch/s]

Training batches on cuda:  99%|█████████▉| 9592/9708 [00:46<00:00, 198.73batch/s]

Training batches on cuda:  99%|█████████▉| 9613/9708 [00:46<00:00, 200.45batch/s]

Training batches on cuda:  99%|█████████▉| 9635/9708 [00:46<00:00, 205.13batch/s]

Training batches on cuda:  99%|█████████▉| 9656/9708 [00:46<00:00, 199.82batch/s]

Training batches on cuda: 100%|█████████▉| 9680/9708 [00:46<00:00, 208.49batch/s]

Training batches on cuda: 100%|█████████▉| 9703/9708 [00:46<00:00, 213.43batch/s]

                                                                                 
Training epochs on cuda:  90%|█████████ | 9/10 [07:48<00:46, 46.91s/epoch, loss=0.0417, prev_loss=0.0417]2021-09-16:19:06:27,816 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:06:28,886 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:06:43,841 INFO     [evaluator.py:657] Evaluation took 14.96s seconds
2021-09-16:19:06:44,0 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [08:04<00:00, 51.88s/epoch, loss=0.0417, prev_loss=0.0417]
Training epochs on cuda: 100%|██████████| 10/10 [08:04<00:00, 48.44s/epoch, loss=0.0417, prev_loss=0.0417]
2021-09-16:19:06:44,1 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmproeqa94r'
2021-09-16:19:06:44,85 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmproeqa94r' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.17ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.29ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.39ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.56ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.55ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.59ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.66ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.66ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.66ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.66ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.66ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.66ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:06:58,897 INFO     [evaluator.py:657] Evaluation took 14.79s seconds
[I 2021-09-16 19:06:58,902] Trial 67 finished with value: 0.0029484765813707625 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.04598277981646929, 'optimizer.lr': 0.04446634410050812, 'negative_sampler.num_negs_per_pos': 23, 'training.batch_size': 32}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:06:58,925 WARNING  [api.py:823] No random seed is specified. Setting to 313024108.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/170 [00:00<00:06, 25.31batch/s]

Training batches on cuda:   5%|▍         | 8/170 [00:00<00:05, 29.66batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:04, 34.15batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:03, 38.51batch/s]

Training batches on cuda:  15%|█▌        | 26/170 [00:00<00:03, 42.11batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:00<00:03, 45.32batch/s]

Training batches on cuda:  22%|██▏       | 38/170 [00:00<00:02, 47.42batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:00<00:02, 49.05batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:01<00:02, 50.32batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:01<00:02, 49.74batch/s]

Training batches on cuda:  36%|███▋      | 62/170 [00:01<00:02, 49.86batch/s]

Training batches on cuda:  40%|████      | 68/170 [00:01<00:02, 50.25batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:01<00:01, 49.97batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:01<00:01, 50.74batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:01<00:01, 51.77batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:01<00:01, 52.36batch/s]

Training batches on cuda:  58%|█████▊    | 98/170 [00:01<00:01, 53.29batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:02<00:01, 54.73batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:02<00:01, 56.10batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:02<00:00, 57.84batch/s]

Training batches on cuda:  73%|███████▎  | 124/170 [00:02<00:00, 58.63batch/s]

Training batches on cuda:  77%|███████▋  | 131/170 [00:02<00:00, 59.70batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:02<00:00, 54.00batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:02<00:00, 55.44batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:02<00:00, 57.08batch/s]

Training batches on cuda:  92%|█████████▏| 157/170 [00:02<00:00, 58.47batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:03<00:00, 58.50batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:03<00:00, 56.66batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.00117, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:30,  3.36s/epoch, loss=0.00117, prev_loss=nan]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:04, 34.48batch/s]

Training batches on cuda:   5%|▌         | 9/170 [00:00<00:04, 37.69batch/s]

Training batches on cuda:   9%|▉         | 15/170 [00:00<00:03, 41.39batch/s]

Training batches on cuda:  12%|█▏        | 21/170 [00:00<00:03, 44.40batch/s]

Training batches on cuda:  16%|█▌        | 27/170 [00:00<00:03, 46.63batch/s]

Training batches on cuda:  19%|█▉        | 33/170 [00:00<00:02, 48.53batch/s]

Training batches on cuda:  23%|██▎       | 39/170 [00:00<00:02, 50.83batch/s]

Training batches on cuda:  26%|██▋       | 45/170 [00:00<00:02, 52.62batch/s]

Training batches on cuda:  30%|███       | 51/170 [00:00<00:02, 52.36batch/s]

Training batches on cuda:  34%|███▎      | 57/170 [00:01<00:02, 52.73batch/s]

Training batches on cuda:  37%|███▋      | 63/170 [00:01<00:02, 52.86batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:01<00:01, 52.03batch/s]

Training batches on cuda:  44%|████▍     | 75/170 [00:01<00:01, 52.10batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:01<00:01, 51.84batch/s]

Training batches on cuda:  51%|█████     | 87/170 [00:01<00:01, 51.28batch/s]

Training batches on cuda:  55%|█████▍    | 93/170 [00:01<00:01, 51.01batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:01<00:01, 51.69batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:02<00:01, 52.20batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:02<00:01, 51.62batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:02<00:01, 51.28batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:02<00:00, 51.04batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:02<00:00, 51.29batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:02<00:00, 51.67batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:02<00:00, 51.12batch/s]

Training batches on cuda:  86%|████████▋ | 147/170 [00:02<00:00, 50.24batch/s]

Training batches on cuda:  90%|█████████ | 153/170 [00:02<00:00, 50.32batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:03<00:00, 48.36batch/s]

Training batches on cuda:  96%|█████████▋| 164/170 [00:03<00:00, 47.27batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:03<00:00, 47.58batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:30,  3.36s/epoch, loss=0.00117, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:27,  3.42s/epoch, loss=0.00117, prev_loss=nan]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:04, 35.28batch/s]

Training batches on cuda:   6%|▌         | 10/170 [00:00<00:04, 39.09batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:03, 42.61batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:03, 45.45batch/s]

Training batches on cuda:  16%|█▋        | 28/170 [00:00<00:03, 47.09batch/s]

Training batches on cuda:  19%|█▉        | 33/170 [00:00<00:03, 43.62batch/s]

Training batches on cuda:  23%|██▎       | 39/170 [00:00<00:02, 46.66batch/s]

Training batches on cuda:  26%|██▋       | 45/170 [00:00<00:02, 48.80batch/s]

Training batches on cuda:  30%|███       | 51/170 [00:01<00:02, 49.78batch/s]

Training batches on cuda:  34%|███▎      | 57/170 [00:01<00:02, 51.00batch/s]

Training batches on cuda:  37%|███▋      | 63/170 [00:01<00:02, 51.07batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:01<00:01, 51.37batch/s]

Training batches on cuda:  44%|████▍     | 75/170 [00:01<00:01, 51.60batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:01<00:01, 51.92batch/s]

Training batches on cuda:  51%|█████     | 87/170 [00:01<00:01, 52.00batch/s]

Training batches on cuda:  55%|█████▍    | 93/170 [00:01<00:01, 51.21batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:01<00:01, 50.86batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:02<00:01, 50.55batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:02<00:01, 50.89batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:02<00:01, 51.34batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:02<00:00, 50.12batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:02<00:00, 50.24batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:02<00:00, 49.77batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:02<00:00, 49.96batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:02<00:00, 48.48batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:03<00:00, 48.18batch/s]

Training batches on cuda:  92%|█████████▏| 157/170 [00:03<00:00, 49.22batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:03<00:00, 49.45batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:03<00:00, 50.60batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:10<00:27,  3.42s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:24,  3.47s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:04, 36.58batch/s]

Training batches on cuda:   5%|▌         | 9/170 [00:00<00:04, 39.15batch/s]

Training batches on cuda:   9%|▉         | 15/170 [00:00<00:03, 42.11batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:03, 42.93batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:00<00:03, 44.67batch/s]

Training batches on cuda:  18%|█▊        | 31/170 [00:00<00:02, 48.15batch/s]

Training batches on cuda:  22%|██▏       | 37/170 [00:00<00:02, 50.43batch/s]

Training batches on cuda:  25%|██▌       | 43/170 [00:00<00:02, 52.07batch/s]

Training batches on cuda:  29%|██▉       | 49/170 [00:00<00:02, 52.76batch/s]

Training batches on cuda:  32%|███▏      | 55/170 [00:01<00:02, 53.34batch/s]

Training batches on cuda:  36%|███▌      | 61/170 [00:01<00:02, 53.63batch/s]

Training batches on cuda:  39%|███▉      | 67/170 [00:01<00:01, 53.88batch/s]

Training batches on cuda:  43%|████▎     | 73/170 [00:01<00:01, 53.57batch/s]

Training batches on cuda:  46%|████▋     | 79/170 [00:01<00:01, 53.61batch/s]

Training batches on cuda:  50%|█████     | 85/170 [00:01<00:01, 54.19batch/s]

Training batches on cuda:  54%|█████▎    | 91/170 [00:01<00:01, 54.57batch/s]

Training batches on cuda:  57%|█████▋    | 97/170 [00:01<00:01, 54.73batch/s]

Training batches on cuda:  61%|██████    | 103/170 [00:01<00:01, 54.54batch/s]

Training batches on cuda:  64%|██████▍   | 109/170 [00:02<00:01, 54.63batch/s]

Training batches on cuda:  68%|██████▊   | 115/170 [00:02<00:01, 54.57batch/s]

Training batches on cuda:  71%|███████   | 121/170 [00:02<00:00, 54.81batch/s]

Training batches on cuda:  75%|███████▍  | 127/170 [00:02<00:00, 54.89batch/s]

Training batches on cuda:  78%|███████▊  | 133/170 [00:02<00:00, 54.90batch/s]

Training batches on cuda:  82%|████████▏ | 139/170 [00:02<00:00, 54.85batch/s]

Training batches on cuda:  85%|████████▌ | 145/170 [00:02<00:00, 54.87batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:02<00:00, 54.72batch/s]

Training batches on cuda:  92%|█████████▏| 157/170 [00:02<00:00, 54.41batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:03<00:00, 54.44batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:03<00:00, 54.38batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:24,  3.47s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  40%|████      | 4/10 [00:13<00:20,  3.46s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:04, 34.23batch/s]

Training batches on cuda:   6%|▌         | 10/170 [00:00<00:04, 38.23batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:03, 42.16batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:03, 45.15batch/s]

Training batches on cuda:  16%|█▋        | 28/170 [00:00<00:02, 47.55batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:00<00:02, 49.63batch/s]

Training batches on cuda:  24%|██▎       | 40/170 [00:00<00:02, 51.20batch/s]

Training batches on cuda:  27%|██▋       | 46/170 [00:00<00:02, 52.15batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:00<00:02, 52.99batch/s]

Training batches on cuda:  34%|███▍      | 58/170 [00:01<00:02, 53.20batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:01<00:02, 52.96batch/s]

Training batches on cuda:  41%|████      | 70/170 [00:01<00:01, 52.09batch/s]

Training batches on cuda:  45%|████▍     | 76/170 [00:01<00:01, 51.99batch/s]

Training batches on cuda:  48%|████▊     | 82/170 [00:01<00:01, 52.42batch/s]

Training batches on cuda:  52%|█████▏    | 88/170 [00:01<00:01, 52.07batch/s]

Training batches on cuda:  55%|█████▌    | 94/170 [00:01<00:01, 50.06batch/s]

Training batches on cuda:  59%|█████▉    | 100/170 [00:01<00:01, 50.87batch/s]

Training batches on cuda:  62%|██████▏   | 106/170 [00:02<00:01, 49.48batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:02<00:01, 48.42batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:02<00:01, 48.90batch/s]

Training batches on cuda:  72%|███████▏  | 122/170 [00:02<00:00, 48.92batch/s]

Training batches on cuda:  75%|███████▍  | 127/170 [00:02<00:00, 48.50batch/s]

Training batches on cuda:  78%|███████▊  | 133/170 [00:02<00:00, 49.47batch/s]

Training batches on cuda:  81%|████████  | 138/170 [00:02<00:00, 48.18batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:02<00:00, 47.49batch/s]

Training batches on cuda:  87%|████████▋ | 148/170 [00:02<00:00, 47.69batch/s]

Training batches on cuda:  90%|█████████ | 153/170 [00:03<00:00, 47.37batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:03<00:00, 45.73batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:03<00:00, 46.15batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:03<00:00, 47.40batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:20,  3.46s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:17,  3.52s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:04, 35.18batch/s]

Training batches on cuda:   5%|▌         | 9/170 [00:00<00:04, 38.30batch/s]

Training batches on cuda:   9%|▉         | 15/170 [00:00<00:03, 41.40batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:03, 43.51batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:00<00:03, 44.96batch/s]

Training batches on cuda:  18%|█▊        | 30/170 [00:00<00:03, 46.24batch/s]

Training batches on cuda:  21%|██        | 36/170 [00:00<00:02, 47.35batch/s]

Training batches on cuda:  25%|██▍       | 42/170 [00:00<00:02, 49.01batch/s]

Training batches on cuda:  28%|██▊       | 48/170 [00:00<00:02, 50.00batch/s]

Training batches on cuda:  31%|███       | 53/170 [00:01<00:02, 49.38batch/s]

Training batches on cuda:  35%|███▍      | 59/170 [00:01<00:02, 49.89batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:01<00:02, 49.56batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:01<00:02, 49.27batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:01<00:02, 47.49batch/s]

Training batches on cuda:  46%|████▋     | 79/170 [00:01<00:02, 43.52batch/s]

Training batches on cuda:  50%|█████     | 85/170 [00:01<00:01, 45.47batch/s]

Training batches on cuda:  54%|█████▎    | 91/170 [00:01<00:01, 47.71batch/s]

Training batches on cuda:  57%|█████▋    | 97/170 [00:02<00:01, 49.59batch/s]

Training batches on cuda:  61%|██████    | 103/170 [00:02<00:01, 50.56batch/s]

Training batches on cuda:  64%|██████▍   | 109/170 [00:02<00:01, 51.18batch/s]

Training batches on cuda:  68%|██████▊   | 115/170 [00:02<00:01, 50.50batch/s]

Training batches on cuda:  71%|███████   | 121/170 [00:02<00:00, 50.68batch/s]

Training batches on cuda:  75%|███████▍  | 127/170 [00:02<00:00, 51.05batch/s]

Training batches on cuda:  78%|███████▊  | 133/170 [00:02<00:00, 51.64batch/s]

Training batches on cuda:  82%|████████▏ | 139/170 [00:02<00:00, 51.91batch/s]

Training batches on cuda:  85%|████████▌ | 145/170 [00:02<00:00, 51.49batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:03<00:00, 50.33batch/s]

Training batches on cuda:  92%|█████████▏| 157/170 [00:03<00:00, 50.56batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:03<00:00, 50.41batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:03<00:00, 51.01batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:21<00:17,  3.52s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  60%|██████    | 6/10 [00:21<00:14,  3.56s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/170 [00:00<00:09, 17.10batch/s]

Training batches on cuda:   4%|▍         | 7/170 [00:00<00:07, 21.09batch/s]

Training batches on cuda:   7%|▋         | 12/170 [00:00<00:06, 25.32batch/s]

Training batches on cuda:  10%|█         | 17/170 [00:00<00:05, 28.99batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:04, 33.06batch/s]

Training batches on cuda:  16%|█▌        | 27/170 [00:00<00:03, 36.15batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:00<00:03, 38.44batch/s]

Training batches on cuda:  22%|██▏       | 37/170 [00:00<00:03, 40.96batch/s]

Training batches on cuda:  25%|██▍       | 42/170 [00:00<00:02, 43.12batch/s]

Training batches on cuda:  28%|██▊       | 48/170 [00:01<00:02, 45.41batch/s]

Training batches on cuda:  32%|███▏      | 54/170 [00:01<00:02, 47.18batch/s]

Training batches on cuda:  35%|███▌      | 60/170 [00:01<00:02, 48.99batch/s]

Training batches on cuda:  39%|███▉      | 66/170 [00:01<00:02, 50.27batch/s]

Training batches on cuda:  42%|████▏     | 72/170 [00:01<00:01, 50.68batch/s]

Training batches on cuda:  46%|████▌     | 78/170 [00:01<00:01, 48.75batch/s]

Training batches on cuda:  49%|████▉     | 84/170 [00:01<00:01, 50.11batch/s]

Training batches on cuda:  53%|█████▎    | 90/170 [00:01<00:01, 48.86batch/s]

Training batches on cuda:  56%|█████▋    | 96/170 [00:02<00:01, 49.04batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:02<00:01, 45.71batch/s]

Training batches on cuda:  63%|██████▎   | 107/170 [00:02<00:01, 47.16batch/s]

Training batches on cuda:  66%|██████▋   | 113/170 [00:02<00:01, 48.80batch/s]

Training batches on cuda:  69%|██████▉   | 118/170 [00:02<00:01, 48.88batch/s]

Training batches on cuda:  73%|███████▎  | 124/170 [00:02<00:00, 48.68batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:02<00:00, 46.82batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:02<00:00, 47.26batch/s]

Training batches on cuda:  82%|████████▏ | 139/170 [00:02<00:00, 47.23batch/s]

Training batches on cuda:  85%|████████▍ | 144/170 [00:03<00:00, 47.92batch/s]

Training batches on cuda:  88%|████████▊ | 149/170 [00:03<00:00, 48.50batch/s]

Training batches on cuda:  91%|█████████ | 154/170 [00:03<00:00, 47.44batch/s]

Training batches on cuda:  94%|█████████▍| 160/170 [00:03<00:00, 49.36batch/s]

Training batches on cuda:  98%|█████████▊| 166/170 [00:03<00:00, 50.41batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:25<00:14,  3.56s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  70%|███████   | 7/10 [00:25<00:10,  3.64s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/170 [00:00<00:04, 35.55batch/s]

Training batches on cuda:   5%|▌         | 9/170 [00:00<00:04, 38.76batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:03, 39.69batch/s]

Training batches on cuda:  11%|█         | 19/170 [00:00<00:03, 42.16batch/s]

Training batches on cuda:  14%|█▍        | 24/170 [00:00<00:03, 44.15batch/s]

Training batches on cuda:  18%|█▊        | 30/170 [00:00<00:03, 45.83batch/s]

Training batches on cuda:  21%|██        | 36/170 [00:00<00:02, 46.82batch/s]

Training batches on cuda:  24%|██▍       | 41/170 [00:00<00:02, 46.92batch/s]

Training batches on cuda:  27%|██▋       | 46/170 [00:00<00:02, 47.22batch/s]

Training batches on cuda:  30%|███       | 51/170 [00:01<00:02, 47.54batch/s]

Training batches on cuda:  34%|███▎      | 57/170 [00:01<00:02, 48.33batch/s]

Training batches on cuda:  37%|███▋      | 63/170 [00:01<00:02, 49.29batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:01<00:01, 50.54batch/s]

Training batches on cuda:  44%|████▍     | 75/170 [00:01<00:01, 51.44batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:01<00:01, 51.15batch/s]

Training batches on cuda:  51%|█████     | 87/170 [00:01<00:01, 50.04batch/s]

Training batches on cuda:  55%|█████▍    | 93/170 [00:01<00:01, 50.51batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:02<00:01, 50.32batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:02<00:01, 51.04batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:02<00:01, 51.09batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:02<00:01, 52.20batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:02<00:00, 52.81batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:02<00:00, 53.55batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:02<00:00, 47.92batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:02<00:00, 48.98batch/s]

Training batches on cuda:  86%|████████▌ | 146/170 [00:02<00:00, 49.28batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:03<00:00, 47.94batch/s]

Training batches on cuda:  92%|█████████▏| 157/170 [00:03<00:00, 49.42batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:03<00:00, 50.26batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:03<00:00, 48.96batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:28<00:10,  3.64s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  80%|████████  | 8/10 [00:28<00:07,  3.65s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/170 [00:00<00:09, 18.52batch/s]

Training batches on cuda:   5%|▍         | 8/170 [00:00<00:07, 22.87batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:05, 27.14batch/s]

Training batches on cuda:  11%|█         | 18/170 [00:00<00:04, 31.36batch/s]

Training batches on cuda:  14%|█▎        | 23/170 [00:00<00:04, 34.90batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:00<00:03, 38.29batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:00<00:03, 40.24batch/s]

Training batches on cuda:  24%|██▎       | 40/170 [00:00<00:02, 43.44batch/s]

Training batches on cuda:  27%|██▋       | 46/170 [00:00<00:02, 45.91batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:01<00:02, 48.01batch/s]

Training batches on cuda:  34%|███▎      | 57/170 [00:01<00:02, 48.06batch/s]

Training batches on cuda:  37%|███▋      | 63/170 [00:01<00:02, 49.20batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:01<00:02, 49.65batch/s]

Training batches on cuda:  44%|████▍     | 75/170 [00:01<00:01, 50.45batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:01<00:01, 50.66batch/s]

Training batches on cuda:  51%|█████     | 87/170 [00:01<00:01, 51.11batch/s]

Training batches on cuda:  55%|█████▍    | 93/170 [00:01<00:01, 50.39batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:02<00:01, 50.61batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:02<00:01, 49.63batch/s]

Training batches on cuda:  65%|██████▌   | 111/170 [00:02<00:01, 50.39batch/s]

Training batches on cuda:  69%|██████▉   | 117/170 [00:02<00:01, 50.59batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:02<00:00, 50.43batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:02<00:00, 50.72batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:02<00:00, 51.49batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:02<00:00, 51.77batch/s]

Training batches on cuda:  86%|████████▋ | 147/170 [00:02<00:00, 50.40batch/s]

Training batches on cuda:  90%|█████████ | 153/170 [00:03<00:00, 48.16batch/s]

Training batches on cuda:  93%|█████████▎| 158/170 [00:03<00:00, 47.73batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:03<00:00, 46.95batch/s]

Training batches on cuda:  99%|█████████▉| 168/170 [00:03<00:00, 47.13batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:32<00:07,  3.65s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda:  90%|█████████ | 9/10 [00:32<00:03,  3.67s/epoch, loss=0.00117, prev_loss=0.00117]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/170 [00:00<00:05, 29.85batch/s]

Training batches on cuda:   5%|▌         | 9/170 [00:00<00:04, 34.15batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:04, 35.04batch/s]

Training batches on cuda:  11%|█         | 19/170 [00:00<00:03, 38.78batch/s]

Training batches on cuda:  14%|█▍        | 24/170 [00:00<00:03, 41.56batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:00<00:03, 42.01batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:00<00:03, 41.45batch/s]

Training batches on cuda:  23%|██▎       | 39/170 [00:00<00:03, 42.94batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:00<00:02, 44.41batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:01<00:02, 46.16batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:01<00:02, 47.09batch/s]

Training batches on cuda:  36%|███▌      | 61/170 [00:01<00:02, 47.11batch/s]

Training batches on cuda:  39%|███▉      | 67/170 [00:01<00:02, 48.09batch/s]

Training batches on cuda:  43%|████▎     | 73/170 [00:01<00:01, 48.78batch/s]

Training batches on cuda:  46%|████▋     | 79/170 [00:01<00:01, 49.30batch/s]

Training batches on cuda:  49%|████▉     | 84/170 [00:01<00:01, 48.93batch/s]

Training batches on cuda:  52%|█████▏    | 89/170 [00:01<00:01, 48.82batch/s]

Training batches on cuda:  55%|█████▌    | 94/170 [00:02<00:01, 48.59batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:02<00:01, 48.35batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:02<00:01, 48.81batch/s]

Training batches on cuda:  64%|██████▍   | 109/170 [00:02<00:01, 48.71batch/s]

Training batches on cuda:  68%|██████▊   | 115/170 [00:02<00:01, 49.76batch/s]

Training batches on cuda:  71%|███████   | 121/170 [00:02<00:00, 50.91batch/s]

Training batches on cuda:  75%|███████▍  | 127/170 [00:02<00:00, 51.78batch/s]

Training batches on cuda:  78%|███████▊  | 133/170 [00:02<00:00, 52.78batch/s]

Training batches on cuda:  82%|████████▏ | 139/170 [00:02<00:00, 52.83batch/s]

Training batches on cuda:  85%|████████▌ | 145/170 [00:02<00:00, 52.65batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:03<00:00, 53.15batch/s]

Training batches on cuda:  92%|█████████▏| 157/170 [00:03<00:00, 52.63batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:03<00:00, 53.46batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:03<00:00, 53.79batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:36<00:03,  3.67s/epoch, loss=0.00117, prev_loss=0.00117]2021-09-16:19:07:35,638 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:07:36,742 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:07:51,680 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:19:07:51,797 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00,  8.53s/epoch, loss=0.00117, prev_loss=0.00117]
Training epochs on cuda: 100%|██████████| 10/10 [00:52<00:00,  5.23s/epoch, loss=0.00117, prev_loss=0.00117]
2021-09-16:19:07:51,798 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpnu7tbiro'
2021-09-16:19:07:51,851 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpnu7tbiro' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.40ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:19:08:06,705 INFO     [evaluator.py:657] Evaluation took 14.82s seconds
[I 2021-09-16 19:08:06,710] Trial 68 finished with value: 0.002639775912776088 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.15339961188702553, 'optimizer.lr': 0.02959423498962937, 'negative_sampler.num_negs_per_pos': 18, 'training.batch_size': 1832}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:08:06,734 WARNING  [api.py:823] No random seed is specified. Setting to 75895115.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/119 [00:00<00:07, 14.72batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:06, 16.31batch/s]

Training batches on cuda:   7%|▋         | 8/119 [00:00<00:06, 17.77batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:05, 18.87batch/s]

Training batches on cuda:  12%|█▏        | 14/119 [00:00<00:05, 19.63batch/s]

Training batches on cuda:  14%|█▍        | 17/119 [00:00<00:05, 20.34batch/s]

Training batches on cuda:  17%|█▋        | 20/119 [00:00<00:04, 20.56batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:01<00:04, 20.76batch/s]

Training batches on cuda:  22%|██▏       | 26/119 [00:01<00:04, 20.79batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:01<00:04, 21.13batch/s]

Training batches on cuda:  27%|██▋       | 32/119 [00:01<00:04, 21.33batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:03, 21.20batch/s]

Training batches on cuda:  32%|███▏      | 38/119 [00:01<00:03, 21.14batch/s]

Training batches on cuda:  34%|███▍      | 41/119 [00:01<00:03, 21.07batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:02<00:03, 20.80batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:02<00:03, 20.75batch/s]

Training batches on cuda:  42%|████▏     | 50/119 [00:02<00:03, 20.54batch/s]

Training batches on cuda:  45%|████▍     | 53/119 [00:02<00:03, 19.77batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:02<00:03, 19.90batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:02<00:03, 19.98batch/s]

Training batches on cuda:  52%|█████▏    | 62/119 [00:02<00:02, 20.18batch/s]

Training batches on cuda:  55%|█████▍    | 65/119 [00:03<00:02, 20.22batch/s]

Training batches on cuda:  57%|█████▋    | 68/119 [00:03<00:02, 20.46batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:03<00:02, 20.80batch/s]

Training batches on cuda:  62%|██████▏   | 74/119 [00:03<00:02, 20.90batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:03<00:02, 20.94batch/s]

Training batches on cuda:  67%|██████▋   | 80/119 [00:03<00:01, 21.01batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:03<00:01, 21.01batch/s]

Training batches on cuda:  72%|███████▏  | 86/119 [00:04<00:01, 21.12batch/s]

Training batches on cuda:  75%|███████▍  | 89/119 [00:04<00:01, 21.23batch/s]

Training batches on cuda:  77%|███████▋  | 92/119 [00:04<00:01, 20.68batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:04<00:01, 20.39batch/s]

Training batches on cuda:  82%|████████▏ | 98/119 [00:04<00:01, 20.11batch/s]

Training batches on cuda:  85%|████████▍ | 101/119 [00:04<00:00, 20.58batch/s]

Training batches on cuda:  87%|████████▋ | 104/119 [00:05<00:00, 20.76batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:05<00:00, 19.57batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:05<00:00, 19.68batch/s]

Training batches on cuda:  94%|█████████▍| 112/119 [00:05<00:00, 20.10batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:05<00:00, 20.19batch/s]

Training batches on cuda:  99%|█████████▉| 118/119 [00:05<00:00, 20.78batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.000507, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:53,  5.97s/epoch, loss=0.000507, prev_loss=nan]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/119 [00:00<00:15,  7.41batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:12,  9.21batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:10, 11.14batch/s]

Training batches on cuda:   8%|▊         | 10/119 [00:00<00:08, 13.04batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:07, 14.83batch/s]

Training batches on cuda:  13%|█▎        | 16/119 [00:00<00:06, 16.17batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:05, 17.53batch/s]

Training batches on cuda:  18%|█▊        | 22/119 [00:01<00:05, 18.60batch/s]

Training batches on cuda:  21%|██        | 25/119 [00:01<00:04, 19.57batch/s]

Training batches on cuda:  24%|██▎       | 28/119 [00:01<00:04, 19.89batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:01<00:04, 20.43batch/s]

Training batches on cuda:  29%|██▊       | 34/119 [00:01<00:04, 20.17batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:01<00:03, 20.54batch/s]

Training batches on cuda:  34%|███▎      | 40/119 [00:01<00:03, 20.59batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:02<00:03, 20.82batch/s]

Training batches on cuda:  39%|███▊      | 46/119 [00:02<00:03, 20.80batch/s]

Training batches on cuda:  41%|████      | 49/119 [00:02<00:03, 20.40batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:02<00:03, 20.71batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:02<00:03, 21.09batch/s]

Training batches on cuda:  49%|████▊     | 58/119 [00:02<00:02, 21.22batch/s]

Training batches on cuda:  51%|█████▏    | 61/119 [00:02<00:02, 21.29batch/s]

Training batches on cuda:  54%|█████▍    | 64/119 [00:03<00:02, 21.20batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:03<00:02, 21.16batch/s]

Training batches on cuda:  59%|█████▉    | 70/119 [00:03<00:02, 21.39batch/s]

Training batches on cuda:  61%|██████▏   | 73/119 [00:03<00:02, 21.63batch/s]

Training batches on cuda:  64%|██████▍   | 76/119 [00:03<00:02, 21.26batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:03<00:01, 21.35batch/s]

Training batches on cuda:  69%|██████▉   | 82/119 [00:03<00:01, 21.35batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:04<00:01, 21.22batch/s]

Training batches on cuda:  74%|███████▍  | 88/119 [00:04<00:01, 20.02batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:04<00:01, 20.44batch/s]

Training batches on cuda:  79%|███████▉  | 94/119 [00:04<00:01, 20.75batch/s]

Training batches on cuda:  82%|████████▏ | 97/119 [00:04<00:01, 21.04batch/s]

Training batches on cuda:  84%|████████▍ | 100/119 [00:04<00:00, 20.99batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:04<00:00, 21.23batch/s]

Training batches on cuda:  89%|████████▉ | 106/119 [00:05<00:00, 21.29batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:05<00:00, 21.59batch/s]

Training batches on cuda:  94%|█████████▍| 112/119 [00:05<00:00, 21.29batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:05<00:00, 21.12batch/s]

Training batches on cuda:  99%|█████████▉| 118/119 [00:05<00:00, 21.21batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:11<00:53,  5.97s/epoch, loss=0.000504, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:47,  5.94s/epoch, loss=0.000504, prev_loss=nan]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/119 [00:00<00:12,  9.26batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:10, 10.98batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:08, 12.90batch/s]

Training batches on cuda:   8%|▊         | 10/119 [00:00<00:07, 14.68batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:06, 16.19batch/s]

Training batches on cuda:  13%|█▎        | 16/119 [00:00<00:05, 17.34batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:05, 18.39batch/s]

Training batches on cuda:  18%|█▊        | 22/119 [00:01<00:05, 19.03batch/s]

Training batches on cuda:  21%|██        | 25/119 [00:01<00:04, 19.51batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:01<00:04, 19.44batch/s]

Training batches on cuda:  25%|██▌       | 30/119 [00:01<00:04, 19.86batch/s]

Training batches on cuda:  28%|██▊       | 33/119 [00:01<00:04, 20.31batch/s]

Training batches on cuda:  30%|███       | 36/119 [00:01<00:04, 20.55batch/s]

Training batches on cuda:  33%|███▎      | 39/119 [00:01<00:03, 20.96batch/s]

Training batches on cuda:  35%|███▌      | 42/119 [00:02<00:03, 21.22batch/s]

Training batches on cuda:  38%|███▊      | 45/119 [00:02<00:03, 21.51batch/s]

Training batches on cuda:  40%|████      | 48/119 [00:02<00:03, 21.71batch/s]

Training batches on cuda:  43%|████▎     | 51/119 [00:02<00:03, 21.77batch/s]

Training batches on cuda:  45%|████▌     | 54/119 [00:02<00:02, 21.86batch/s]

Training batches on cuda:  48%|████▊     | 57/119 [00:02<00:02, 21.85batch/s]

Training batches on cuda:  50%|█████     | 60/119 [00:02<00:02, 21.42batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:03<00:02, 21.18batch/s]

Training batches on cuda:  55%|█████▌    | 66/119 [00:03<00:02, 21.21batch/s]

Training batches on cuda:  58%|█████▊    | 69/119 [00:03<00:02, 21.55batch/s]

Training batches on cuda:  61%|██████    | 72/119 [00:03<00:02, 21.62batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:03<00:02, 21.57batch/s]

Training batches on cuda:  66%|██████▌   | 78/119 [00:03<00:01, 21.72batch/s]

Training batches on cuda:  68%|██████▊   | 81/119 [00:03<00:01, 21.89batch/s]

Training batches on cuda:  71%|███████   | 84/119 [00:03<00:01, 21.95batch/s]

Training batches on cuda:  73%|███████▎  | 87/119 [00:04<00:01, 21.90batch/s]

Training batches on cuda:  76%|███████▌  | 90/119 [00:04<00:01, 22.03batch/s]

Training batches on cuda:  78%|███████▊  | 93/119 [00:04<00:01, 21.43batch/s]

Training batches on cuda:  81%|████████  | 96/119 [00:04<00:01, 21.33batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:04<00:00, 21.33batch/s]

Training batches on cuda:  86%|████████▌ | 102/119 [00:04<00:00, 21.18batch/s]

Training batches on cuda:  88%|████████▊ | 105/119 [00:04<00:00, 21.09batch/s]

Training batches on cuda:  91%|█████████ | 108/119 [00:05<00:00, 21.01batch/s]

Training batches on cuda:  93%|█████████▎| 111/119 [00:05<00:00, 21.04batch/s]

Training batches on cuda:  96%|█████████▌| 114/119 [00:05<00:00, 20.82batch/s]

Training batches on cuda:  98%|█████████▊| 117/119 [00:05<00:00, 20.83batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:17<00:47,  5.94s/epoch, loss=0.000504, prev_loss=0.000504]
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:41,  5.93s/epoch, loss=0.000504, prev_loss=0.000504]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/119 [00:00<00:12,  9.54batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:10, 11.35batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:08, 13.14batch/s]

Training batches on cuda:   8%|▊         | 10/119 [00:00<00:07, 14.80batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:06, 16.20batch/s]

Training batches on cuda:  13%|█▎        | 16/119 [00:00<00:05, 17.35batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:05, 18.28batch/s]

Training batches on cuda:  18%|█▊        | 22/119 [00:01<00:05, 18.99batch/s]

Training batches on cuda:  21%|██        | 25/119 [00:01<00:04, 19.81batch/s]

Training batches on cuda:  24%|██▎       | 28/119 [00:01<00:04, 19.99batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:01<00:04, 20.12batch/s]

Training batches on cuda:  29%|██▊       | 34/119 [00:01<00:04, 20.10batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:01<00:04, 20.36batch/s]

Training batches on cuda:  34%|███▎      | 40/119 [00:01<00:03, 20.78batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:02<00:03, 21.26batch/s]

Training batches on cuda:  39%|███▊      | 46/119 [00:02<00:03, 20.92batch/s]

Training batches on cuda:  41%|████      | 49/119 [00:02<00:03, 20.75batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:02<00:03, 20.72batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:02<00:03, 20.74batch/s]

Training batches on cuda:  49%|████▊     | 58/119 [00:02<00:02, 20.72batch/s]

Training batches on cuda:  51%|█████▏    | 61/119 [00:02<00:02, 20.87batch/s]

Training batches on cuda:  54%|█████▍    | 64/119 [00:03<00:02, 20.78batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:03<00:02, 20.57batch/s]

Training batches on cuda:  59%|█████▉    | 70/119 [00:03<00:02, 20.74batch/s]

Training batches on cuda:  61%|██████▏   | 73/119 [00:03<00:02, 20.54batch/s]

Training batches on cuda:  64%|██████▍   | 76/119 [00:03<00:02, 20.52batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:03<00:01, 20.62batch/s]

Training batches on cuda:  69%|██████▉   | 82/119 [00:03<00:01, 20.79batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:04<00:01, 20.89batch/s]

Training batches on cuda:  74%|███████▍  | 88/119 [00:04<00:01, 20.94batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:04<00:01, 21.01batch/s]

Training batches on cuda:  79%|███████▉  | 94/119 [00:04<00:01, 21.19batch/s]

Training batches on cuda:  82%|████████▏ | 97/119 [00:04<00:01, 21.14batch/s]

Training batches on cuda:  84%|████████▍ | 100/119 [00:04<00:00, 21.30batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:04<00:00, 21.45batch/s]

Training batches on cuda:  89%|████████▉ | 106/119 [00:05<00:00, 21.36batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:05<00:00, 20.96batch/s]

Training batches on cuda:  94%|█████████▍| 112/119 [00:05<00:00, 21.07batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:05<00:00, 21.02batch/s]

Training batches on cuda:  99%|█████████▉| 118/119 [00:05<00:00, 21.29batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:23<00:41,  5.93s/epoch, loss=0.000504, prev_loss=0.000504]
Training epochs on cuda:  40%|████      | 4/10 [00:23<00:35,  5.95s/epoch, loss=0.000504, prev_loss=0.000504]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/119 [00:00<00:12,  9.26batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:10, 11.11batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:08, 13.01batch/s]

Training batches on cuda:   8%|▊         | 9/119 [00:00<00:07, 14.32batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:06, 15.59batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:06, 15.78batch/s]

Training batches on cuda:  13%|█▎        | 15/119 [00:00<00:06, 16.38batch/s]

Training batches on cuda:  14%|█▍        | 17/119 [00:00<00:05, 17.26batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:01<00:05, 17.96batch/s]

Training batches on cuda:  18%|█▊        | 22/119 [00:01<00:05, 18.53batch/s]

Training batches on cuda:  21%|██        | 25/119 [00:01<00:04, 19.05batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:01<00:04, 19.31batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:01<00:04, 18.48batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:01<00:04, 18.78batch/s]

Training batches on cuda:  29%|██▊       | 34/119 [00:01<00:04, 19.41batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:01<00:04, 19.85batch/s]

Training batches on cuda:  34%|███▎      | 40/119 [00:02<00:03, 20.32batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:02<00:03, 20.60batch/s]

Training batches on cuda:  39%|███▊      | 46/119 [00:02<00:03, 20.77batch/s]

Training batches on cuda:  41%|████      | 49/119 [00:02<00:03, 21.02batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:02<00:03, 20.85batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:02<00:03, 20.97batch/s]

Training batches on cuda:  49%|████▊     | 58/119 [00:02<00:02, 20.68batch/s]

Training batches on cuda:  51%|█████▏    | 61/119 [00:03<00:02, 20.82batch/s]

Training batches on cuda:  54%|█████▍    | 64/119 [00:03<00:02, 20.80batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:03<00:02, 20.71batch/s]

Training batches on cuda:  59%|█████▉    | 70/119 [00:03<00:02, 20.71batch/s]

Training batches on cuda:  61%|██████▏   | 73/119 [00:03<00:02, 20.76batch/s]

Training batches on cuda:  64%|██████▍   | 76/119 [00:03<00:02, 20.96batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:03<00:01, 21.02batch/s]

Training batches on cuda:  69%|██████▉   | 82/119 [00:04<00:01, 21.18batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:04<00:01, 20.99batch/s]

Training batches on cuda:  74%|███████▍  | 88/119 [00:04<00:01, 21.21batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:04<00:01, 21.39batch/s]

Training batches on cuda:  79%|███████▉  | 94/119 [00:04<00:01, 21.71batch/s]

Training batches on cuda:  82%|████████▏ | 97/119 [00:04<00:01, 21.76batch/s]

Training batches on cuda:  84%|████████▍ | 100/119 [00:04<00:00, 21.58batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:05<00:00, 21.47batch/s]

Training batches on cuda:  89%|████████▉ | 106/119 [00:05<00:00, 21.22batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:05<00:00, 21.29batch/s]

Training batches on cuda:  94%|█████████▍| 112/119 [00:05<00:00, 21.36batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:05<00:00, 21.20batch/s]

Training batches on cuda:  99%|█████████▉| 118/119 [00:05<00:00, 21.31batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:29<00:35,  5.95s/epoch, loss=0.000503, prev_loss=0.000504]
Training epochs on cuda:  50%|█████     | 5/10 [00:29<00:29,  6.00s/epoch, loss=0.000503, prev_loss=0.000504]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/119 [00:00<00:12,  9.13batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:10, 10.99batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:08, 12.89batch/s]

Training batches on cuda:   8%|▊         | 10/119 [00:00<00:07, 14.66batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:06, 16.06batch/s]

Training batches on cuda:  13%|█▎        | 16/119 [00:00<00:05, 17.37batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:05, 18.33batch/s]

Training batches on cuda:  18%|█▊        | 22/119 [00:01<00:05, 19.07batch/s]

Training batches on cuda:  21%|██        | 25/119 [00:01<00:04, 19.70batch/s]

Training batches on cuda:  24%|██▎       | 28/119 [00:01<00:04, 20.13batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:01<00:04, 20.52batch/s]

Training batches on cuda:  29%|██▊       | 34/119 [00:01<00:04, 20.83batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:01<00:03, 20.98batch/s]

Training batches on cuda:  34%|███▎      | 40/119 [00:01<00:03, 21.13batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:02<00:03, 21.22batch/s]

Training batches on cuda:  39%|███▊      | 46/119 [00:02<00:03, 21.14batch/s]

Training batches on cuda:  41%|████      | 49/119 [00:02<00:03, 21.02batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:02<00:03, 21.09batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:02<00:03, 21.27batch/s]

Training batches on cuda:  49%|████▊     | 58/119 [00:02<00:02, 21.42batch/s]

Training batches on cuda:  51%|█████▏    | 61/119 [00:02<00:02, 21.65batch/s]

Training batches on cuda:  54%|█████▍    | 64/119 [00:03<00:02, 22.02batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:03<00:02, 22.03batch/s]

Training batches on cuda:  59%|█████▉    | 70/119 [00:03<00:02, 22.01batch/s]

Training batches on cuda:  61%|██████▏   | 73/119 [00:03<00:02, 22.04batch/s]

Training batches on cuda:  64%|██████▍   | 76/119 [00:03<00:01, 22.09batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:03<00:01, 22.08batch/s]

Training batches on cuda:  69%|██████▉   | 82/119 [00:03<00:01, 21.99batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:04<00:01, 21.91batch/s]

Training batches on cuda:  74%|███████▍  | 88/119 [00:04<00:01, 21.78batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:04<00:01, 21.74batch/s]

Training batches on cuda:  79%|███████▉  | 94/119 [00:04<00:01, 21.74batch/s]

Training batches on cuda:  82%|████████▏ | 97/119 [00:04<00:01, 21.44batch/s]

Training batches on cuda:  84%|████████▍ | 100/119 [00:04<00:00, 21.36batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:04<00:00, 21.45batch/s]

Training batches on cuda:  89%|████████▉ | 106/119 [00:04<00:00, 21.50batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:05<00:00, 21.23batch/s]

Training batches on cuda:  94%|█████████▍| 112/119 [00:05<00:00, 21.25batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:05<00:00, 21.13batch/s]

Training batches on cuda:  99%|█████████▉| 118/119 [00:05<00:00, 21.03batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:35<00:29,  6.00s/epoch, loss=0.000502, prev_loss=0.000503]
Training epochs on cuda:  60%|██████    | 6/10 [00:35<00:23,  5.97s/epoch, loss=0.000502, prev_loss=0.000503]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/119 [00:00<00:12,  9.41batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:10, 11.24batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:08, 13.08batch/s]

Training batches on cuda:   8%|▊         | 10/119 [00:00<00:07, 14.83batch/s]

Training batches on cuda:  10%|█         | 12/119 [00:00<00:06, 15.78batch/s]

Training batches on cuda:  12%|█▏        | 14/119 [00:00<00:06, 16.84batch/s]

Training batches on cuda:  14%|█▍        | 17/119 [00:00<00:05, 17.88batch/s]

Training batches on cuda:  17%|█▋        | 20/119 [00:01<00:05, 19.00batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:01<00:04, 19.53batch/s]

Training batches on cuda:  22%|██▏       | 26/119 [00:01<00:04, 19.96batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:01<00:04, 20.27batch/s]

Training batches on cuda:  27%|██▋       | 32/119 [00:01<00:04, 20.93batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:03, 21.20batch/s]

Training batches on cuda:  32%|███▏      | 38/119 [00:01<00:03, 21.39batch/s]

Training batches on cuda:  34%|███▍      | 41/119 [00:01<00:03, 21.42batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:02<00:03, 21.61batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:02<00:03, 21.23batch/s]

Training batches on cuda:  42%|████▏     | 50/119 [00:02<00:03, 21.16batch/s]

Training batches on cuda:  45%|████▍     | 53/119 [00:02<00:03, 21.37batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:02<00:02, 21.24batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:02<00:02, 20.73batch/s]

Training batches on cuda:  52%|█████▏    | 62/119 [00:02<00:02, 20.77batch/s]

Training batches on cuda:  55%|█████▍    | 65/119 [00:03<00:02, 20.72batch/s]

Training batches on cuda:  57%|█████▋    | 68/119 [00:03<00:02, 20.43batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:03<00:02, 20.56batch/s]

Training batches on cuda:  62%|██████▏   | 74/119 [00:03<00:02, 20.80batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:03<00:02, 20.95batch/s]

Training batches on cuda:  67%|██████▋   | 80/119 [00:03<00:01, 21.19batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:03<00:01, 21.34batch/s]

Training batches on cuda:  72%|███████▏  | 86/119 [00:04<00:01, 21.30batch/s]

Training batches on cuda:  75%|███████▍  | 89/119 [00:04<00:01, 20.93batch/s]

Training batches on cuda:  77%|███████▋  | 92/119 [00:04<00:01, 20.25batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:04<00:01, 20.68batch/s]

Training batches on cuda:  82%|████████▏ | 98/119 [00:04<00:01, 20.71batch/s]

Training batches on cuda:  85%|████████▍ | 101/119 [00:04<00:00, 20.83batch/s]

Training batches on cuda:  87%|████████▋ | 104/119 [00:05<00:00, 20.98batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:05<00:00, 21.23batch/s]

Training batches on cuda:  92%|█████████▏| 110/119 [00:05<00:00, 21.38batch/s]

Training batches on cuda:  95%|█████████▍| 113/119 [00:05<00:00, 21.66batch/s]

Training batches on cuda:  97%|█████████▋| 116/119 [00:05<00:00, 21.42batch/s]

Training batches on cuda: 100%|██████████| 119/119 [00:05<00:00, 22.85batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:41<00:23,  5.97s/epoch, loss=0.000497, prev_loss=0.000502]
Training epochs on cuda:  70%|███████   | 7/10 [00:41<00:17,  5.98s/epoch, loss=0.000497, prev_loss=0.000502]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/119 [00:00<00:07, 15.54batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:06, 16.90batch/s]

Training batches on cuda:   7%|▋         | 8/119 [00:00<00:06, 18.19batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:05, 19.33batch/s]

Training batches on cuda:  12%|█▏        | 14/119 [00:00<00:05, 20.08batch/s]

Training batches on cuda:  14%|█▍        | 17/119 [00:00<00:04, 20.83batch/s]

Training batches on cuda:  17%|█▋        | 20/119 [00:00<00:04, 20.96batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:01<00:04, 20.83batch/s]

Training batches on cuda:  22%|██▏       | 26/119 [00:01<00:04, 21.12batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:01<00:04, 21.10batch/s]

Training batches on cuda:  27%|██▋       | 32/119 [00:01<00:04, 20.71batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:04, 20.95batch/s]

Training batches on cuda:  32%|███▏      | 38/119 [00:01<00:03, 21.15batch/s]

Training batches on cuda:  34%|███▍      | 41/119 [00:01<00:03, 21.21batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:02<00:03, 21.34batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:02<00:03, 21.02batch/s]

Training batches on cuda:  42%|████▏     | 50/119 [00:02<00:03, 21.08batch/s]

Training batches on cuda:  45%|████▍     | 53/119 [00:02<00:03, 21.31batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:02<00:02, 21.36batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:02<00:02, 21.19batch/s]

Training batches on cuda:  52%|█████▏    | 62/119 [00:02<00:02, 20.75batch/s]

Training batches on cuda:  55%|█████▍    | 65/119 [00:03<00:02, 20.57batch/s]

Training batches on cuda:  57%|█████▋    | 68/119 [00:03<00:02, 20.60batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:03<00:02, 20.61batch/s]

Training batches on cuda:  62%|██████▏   | 74/119 [00:03<00:02, 20.55batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:03<00:02, 20.39batch/s]

Training batches on cuda:  67%|██████▋   | 80/119 [00:03<00:01, 20.49batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:03<00:01, 20.40batch/s]

Training batches on cuda:  72%|███████▏  | 86/119 [00:04<00:01, 20.55batch/s]

Training batches on cuda:  75%|███████▍  | 89/119 [00:04<00:01, 20.69batch/s]

Training batches on cuda:  77%|███████▋  | 92/119 [00:04<00:01, 20.69batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:04<00:01, 20.41batch/s]

Training batches on cuda:  82%|████████▏ | 98/119 [00:04<00:01, 20.51batch/s]

Training batches on cuda:  85%|████████▍ | 101/119 [00:04<00:00, 20.61batch/s]

Training batches on cuda:  87%|████████▋ | 104/119 [00:04<00:00, 20.54batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:05<00:00, 20.49batch/s]

Training batches on cuda:  92%|█████████▏| 110/119 [00:05<00:00, 20.25batch/s]

Training batches on cuda:  95%|█████████▍| 113/119 [00:05<00:00, 20.45batch/s]

Training batches on cuda:  97%|█████████▋| 116/119 [00:05<00:00, 20.21batch/s]

Training batches on cuda: 100%|██████████| 119/119 [00:05<00:00, 21.77batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:47<00:17,  5.98s/epoch, loss=0.000485, prev_loss=0.000497]
Training epochs on cuda:  80%|████████  | 8/10 [00:47<00:11,  5.98s/epoch, loss=0.000485, prev_loss=0.000497]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/119 [00:00<00:13,  8.52batch/s]

Training batches on cuda:   3%|▎         | 3/119 [00:00<00:11, 10.24batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:09, 11.97batch/s]

Training batches on cuda:   7%|▋         | 8/119 [00:00<00:08, 13.86batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:06, 15.61batch/s]

Training batches on cuda:  12%|█▏        | 14/119 [00:00<00:06, 16.93batch/s]

Training batches on cuda:  14%|█▍        | 17/119 [00:00<00:05, 18.35batch/s]

Training batches on cuda:  17%|█▋        | 20/119 [00:01<00:05, 19.05batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:01<00:04, 19.52batch/s]

Training batches on cuda:  22%|██▏       | 26/119 [00:01<00:04, 19.78batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:01<00:04, 20.08batch/s]

Training batches on cuda:  27%|██▋       | 32/119 [00:01<00:04, 20.71batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:03, 21.25batch/s]

Training batches on cuda:  32%|███▏      | 38/119 [00:01<00:03, 21.50batch/s]

Training batches on cuda:  34%|███▍      | 41/119 [00:01<00:03, 21.45batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:02<00:03, 21.67batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:02<00:03, 21.59batch/s]

Training batches on cuda:  42%|████▏     | 50/119 [00:02<00:03, 21.50batch/s]

Training batches on cuda:  45%|████▍     | 53/119 [00:02<00:03, 21.67batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:02<00:02, 21.69batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:02<00:03, 19.68batch/s]

Training batches on cuda:  52%|█████▏    | 62/119 [00:03<00:02, 20.34batch/s]

Training batches on cuda:  55%|█████▍    | 65/119 [00:03<00:02, 20.85batch/s]

Training batches on cuda:  57%|█████▋    | 68/119 [00:03<00:02, 20.82batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:03<00:02, 21.16batch/s]

Training batches on cuda:  62%|██████▏   | 74/119 [00:03<00:02, 21.31batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:03<00:01, 21.31batch/s]

Training batches on cuda:  67%|██████▋   | 80/119 [00:03<00:01, 21.50batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:03<00:01, 21.64batch/s]

Training batches on cuda:  72%|███████▏  | 86/119 [00:04<00:01, 21.41batch/s]

Training batches on cuda:  75%|███████▍  | 89/119 [00:04<00:01, 21.70batch/s]

Training batches on cuda:  77%|███████▋  | 92/119 [00:04<00:01, 22.02batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:04<00:01, 21.46batch/s]

Training batches on cuda:  82%|████████▏ | 98/119 [00:04<00:00, 21.79batch/s]

Training batches on cuda:  85%|████████▍ | 101/119 [00:04<00:00, 21.83batch/s]

Training batches on cuda:  87%|████████▋ | 104/119 [00:04<00:00, 22.18batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:05<00:00, 22.28batch/s]

Training batches on cuda:  92%|█████████▏| 110/119 [00:05<00:00, 22.04batch/s]

Training batches on cuda:  95%|█████████▍| 113/119 [00:05<00:00, 21.93batch/s]

Training batches on cuda:  97%|█████████▋| 116/119 [00:05<00:00, 21.78batch/s]

Training batches on cuda: 100%|██████████| 119/119 [00:05<00:00, 22.85batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:53<00:11,  5.98s/epoch, loss=0.000464, prev_loss=0.000485]
Training epochs on cuda:  90%|█████████ | 9/10 [00:53<00:05,  5.95s/epoch, loss=0.000464, prev_loss=0.000485]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/119 [00:00<00:08, 14.22batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:07, 15.84batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:06, 16.85batch/s]

Training batches on cuda:   8%|▊         | 10/119 [00:00<00:05, 18.19batch/s]

Training batches on cuda:  10%|█         | 12/119 [00:00<00:05, 18.52batch/s]

Training batches on cuda:  13%|█▎        | 15/119 [00:00<00:05, 19.23batch/s]

Training batches on cuda:  15%|█▌        | 18/119 [00:00<00:05, 19.79batch/s]

Training batches on cuda:  18%|█▊        | 21/119 [00:01<00:04, 20.20batch/s]

Training batches on cuda:  20%|██        | 24/119 [00:01<00:04, 20.73batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:01<00:04, 20.90batch/s]

Training batches on cuda:  25%|██▌       | 30/119 [00:01<00:04, 21.08batch/s]

Training batches on cuda:  28%|██▊       | 33/119 [00:01<00:04, 21.19batch/s]

Training batches on cuda:  30%|███       | 36/119 [00:01<00:03, 21.38batch/s]

Training batches on cuda:  33%|███▎      | 39/119 [00:01<00:03, 21.41batch/s]

Training batches on cuda:  35%|███▌      | 42/119 [00:02<00:03, 21.66batch/s]

Training batches on cuda:  38%|███▊      | 45/119 [00:02<00:03, 21.87batch/s]

Training batches on cuda:  40%|████      | 48/119 [00:02<00:03, 21.92batch/s]

Training batches on cuda:  43%|████▎     | 51/119 [00:02<00:03, 21.84batch/s]

Training batches on cuda:  45%|████▌     | 54/119 [00:02<00:03, 21.36batch/s]

Training batches on cuda:  48%|████▊     | 57/119 [00:02<00:02, 21.37batch/s]

Training batches on cuda:  50%|█████     | 60/119 [00:02<00:02, 20.97batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:03<00:02, 20.87batch/s]

Training batches on cuda:  55%|█████▌    | 66/119 [00:03<00:02, 20.88batch/s]

Training batches on cuda:  58%|█████▊    | 69/119 [00:03<00:02, 20.95batch/s]

Training batches on cuda:  61%|██████    | 72/119 [00:03<00:02, 21.10batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:03<00:02, 20.84batch/s]

Training batches on cuda:  66%|██████▌   | 78/119 [00:03<00:01, 20.98batch/s]

Training batches on cuda:  68%|██████▊   | 81/119 [00:03<00:01, 21.30batch/s]

Training batches on cuda:  71%|███████   | 84/119 [00:03<00:01, 21.40batch/s]

Training batches on cuda:  73%|███████▎  | 87/119 [00:04<00:01, 21.44batch/s]

Training batches on cuda:  76%|███████▌  | 90/119 [00:04<00:01, 21.62batch/s]

Training batches on cuda:  78%|███████▊  | 93/119 [00:04<00:01, 21.80batch/s]

Training batches on cuda:  81%|████████  | 96/119 [00:04<00:01, 21.75batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:04<00:00, 21.49batch/s]

Training batches on cuda:  86%|████████▌ | 102/119 [00:04<00:00, 21.64batch/s]

Training batches on cuda:  88%|████████▊ | 105/119 [00:04<00:00, 21.78batch/s]

Training batches on cuda:  91%|█████████ | 108/119 [00:05<00:00, 21.89batch/s]

Training batches on cuda:  93%|█████████▎| 111/119 [00:05<00:00, 21.98batch/s]

Training batches on cuda:  96%|█████████▌| 114/119 [00:05<00:00, 22.07batch/s]

Training batches on cuda:  98%|█████████▊| 117/119 [00:05<00:00, 21.80batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:59<00:05,  5.95s/epoch, loss=0.000424, prev_loss=0.000464]2021-09-16:19:09:06,852 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:09:07,828 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:09:22,840 INFO     [evaluator.py:657] Evaluation took 15.01s seconds
2021-09-16:19:09:22,982 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 10.78s/epoch, loss=0.000424, prev_loss=0.000464]
Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00,  7.57s/epoch, loss=0.000424, prev_loss=0.000464]
2021-09-16:19:09:22,995 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpzrlxr41m'
2021-09-16:19:09:23,41 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpzrlxr41m' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.31ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.40ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:14, 2.40ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.47ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:13, 2.52ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.55ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.58ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.61ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.60ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.61ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:09:37,943 INFO     [evaluator.py:657] Evaluation took 14.89s seconds
[I 2021-09-16 19:09:37,948] Trial 69 finished with value: 0.8040631408391811 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.06782760069344461, 'optimizer.lr': 0.03594352890331663, 'negative_sampler.num_negs_per_pos': 34, 'training.batch_size': 2632}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:09:37,971 WARNING  [api.py:823] No random seed is specified. Setting to 595498722.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/146 [00:00<00:18,  8.04batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:13, 10.38batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:10, 13.12batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:08, 16.14batch/s]

Training batches on cuda:  12%|█▏        | 17/146 [00:00<00:06, 18.93batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:05, 21.82batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:04, 24.58batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:04, 27.10batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:03, 29.12batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:01<00:03, 30.94batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:01<00:03, 32.22batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:03, 33.19batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:02, 33.99batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:02, 34.29batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:02, 34.86batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:01<00:02, 35.13batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:01<00:02, 35.34batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:02<00:02, 35.43batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 35.56batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:02<00:01, 35.45batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:02<00:01, 35.55batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:02<00:01, 35.53batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 35.59batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:02<00:01, 35.72batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:02<00:01, 35.70batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:02<00:01, 35.74batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 35.72batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:03<00:01, 35.66batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:03<00:00, 35.70batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:03<00:00, 35.72batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:03<00:00, 35.68batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:03<00:00, 35.68batch/s]

Training batches on cuda:  88%|████████▊ | 129/146 [00:03<00:00, 35.80batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:03<00:00, 35.77batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:03<00:00, 35.57batch/s]

Training batches on cuda:  97%|█████████▋| 141/146 [00:04<00:00, 35.82batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 36.07batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000648, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:40,  4.48s/epoch, loss=0.000648, prev_loss=nan]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:07, 19.44batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:06, 20.85batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:05, 23.83batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:05, 26.50batch/s]

Training batches on cuda:  12%|█▏        | 17/146 [00:00<00:04, 28.79batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:04, 30.58batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:03, 31.91batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:03, 32.96batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:03, 33.73batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:01<00:03, 34.32batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:01<00:03, 34.72batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:02, 35.01batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:02, 35.21batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:02, 35.40batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:02, 35.23batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:01<00:02, 34.78batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:01<00:02, 34.25batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:02<00:02, 33.99batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 33.73batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:02<00:02, 33.91batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:02<00:01, 33.94batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:02<00:01, 33.70batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 33.73batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:02<00:01, 33.91batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:02<00:01, 33.66batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:02<00:01, 33.86batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 33.56batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:03<00:01, 33.32batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:03<00:01, 32.89batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:03<00:00, 33.03batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:03<00:00, 33.32batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:03<00:00, 33.01batch/s]

Training batches on cuda:  88%|████████▊ | 129/146 [00:03<00:00, 33.00batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:03<00:00, 33.17batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 31.27batch/s]

Training batches on cuda:  97%|█████████▋| 141/146 [00:04<00:00, 31.72batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 32.27batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:09<00:40,  4.48s/epoch, loss=0.00062, prev_loss=nan] 
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:36,  4.50s/epoch, loss=0.00062, prev_loss=nan]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/146 [00:00<00:05, 25.76batch/s]

Training batches on cuda:   5%|▍         | 7/146 [00:00<00:05, 27.55batch/s]

Training batches on cuda:   8%|▊         | 11/146 [00:00<00:04, 29.45batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:04, 30.98batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:03, 32.05batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:04, 28.80batch/s]

Training batches on cuda:  18%|█▊        | 26/146 [00:00<00:03, 30.11batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:00<00:03, 31.20batch/s]

Training batches on cuda:  23%|██▎       | 34/146 [00:01<00:03, 31.70batch/s]

Training batches on cuda:  26%|██▌       | 38/146 [00:01<00:03, 31.78batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:01<00:03, 32.83batch/s]

Training batches on cuda:  32%|███▏      | 46/146 [00:01<00:02, 33.76batch/s]

Training batches on cuda:  34%|███▍      | 50/146 [00:01<00:02, 33.85batch/s]

Training batches on cuda:  37%|███▋      | 54/146 [00:01<00:02, 33.71batch/s]

Training batches on cuda:  40%|███▉      | 58/146 [00:01<00:02, 33.33batch/s]

Training batches on cuda:  42%|████▏     | 62/146 [00:01<00:02, 32.12batch/s]

Training batches on cuda:  45%|████▌     | 66/146 [00:02<00:02, 32.73batch/s]

Training batches on cuda:  48%|████▊     | 70/146 [00:02<00:02, 32.94batch/s]

Training batches on cuda:  51%|█████     | 74/146 [00:02<00:02, 33.45batch/s]

Training batches on cuda:  53%|█████▎    | 78/146 [00:02<00:02, 33.73batch/s]

Training batches on cuda:  56%|█████▌    | 82/146 [00:02<00:01, 33.80batch/s]

Training batches on cuda:  59%|█████▉    | 86/146 [00:02<00:01, 34.00batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:02<00:01, 34.18batch/s]

Training batches on cuda:  64%|██████▍   | 94/146 [00:02<00:01, 34.32batch/s]

Training batches on cuda:  67%|██████▋   | 98/146 [00:02<00:01, 34.19batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:03<00:01, 34.05batch/s]

Training batches on cuda:  73%|███████▎  | 106/146 [00:03<00:01, 33.87batch/s]

Training batches on cuda:  75%|███████▌  | 110/146 [00:03<00:01, 33.73batch/s]

Training batches on cuda:  78%|███████▊  | 114/146 [00:03<00:00, 33.20batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:03<00:00, 33.25batch/s]

Training batches on cuda:  84%|████████▎ | 122/146 [00:03<00:00, 33.65batch/s]

Training batches on cuda:  86%|████████▋ | 126/146 [00:03<00:00, 33.72batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:03<00:00, 34.04batch/s]

Training batches on cuda:  92%|█████████▏| 134/146 [00:04<00:00, 33.97batch/s]

Training batches on cuda:  95%|█████████▍| 138/146 [00:04<00:00, 34.05batch/s]

Training batches on cuda:  97%|█████████▋| 142/146 [00:04<00:00, 34.14batch/s]

Training batches on cuda: 100%|██████████| 146/146 [00:04<00:00, 34.69batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:36,  4.50s/epoch, loss=0.00062, prev_loss=0.00062]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:31,  4.53s/epoch, loss=0.00062, prev_loss=0.00062]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/146 [00:00<00:19,  7.46batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:14,  9.69batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:11, 12.20batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:08, 15.05batch/s]

Training batches on cuda:  12%|█▏        | 17/146 [00:00<00:07, 17.90batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:06, 20.77batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:05, 23.15batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:01<00:04, 25.55batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:04, 25.72batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:01<00:03, 27.64batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:01<00:03, 28.96batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:03, 30.00batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:03, 30.42batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:02, 31.14batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:02, 31.52batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:02<00:02, 31.84batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:02<00:02, 32.00batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:02<00:02, 31.86batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 32.47batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:02<00:02, 32.03batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:02<00:02, 32.43batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:02<00:01, 32.78batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 33.19batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:02<00:01, 33.03batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:03<00:01, 33.15batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:03<00:01, 33.03batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 32.97batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:03<00:01, 32.04batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:03<00:01, 31.79batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:03<00:00, 32.12batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:03<00:00, 32.08batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:03<00:00, 32.23batch/s]

Training batches on cuda:  88%|████████▊ | 129/146 [00:04<00:00, 32.56batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:04<00:00, 32.89batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 32.85batch/s]

Training batches on cuda:  97%|█████████▋| 141/146 [00:04<00:00, 32.58batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 32.61batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:31,  4.53s/epoch, loss=0.00062, prev_loss=0.00062]
Training epochs on cuda:  40%|████      | 4/10 [00:18<00:27,  4.62s/epoch, loss=0.00062, prev_loss=0.00062]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/146 [00:00<00:05, 26.83batch/s]

Training batches on cuda:   5%|▍         | 7/146 [00:00<00:04, 28.71batch/s]

Training batches on cuda:   8%|▊         | 11/146 [00:00<00:04, 30.36batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:04, 31.46batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:03, 32.78batch/s]

Training batches on cuda:  16%|█▌        | 23/146 [00:00<00:03, 32.97batch/s]

Training batches on cuda:  18%|█▊        | 27/146 [00:00<00:03, 33.38batch/s]

Training batches on cuda:  21%|██        | 31/146 [00:00<00:03, 33.67batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:01<00:03, 33.78batch/s]

Training batches on cuda:  27%|██▋       | 39/146 [00:01<00:03, 33.35batch/s]

Training batches on cuda:  29%|██▉       | 43/146 [00:01<00:03, 33.44batch/s]

Training batches on cuda:  32%|███▏      | 47/146 [00:01<00:02, 33.68batch/s]

Training batches on cuda:  35%|███▍      | 51/146 [00:01<00:02, 33.69batch/s]

Training batches on cuda:  38%|███▊      | 55/146 [00:01<00:02, 34.02batch/s]

Training batches on cuda:  40%|████      | 59/146 [00:01<00:02, 33.86batch/s]

Training batches on cuda:  43%|████▎     | 63/146 [00:01<00:02, 33.54batch/s]

Training batches on cuda:  46%|████▌     | 67/146 [00:01<00:02, 33.03batch/s]

Training batches on cuda:  49%|████▊     | 71/146 [00:02<00:02, 31.85batch/s]

Training batches on cuda:  51%|█████▏    | 75/146 [00:02<00:02, 32.38batch/s]

Training batches on cuda:  54%|█████▍    | 79/146 [00:02<00:02, 32.56batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:02<00:01, 32.66batch/s]

Training batches on cuda:  60%|█████▉    | 87/146 [00:02<00:01, 32.77batch/s]

Training batches on cuda:  62%|██████▏   | 91/146 [00:02<00:01, 32.97batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:02<00:01, 33.23batch/s]

Training batches on cuda:  68%|██████▊   | 99/146 [00:02<00:01, 33.34batch/s]

Training batches on cuda:  71%|███████   | 103/146 [00:03<00:01, 33.31batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:03<00:01, 33.36batch/s]

Training batches on cuda:  76%|███████▌  | 111/146 [00:03<00:01, 32.61batch/s]

Training batches on cuda:  79%|███████▉  | 115/146 [00:03<00:00, 32.70batch/s]

Training batches on cuda:  82%|████████▏ | 119/146 [00:03<00:00, 32.85batch/s]

Training batches on cuda:  84%|████████▍ | 123/146 [00:03<00:00, 32.58batch/s]

Training batches on cuda:  87%|████████▋ | 127/146 [00:03<00:00, 32.42batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:03<00:00, 32.23batch/s]

Training batches on cuda:  92%|█████████▏| 135/146 [00:04<00:00, 32.14batch/s]

Training batches on cuda:  95%|█████████▌| 139/146 [00:04<00:00, 32.20batch/s]

Training batches on cuda:  98%|█████████▊| 143/146 [00:04<00:00, 32.17batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:23<00:27,  4.62s/epoch, loss=0.00062, prev_loss=0.00062]
Training epochs on cuda:  50%|█████     | 5/10 [00:23<00:23,  4.63s/epoch, loss=0.00062, prev_loss=0.00062]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/146 [00:00<00:06, 22.94batch/s]

Training batches on cuda:   5%|▍         | 7/146 [00:00<00:05, 25.28batch/s]

Training batches on cuda:   8%|▊         | 11/146 [00:00<00:04, 27.37batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:04, 28.86batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:04, 30.30batch/s]

Training batches on cuda:  16%|█▌        | 23/146 [00:00<00:03, 31.00batch/s]

Training batches on cuda:  18%|█▊        | 27/146 [00:00<00:03, 31.91batch/s]

Training batches on cuda:  21%|██        | 31/146 [00:00<00:03, 32.66batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:01<00:03, 32.42batch/s]

Training batches on cuda:  27%|██▋       | 39/146 [00:01<00:03, 32.37batch/s]

Training batches on cuda:  29%|██▉       | 43/146 [00:01<00:03, 32.81batch/s]

Training batches on cuda:  32%|███▏      | 47/146 [00:01<00:02, 33.26batch/s]

Training batches on cuda:  35%|███▍      | 51/146 [00:01<00:02, 33.73batch/s]

Training batches on cuda:  38%|███▊      | 55/146 [00:01<00:02, 33.89batch/s]

Training batches on cuda:  40%|████      | 59/146 [00:01<00:02, 33.49batch/s]

Training batches on cuda:  43%|████▎     | 63/146 [00:01<00:02, 33.18batch/s]

Training batches on cuda:  46%|████▌     | 67/146 [00:02<00:02, 33.13batch/s]

Training batches on cuda:  49%|████▊     | 71/146 [00:02<00:02, 33.27batch/s]

Training batches on cuda:  51%|█████▏    | 75/146 [00:02<00:02, 33.32batch/s]

Training batches on cuda:  54%|█████▍    | 79/146 [00:02<00:02, 33.20batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:02<00:01, 32.76batch/s]

Training batches on cuda:  60%|█████▉    | 87/146 [00:02<00:01, 32.52batch/s]

Training batches on cuda:  62%|██████▏   | 91/146 [00:02<00:01, 32.40batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:02<00:01, 32.36batch/s]

Training batches on cuda:  68%|██████▊   | 99/146 [00:03<00:01, 31.43batch/s]

Training batches on cuda:  71%|███████   | 103/146 [00:03<00:01, 31.62batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:03<00:01, 32.19batch/s]

Training batches on cuda:  76%|███████▌  | 111/146 [00:03<00:01, 32.53batch/s]

Training batches on cuda:  79%|███████▉  | 115/146 [00:03<00:00, 32.69batch/s]

Training batches on cuda:  82%|████████▏ | 119/146 [00:03<00:00, 32.34batch/s]

Training batches on cuda:  84%|████████▍ | 123/146 [00:03<00:00, 32.90batch/s]

Training batches on cuda:  87%|████████▋ | 127/146 [00:03<00:00, 33.65batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:04<00:00, 33.37batch/s]

Training batches on cuda:  92%|█████████▏| 135/146 [00:04<00:00, 32.43batch/s]

Training batches on cuda:  95%|█████████▌| 139/146 [00:04<00:00, 32.65batch/s]

Training batches on cuda:  98%|█████████▊| 143/146 [00:04<00:00, 32.98batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:27<00:23,  4.63s/epoch, loss=0.00062, prev_loss=0.00062]
Training epochs on cuda:  60%|██████    | 6/10 [00:27<00:18,  4.65s/epoch, loss=0.00062, prev_loss=0.00062]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/146 [00:00<00:16,  8.82batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:12, 11.28batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:09, 13.89batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:07, 16.76batch/s]

Training batches on cuda:  12%|█▏        | 17/146 [00:00<00:06, 19.68batch/s]

Training batches on cuda:  14%|█▎        | 20/146 [00:00<00:05, 21.92batch/s]

Training batches on cuda:  16%|█▋        | 24/146 [00:00<00:05, 24.27batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:04, 26.78batch/s]

Training batches on cuda:  22%|██▏       | 32/146 [00:01<00:04, 28.24batch/s]

Training batches on cuda:  25%|██▍       | 36/146 [00:01<00:03, 29.15batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:01<00:03, 29.93batch/s]

Training batches on cuda:  30%|███       | 44/146 [00:01<00:03, 30.93batch/s]

Training batches on cuda:  33%|███▎      | 48/146 [00:01<00:03, 31.07batch/s]

Training batches on cuda:  36%|███▌      | 52/146 [00:01<00:02, 31.74batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:01<00:02, 32.16batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:01<00:02, 32.45batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:02<00:02, 30.06batch/s]

Training batches on cuda:  47%|████▋     | 68/146 [00:02<00:02, 31.18batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:02<00:02, 31.79batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:02<00:02, 32.24batch/s]

Training batches on cuda:  55%|█████▍    | 80/146 [00:02<00:02, 32.31batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:02<00:01, 32.36batch/s]

Training batches on cuda:  60%|██████    | 88/146 [00:02<00:01, 32.34batch/s]

Training batches on cuda:  63%|██████▎   | 92/146 [00:02<00:01, 32.76batch/s]

Training batches on cuda:  66%|██████▌   | 96/146 [00:03<00:01, 32.37batch/s]

Training batches on cuda:  68%|██████▊   | 100/146 [00:03<00:01, 31.61batch/s]

Training batches on cuda:  71%|███████   | 104/146 [00:03<00:01, 31.55batch/s]

Training batches on cuda:  74%|███████▍  | 108/146 [00:03<00:01, 31.75batch/s]

Training batches on cuda:  77%|███████▋  | 112/146 [00:03<00:01, 32.20batch/s]

Training batches on cuda:  79%|███████▉  | 116/146 [00:03<00:00, 31.96batch/s]

Training batches on cuda:  82%|████████▏ | 120/146 [00:03<00:00, 32.35batch/s]

Training batches on cuda:  85%|████████▍ | 124/146 [00:03<00:00, 32.53batch/s]

Training batches on cuda:  88%|████████▊ | 128/146 [00:04<00:00, 32.63batch/s]

Training batches on cuda:  90%|█████████ | 132/146 [00:04<00:00, 32.68batch/s]

Training batches on cuda:  93%|█████████▎| 136/146 [00:04<00:00, 31.94batch/s]

Training batches on cuda:  96%|█████████▌| 140/146 [00:04<00:00, 31.88batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:04<00:00, 32.57batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:32<00:18,  4.65s/epoch, loss=0.00062, prev_loss=0.00062]
Training epochs on cuda:  70%|███████   | 7/10 [00:32<00:14,  4.71s/epoch, loss=0.00062, prev_loss=0.00062]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:09, 15.83batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:07, 18.24batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:06, 21.12batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:05, 23.92batch/s]

Training batches on cuda:  12%|█▏        | 17/146 [00:00<00:04, 26.38batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:04, 28.27batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:04, 29.76batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:03, 30.99batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:01<00:03, 31.82batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:01<00:03, 32.22batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:01<00:03, 32.42batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:01<00:03, 32.58batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:01<00:02, 33.01batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:02, 33.57batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:02, 33.74batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:01<00:02, 32.36batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:02<00:02, 32.60batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:02<00:02, 32.25batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:02<00:02, 32.46batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:02<00:02, 32.90batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:02<00:01, 33.22batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:02<00:01, 33.48batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:02<00:01, 33.46batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:02<00:01, 33.51batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:02<00:01, 33.59batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:03<00:01, 33.78batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:03<00:01, 33.45batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:03<00:01, 32.96batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:03<00:00, 33.08batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:03<00:00, 33.25batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:03<00:00, 33.17batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:03<00:00, 32.73batch/s]

Training batches on cuda:  88%|████████▊ | 129/146 [00:03<00:00, 33.22batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:04<00:00, 32.05batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:04<00:00, 32.06batch/s]

Training batches on cuda:  97%|█████████▋| 141/146 [00:04<00:00, 31.36batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:04<00:00, 31.90batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:37<00:14,  4.71s/epoch, loss=0.00062, prev_loss=0.00062]
Training epochs on cuda:  80%|████████  | 8/10 [00:37<00:09,  4.70s/epoch, loss=0.00062, prev_loss=0.00062]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:07, 18.85batch/s]

Training batches on cuda:   4%|▍         | 6/146 [00:00<00:06, 21.41batch/s]

Training batches on cuda:   7%|▋         | 10/146 [00:00<00:05, 24.00batch/s]

Training batches on cuda:  10%|▉         | 14/146 [00:00<00:05, 26.19batch/s]

Training batches on cuda:  12%|█▏        | 18/146 [00:00<00:04, 28.08batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:04, 29.35batch/s]

Training batches on cuda:  18%|█▊        | 26/146 [00:00<00:03, 30.35batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:00<00:03, 31.10batch/s]

Training batches on cuda:  23%|██▎       | 34/146 [00:01<00:03, 31.62batch/s]

Training batches on cuda:  26%|██▌       | 38/146 [00:01<00:03, 32.01batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:01<00:03, 32.01batch/s]

Training batches on cuda:  32%|███▏      | 46/146 [00:01<00:03, 32.17batch/s]

Training batches on cuda:  34%|███▍      | 50/146 [00:01<00:02, 32.45batch/s]

Training batches on cuda:  37%|███▋      | 54/146 [00:01<00:02, 32.78batch/s]

Training batches on cuda:  40%|███▉      | 58/146 [00:01<00:02, 32.77batch/s]

Training batches on cuda:  42%|████▏     | 62/146 [00:01<00:02, 32.84batch/s]

Training batches on cuda:  45%|████▌     | 66/146 [00:02<00:02, 32.61batch/s]

Training batches on cuda:  48%|████▊     | 70/146 [00:02<00:02, 32.92batch/s]

Training batches on cuda:  51%|█████     | 74/146 [00:02<00:02, 32.96batch/s]

Training batches on cuda:  53%|█████▎    | 78/146 [00:02<00:02, 33.30batch/s]

Training batches on cuda:  56%|█████▌    | 82/146 [00:02<00:02, 30.78batch/s]

Training batches on cuda:  59%|█████▉    | 86/146 [00:02<00:01, 31.64batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:02<00:01, 32.01batch/s]

Training batches on cuda:  64%|██████▍   | 94/146 [00:02<00:01, 32.29batch/s]

Training batches on cuda:  67%|██████▋   | 98/146 [00:03<00:01, 32.68batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:03<00:01, 32.87batch/s]

Training batches on cuda:  73%|███████▎  | 106/146 [00:03<00:01, 33.35batch/s]

Training batches on cuda:  75%|███████▌  | 110/146 [00:03<00:01, 33.45batch/s]

Training batches on cuda:  78%|███████▊  | 114/146 [00:03<00:00, 33.09batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:03<00:00, 33.63batch/s]

Training batches on cuda:  84%|████████▎ | 122/146 [00:03<00:00, 33.49batch/s]

Training batches on cuda:  86%|████████▋ | 126/146 [00:03<00:00, 33.37batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:03<00:00, 33.65batch/s]

Training batches on cuda:  92%|█████████▏| 134/146 [00:04<00:00, 33.33batch/s]

Training batches on cuda:  95%|█████████▍| 138/146 [00:04<00:00, 33.22batch/s]

Training batches on cuda:  97%|█████████▋| 142/146 [00:04<00:00, 33.24batch/s]

Training batches on cuda: 100%|██████████| 146/146 [00:04<00:00, 34.10batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:42<00:09,  4.70s/epoch, loss=0.00062, prev_loss=0.00062]
Training epochs on cuda:  90%|█████████ | 9/10 [00:42<00:04,  4.70s/epoch, loss=0.00062, prev_loss=0.00062]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:08, 17.48batch/s]

Training batches on cuda:   4%|▍         | 6/146 [00:00<00:06, 20.09batch/s]

Training batches on cuda:   7%|▋         | 10/146 [00:00<00:06, 22.58batch/s]

Training batches on cuda:  10%|▉         | 14/146 [00:00<00:05, 25.10batch/s]

Training batches on cuda:  12%|█▏        | 18/146 [00:00<00:04, 26.85batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:04, 27.67batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:04, 29.20batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:04, 28.85batch/s]

Training batches on cuda:  22%|██▏       | 32/146 [00:01<00:03, 30.43batch/s]

Training batches on cuda:  25%|██▍       | 36/146 [00:01<00:03, 31.23batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:01<00:03, 31.75batch/s]

Training batches on cuda:  30%|███       | 44/146 [00:01<00:03, 32.15batch/s]

Training batches on cuda:  33%|███▎      | 48/146 [00:01<00:02, 32.67batch/s]

Training batches on cuda:  36%|███▌      | 52/146 [00:01<00:02, 32.94batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:01<00:02, 32.78batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:01<00:02, 33.08batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:02<00:02, 33.53batch/s]

Training batches on cuda:  47%|████▋     | 68/146 [00:02<00:02, 33.72batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:02<00:02, 33.83batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:02<00:02, 33.94batch/s]

Training batches on cuda:  55%|█████▍    | 80/146 [00:02<00:01, 33.04batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:02<00:01, 32.46batch/s]

Training batches on cuda:  60%|██████    | 88/146 [00:02<00:01, 31.94batch/s]

Training batches on cuda:  63%|██████▎   | 92/146 [00:02<00:01, 30.44batch/s]

Training batches on cuda:  66%|██████▌   | 96/146 [00:03<00:01, 31.45batch/s]

Training batches on cuda:  68%|██████▊   | 100/146 [00:03<00:01, 32.14batch/s]

Training batches on cuda:  71%|███████   | 104/146 [00:03<00:01, 32.77batch/s]

Training batches on cuda:  74%|███████▍  | 108/146 [00:03<00:01, 32.94batch/s]

Training batches on cuda:  77%|███████▋  | 112/146 [00:03<00:01, 33.16batch/s]

Training batches on cuda:  79%|███████▉  | 116/146 [00:03<00:00, 33.62batch/s]

Training batches on cuda:  82%|████████▏ | 120/146 [00:03<00:00, 33.73batch/s]

Training batches on cuda:  85%|████████▍ | 124/146 [00:03<00:00, 33.48batch/s]

Training batches on cuda:  88%|████████▊ | 128/146 [00:03<00:00, 33.59batch/s]

Training batches on cuda:  90%|█████████ | 132/146 [00:04<00:00, 33.65batch/s]

Training batches on cuda:  93%|█████████▎| 136/146 [00:04<00:00, 33.53batch/s]

Training batches on cuda:  96%|█████████▌| 140/146 [00:04<00:00, 33.33batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:04<00:00, 33.21batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:46<00:04,  4.70s/epoch, loss=0.00062, prev_loss=0.00062]2021-09-16:19:10:25,273 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:10:26,313 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:10:41,334 INFO     [evaluator.py:657] Evaluation took 15.02s seconds
2021-09-16:19:10:41,481 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00,  9.57s/epoch, loss=0.00062, prev_loss=0.00062]
Training epochs on cuda: 100%|██████████| 10/10 [01:02<00:00,  6.30s/epoch, loss=0.00062, prev_loss=0.00062]
2021-09-16:19:10:41,482 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpu_0u9hse'
2021-09-16:19:10:41,546 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpu_0u9hse' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:19, 1.96ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:17, 2.12ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:16, 2.25ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.35ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.49ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:14, 2.53ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.55ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.62ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.60ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:10:56,468 INFO     [evaluator.py:657] Evaluation took 14.91s seconds
[I 2021-09-16 19:10:56,473] Trial 70 finished with value: 0.0011109278065398742 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.44422584409969545, 'optimizer.lr': 0.012568315087363419, 'negative_sampler.num_negs_per_pos': 27, 'training.batch_size': 2132}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:10:56,496 WARNING  [api.py:823] No random seed is specified. Setting to 4192160436.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/180 [00:00<00:07, 24.07batch/s]

Training batches on cuda:   4%|▍         | 7/180 [00:00<00:06, 25.89batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:06, 27.56batch/s]

Training batches on cuda:   8%|▊         | 15/180 [00:00<00:05, 28.82batch/s]

Training batches on cuda:  11%|█         | 19/180 [00:00<00:05, 29.92batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 30.64batch/s]

Training batches on cuda:  15%|█▌        | 27/180 [00:00<00:04, 31.03batch/s]

Training batches on cuda:  17%|█▋        | 31/180 [00:00<00:04, 31.53batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:04, 31.78batch/s]

Training batches on cuda:  22%|██▏       | 39/180 [00:01<00:04, 31.99batch/s]

Training batches on cuda:  24%|██▍       | 43/180 [00:01<00:04, 31.73batch/s]

Training batches on cuda:  26%|██▌       | 47/180 [00:01<00:04, 31.22batch/s]

Training batches on cuda:  28%|██▊       | 51/180 [00:01<00:04, 31.46batch/s]

Training batches on cuda:  31%|███       | 55/180 [00:01<00:03, 31.40batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:01<00:03, 31.81batch/s]

Training batches on cuda:  35%|███▌      | 63/180 [00:02<00:03, 31.76batch/s]

Training batches on cuda:  37%|███▋      | 67/180 [00:02<00:03, 32.01batch/s]

Training batches on cuda:  39%|███▉      | 71/180 [00:02<00:03, 32.35batch/s]

Training batches on cuda:  42%|████▏     | 75/180 [00:02<00:03, 32.35batch/s]

Training batches on cuda:  44%|████▍     | 79/180 [00:02<00:03, 32.43batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:02<00:03, 32.33batch/s]

Training batches on cuda:  48%|████▊     | 87/180 [00:02<00:02, 32.52batch/s]

Training batches on cuda:  51%|█████     | 91/180 [00:02<00:02, 32.54batch/s]

Training batches on cuda:  53%|█████▎    | 95/180 [00:02<00:02, 32.63batch/s]

Training batches on cuda:  55%|█████▌    | 99/180 [00:03<00:02, 32.29batch/s]

Training batches on cuda:  57%|█████▋    | 103/180 [00:03<00:02, 32.28batch/s]

Training batches on cuda:  59%|█████▉    | 107/180 [00:03<00:02, 32.46batch/s]

Training batches on cuda:  62%|██████▏   | 111/180 [00:03<00:02, 32.35batch/s]

Training batches on cuda:  64%|██████▍   | 115/180 [00:03<00:02, 32.26batch/s]

Training batches on cuda:  66%|██████▌   | 119/180 [00:03<00:01, 32.27batch/s]

Training batches on cuda:  68%|██████▊   | 123/180 [00:03<00:01, 32.22batch/s]

Training batches on cuda:  71%|███████   | 127/180 [00:03<00:01, 32.30batch/s]

Training batches on cuda:  73%|███████▎  | 131/180 [00:04<00:01, 32.30batch/s]

Training batches on cuda:  75%|███████▌  | 135/180 [00:04<00:01, 32.45batch/s]

Training batches on cuda:  77%|███████▋  | 139/180 [00:04<00:01, 32.13batch/s]

Training batches on cuda:  79%|███████▉  | 143/180 [00:04<00:01, 31.54batch/s]

Training batches on cuda:  82%|████████▏ | 147/180 [00:04<00:01, 31.88batch/s]

Training batches on cuda:  84%|████████▍ | 151/180 [00:04<00:00, 32.00batch/s]

Training batches on cuda:  86%|████████▌ | 155/180 [00:04<00:00, 32.22batch/s]

Training batches on cuda:  88%|████████▊ | 159/180 [00:04<00:00, 32.52batch/s]

Training batches on cuda:  91%|█████████ | 163/180 [00:05<00:00, 32.55batch/s]

Training batches on cuda:  93%|█████████▎| 167/180 [00:05<00:00, 32.68batch/s]

Training batches on cuda:  95%|█████████▌| 171/180 [00:05<00:00, 32.74batch/s]

Training batches on cuda:  97%|█████████▋| 175/180 [00:05<00:00, 32.72batch/s]

Training batches on cuda:  99%|█████████▉| 179/180 [00:05<00:00, 32.84batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.000762, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:52,  5.87s/epoch, loss=0.000762, prev_loss=nan]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:11, 15.77batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:09, 18.55batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:07, 21.41batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 23.79batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:06, 25.88batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:05, 27.35batch/s]

Training batches on cuda:  14%|█▍        | 25/180 [00:00<00:05, 27.57batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:00<00:05, 29.03batch/s]

Training batches on cuda:  18%|█▊        | 33/180 [00:01<00:04, 30.00batch/s]

Training batches on cuda:  21%|██        | 37/180 [00:01<00:04, 30.84batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:04, 31.78batch/s]

Training batches on cuda:  25%|██▌       | 45/180 [00:01<00:04, 32.17batch/s]

Training batches on cuda:  27%|██▋       | 49/180 [00:01<00:03, 32.77batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:03, 32.26batch/s]

Training batches on cuda:  32%|███▏      | 57/180 [00:01<00:03, 32.05batch/s]

Training batches on cuda:  34%|███▍      | 61/180 [00:01<00:03, 32.33batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:03, 32.06batch/s]

Training batches on cuda:  38%|███▊      | 69/180 [00:02<00:03, 32.12batch/s]

Training batches on cuda:  41%|████      | 73/180 [00:02<00:03, 32.31batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 32.05batch/s]

Training batches on cuda:  45%|████▌     | 81/180 [00:02<00:03, 31.34batch/s]

Training batches on cuda:  47%|████▋     | 85/180 [00:02<00:03, 30.77batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:02<00:02, 31.20batch/s]

Training batches on cuda:  52%|█████▏    | 93/180 [00:02<00:02, 31.42batch/s]

Training batches on cuda:  54%|█████▍    | 97/180 [00:03<00:02, 31.32batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 31.33batch/s]

Training batches on cuda:  58%|█████▊    | 105/180 [00:03<00:02, 30.76batch/s]

Training batches on cuda:  61%|██████    | 109/180 [00:03<00:02, 30.84batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:03<00:02, 30.97batch/s]

Training batches on cuda:  65%|██████▌   | 117/180 [00:03<00:02, 31.16batch/s]

Training batches on cuda:  67%|██████▋   | 121/180 [00:03<00:01, 31.50batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:03<00:01, 31.54batch/s]

Training batches on cuda:  72%|███████▏  | 129/180 [00:04<00:01, 31.84batch/s]

Training batches on cuda:  74%|███████▍  | 133/180 [00:04<00:01, 32.12batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:04<00:01, 32.48batch/s]

Training batches on cuda:  78%|███████▊  | 141/180 [00:04<00:01, 32.63batch/s]

Training batches on cuda:  81%|████████  | 145/180 [00:04<00:01, 32.79batch/s]

Training batches on cuda:  83%|████████▎ | 149/180 [00:04<00:00, 31.92batch/s]

Training batches on cuda:  85%|████████▌ | 153/180 [00:04<00:00, 32.29batch/s]

Training batches on cuda:  87%|████████▋ | 157/180 [00:04<00:00, 32.45batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:05<00:00, 32.63batch/s]

Training batches on cuda:  92%|█████████▏| 165/180 [00:05<00:00, 32.29batch/s]

Training batches on cuda:  94%|█████████▍| 169/180 [00:05<00:00, 32.40batch/s]

Training batches on cuda:  96%|█████████▌| 173/180 [00:05<00:00, 32.60batch/s]

Training batches on cuda:  98%|█████████▊| 177/180 [00:05<00:00, 32.17batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:11<00:52,  5.87s/epoch, loss=0.000759, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:46,  5.87s/epoch, loss=0.000759, prev_loss=nan]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/180 [00:00<00:17,  9.99batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:13, 12.57batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:11, 15.22batch/s]

Training batches on cuda:   7%|▋         | 12/180 [00:00<00:09, 18.14batch/s]

Training batches on cuda:   8%|▊         | 15/180 [00:00<00:08, 20.41batch/s]

Training batches on cuda:  11%|█         | 19/180 [00:00<00:07, 22.90batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:06, 24.42batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 26.72batch/s]

Training batches on cuda:  17%|█▋        | 30/180 [00:01<00:05, 28.84batch/s]

Training batches on cuda:  19%|█▉        | 34/180 [00:01<00:04, 30.44batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:04, 31.79batch/s]

Training batches on cuda:  23%|██▎       | 42/180 [00:01<00:04, 32.81batch/s]

Training batches on cuda:  26%|██▌       | 46/180 [00:01<00:03, 33.57batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:03, 33.95batch/s]

Training batches on cuda:  30%|███       | 54/180 [00:01<00:03, 34.32batch/s]

Training batches on cuda:  32%|███▏      | 58/180 [00:01<00:03, 34.48batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:01<00:03, 34.79batch/s]

Training batches on cuda:  37%|███▋      | 66/180 [00:02<00:03, 34.06batch/s]

Training batches on cuda:  39%|███▉      | 70/180 [00:02<00:03, 33.54batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 32.42batch/s]

Training batches on cuda:  43%|████▎     | 78/180 [00:02<00:03, 31.99batch/s]

Training batches on cuda:  46%|████▌     | 82/180 [00:02<00:03, 31.89batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:02<00:02, 31.42batch/s]

Training batches on cuda:  50%|█████     | 90/180 [00:02<00:02, 31.55batch/s]

Training batches on cuda:  52%|█████▏    | 94/180 [00:02<00:02, 31.66batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 32.10batch/s]

Training batches on cuda:  57%|█████▋    | 102/180 [00:03<00:02, 32.30batch/s]

Training batches on cuda:  59%|█████▉    | 106/180 [00:03<00:02, 32.50batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 32.14batch/s]

Training batches on cuda:  63%|██████▎   | 114/180 [00:03<00:02, 32.20batch/s]

Training batches on cuda:  66%|██████▌   | 118/180 [00:03<00:01, 31.80batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:03<00:01, 31.85batch/s]

Training batches on cuda:  70%|███████   | 126/180 [00:03<00:01, 32.02batch/s]

Training batches on cuda:  72%|███████▏  | 130/180 [00:04<00:01, 32.18batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 32.16batch/s]

Training batches on cuda:  77%|███████▋  | 138/180 [00:04<00:01, 32.37batch/s]

Training batches on cuda:  79%|███████▉  | 142/180 [00:04<00:01, 32.48batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:04<00:01, 32.67batch/s]

Training batches on cuda:  83%|████████▎ | 150/180 [00:04<00:00, 32.59batch/s]

Training batches on cuda:  86%|████████▌ | 154/180 [00:04<00:00, 32.60batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:04<00:00, 32.62batch/s]

Training batches on cuda:  90%|█████████ | 162/180 [00:05<00:00, 32.66batch/s]

Training batches on cuda:  92%|█████████▏| 166/180 [00:05<00:00, 32.52batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:05<00:00, 32.38batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:05<00:00, 32.23batch/s]

Training batches on cuda:  99%|█████████▉| 178/180 [00:05<00:00, 32.11batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:17<00:46,  5.87s/epoch, loss=0.000741, prev_loss=0.000759]
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:41,  5.87s/epoch, loss=0.000741, prev_loss=0.000759]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:09, 18.95batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:08, 21.62batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:07, 24.10batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 26.04batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:05, 27.42batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:05, 28.35batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 28.76batch/s]

Training batches on cuda:  17%|█▋        | 30/180 [00:00<00:05, 29.32batch/s]

Training batches on cuda:  19%|█▉        | 34/180 [00:01<00:04, 29.79batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:04, 30.94batch/s]

Training batches on cuda:  23%|██▎       | 42/180 [00:01<00:04, 31.61batch/s]

Training batches on cuda:  26%|██▌       | 46/180 [00:01<00:04, 31.78batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 32.19batch/s]

Training batches on cuda:  30%|███       | 54/180 [00:01<00:03, 32.43batch/s]

Training batches on cuda:  32%|███▏      | 58/180 [00:01<00:03, 32.41batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:01<00:03, 32.33batch/s]

Training batches on cuda:  37%|███▋      | 66/180 [00:02<00:03, 32.41batch/s]

Training batches on cuda:  39%|███▉      | 70/180 [00:02<00:03, 32.37batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 32.21batch/s]

Training batches on cuda:  43%|████▎     | 78/180 [00:02<00:03, 32.10batch/s]

Training batches on cuda:  46%|████▌     | 82/180 [00:02<00:03, 32.05batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:02<00:02, 32.22batch/s]

Training batches on cuda:  50%|█████     | 90/180 [00:02<00:02, 31.81batch/s]

Training batches on cuda:  52%|█████▏    | 94/180 [00:02<00:02, 32.19batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 32.20batch/s]

Training batches on cuda:  57%|█████▋    | 102/180 [00:03<00:02, 32.13batch/s]

Training batches on cuda:  59%|█████▉    | 106/180 [00:03<00:02, 32.25batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 31.85batch/s]

Training batches on cuda:  63%|██████▎   | 114/180 [00:03<00:02, 31.80batch/s]

Training batches on cuda:  66%|██████▌   | 118/180 [00:03<00:01, 31.82batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:03<00:01, 31.69batch/s]

Training batches on cuda:  70%|███████   | 126/180 [00:03<00:01, 31.45batch/s]

Training batches on cuda:  72%|███████▏  | 130/180 [00:04<00:01, 31.77batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 31.47batch/s]

Training batches on cuda:  77%|███████▋  | 138/180 [00:04<00:01, 31.65batch/s]

Training batches on cuda:  79%|███████▉  | 142/180 [00:04<00:01, 31.58batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:04<00:01, 31.67batch/s]

Training batches on cuda:  83%|████████▎ | 150/180 [00:04<00:00, 31.86batch/s]

Training batches on cuda:  86%|████████▌ | 154/180 [00:04<00:00, 31.49batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:04<00:00, 31.87batch/s]

Training batches on cuda:  90%|█████████ | 162/180 [00:05<00:00, 32.17batch/s]

Training batches on cuda:  92%|█████████▏| 166/180 [00:05<00:00, 32.18batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:05<00:00, 32.41batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:05<00:00, 32.23batch/s]

Training batches on cuda:  99%|█████████▉| 178/180 [00:05<00:00, 32.40batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:23<00:41,  5.87s/epoch, loss=0.00067, prev_loss=0.000741] 
Training epochs on cuda:  40%|████      | 4/10 [00:23<00:35,  5.90s/epoch, loss=0.00067, prev_loss=0.000741]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:09, 18.90batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:08, 21.49batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:07, 23.86batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 25.84batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:05, 27.59batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:05, 28.84batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 29.71batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:00<00:05, 28.74batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 28.94batch/s]

Training batches on cuda:  20%|██        | 36/180 [00:01<00:04, 29.35batch/s]

Training batches on cuda:  22%|██▏       | 40/180 [00:01<00:04, 30.03batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:04, 30.27batch/s]

Training batches on cuda:  27%|██▋       | 48/180 [00:01<00:04, 30.93batch/s]

Training batches on cuda:  29%|██▉       | 52/180 [00:01<00:04, 30.27batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:01<00:04, 30.36batch/s]

Training batches on cuda:  33%|███▎      | 60/180 [00:01<00:03, 30.86batch/s]

Training batches on cuda:  36%|███▌      | 64/180 [00:02<00:03, 31.08batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:03, 31.43batch/s]

Training batches on cuda:  40%|████      | 72/180 [00:02<00:03, 31.71batch/s]

Training batches on cuda:  42%|████▏     | 76/180 [00:02<00:03, 30.97batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:03, 31.35batch/s]

Training batches on cuda:  47%|████▋     | 84/180 [00:02<00:03, 30.99batch/s]

Training batches on cuda:  49%|████▉     | 88/180 [00:02<00:02, 30.93batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:02<00:02, 31.28batch/s]

Training batches on cuda:  53%|█████▎    | 96/180 [00:03<00:02, 31.48batch/s]

Training batches on cuda:  56%|█████▌    | 100/180 [00:03<00:02, 28.85batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:03<00:02, 29.49batch/s]

Training batches on cuda:  60%|██████    | 108/180 [00:03<00:02, 30.41batch/s]

Training batches on cuda:  62%|██████▏   | 112/180 [00:03<00:02, 30.90batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:03<00:02, 31.09batch/s]

Training batches on cuda:  67%|██████▋   | 120/180 [00:03<00:01, 31.36batch/s]

Training batches on cuda:  69%|██████▉   | 124/180 [00:04<00:01, 31.54batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:04<00:01, 31.40batch/s]

Training batches on cuda:  73%|███████▎  | 132/180 [00:04<00:01, 31.22batch/s]

Training batches on cuda:  76%|███████▌  | 136/180 [00:04<00:01, 31.24batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:04<00:01, 31.07batch/s]

Training batches on cuda:  80%|████████  | 144/180 [00:04<00:01, 31.30batch/s]

Training batches on cuda:  82%|████████▏ | 148/180 [00:04<00:01, 31.13batch/s]

Training batches on cuda:  84%|████████▍ | 152/180 [00:04<00:00, 31.21batch/s]

Training batches on cuda:  87%|████████▋ | 156/180 [00:05<00:00, 31.08batch/s]

Training batches on cuda:  89%|████████▉ | 160/180 [00:05<00:00, 31.01batch/s]

Training batches on cuda:  91%|█████████ | 164/180 [00:05<00:00, 31.18batch/s]

Training batches on cuda:  93%|█████████▎| 168/180 [00:05<00:00, 30.99batch/s]

Training batches on cuda:  96%|█████████▌| 172/180 [00:05<00:00, 31.27batch/s]

Training batches on cuda:  98%|█████████▊| 176/180 [00:05<00:00, 31.56batch/s]

Training batches on cuda: 100%|██████████| 180/180 [00:05<00:00, 32.58batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:29<00:35,  5.90s/epoch, loss=0.00052, prev_loss=0.00067] 
Training epochs on cuda:  50%|█████     | 5/10 [00:29<00:29,  5.95s/epoch, loss=0.00052, prev_loss=0.00067]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:10, 16.70batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:08, 19.50batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:07, 22.01batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 24.35batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:06, 26.36batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:05, 27.71batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 28.88batch/s]

Training batches on cuda:  17%|█▋        | 30/180 [00:00<00:05, 29.66batch/s]

Training batches on cuda:  19%|█▉        | 34/180 [00:01<00:04, 29.99batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:04, 30.56batch/s]

Training batches on cuda:  23%|██▎       | 42/180 [00:01<00:04, 30.50batch/s]

Training batches on cuda:  26%|██▌       | 46/180 [00:01<00:04, 31.10batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 31.07batch/s]

Training batches on cuda:  30%|███       | 54/180 [00:01<00:04, 30.66batch/s]

Training batches on cuda:  32%|███▏      | 58/180 [00:01<00:03, 31.02batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:02<00:03, 31.01batch/s]

Training batches on cuda:  37%|███▋      | 66/180 [00:02<00:03, 31.12batch/s]

Training batches on cuda:  39%|███▉      | 70/180 [00:02<00:03, 31.42batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 31.77batch/s]

Training batches on cuda:  43%|████▎     | 78/180 [00:02<00:03, 30.95batch/s]

Training batches on cuda:  46%|████▌     | 82/180 [00:02<00:03, 31.12batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:02<00:02, 31.37batch/s]

Training batches on cuda:  50%|█████     | 90/180 [00:02<00:02, 31.85batch/s]

Training batches on cuda:  52%|█████▏    | 94/180 [00:03<00:02, 32.19batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 31.48batch/s]

Training batches on cuda:  57%|█████▋    | 102/180 [00:03<00:02, 31.60batch/s]

Training batches on cuda:  59%|█████▉    | 106/180 [00:03<00:02, 31.60batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 31.75batch/s]

Training batches on cuda:  63%|██████▎   | 114/180 [00:03<00:02, 32.31batch/s]

Training batches on cuda:  66%|██████▌   | 118/180 [00:03<00:01, 32.18batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:03<00:01, 31.20batch/s]

Training batches on cuda:  70%|███████   | 126/180 [00:04<00:01, 31.61batch/s]

Training batches on cuda:  72%|███████▏  | 130/180 [00:04<00:01, 31.26batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 31.68batch/s]

Training batches on cuda:  77%|███████▋  | 138/180 [00:04<00:01, 32.00batch/s]

Training batches on cuda:  79%|███████▉  | 142/180 [00:04<00:01, 31.86batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:04<00:01, 31.91batch/s]

Training batches on cuda:  83%|████████▎ | 150/180 [00:04<00:00, 31.87batch/s]

Training batches on cuda:  86%|████████▌ | 154/180 [00:04<00:00, 31.70batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:05<00:00, 31.24batch/s]

Training batches on cuda:  90%|█████████ | 162/180 [00:05<00:00, 31.38batch/s]

Training batches on cuda:  92%|█████████▏| 166/180 [00:05<00:00, 30.96batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:05<00:00, 31.45batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:05<00:00, 31.57batch/s]

Training batches on cuda:  99%|█████████▉| 178/180 [00:05<00:00, 30.79batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:35<00:29,  5.95s/epoch, loss=0.000413, prev_loss=0.00052]
Training epochs on cuda:  60%|██████    | 6/10 [00:35<00:23,  5.98s/epoch, loss=0.000413, prev_loss=0.00052]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:10, 16.31batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:09, 18.98batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:07, 21.54batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 23.90batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:06, 25.60batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:05, 27.37batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 28.73batch/s]

Training batches on cuda:  17%|█▋        | 30/180 [00:01<00:05, 29.51batch/s]

Training batches on cuda:  18%|█▊        | 33/180 [00:01<00:05, 26.59batch/s]

Training batches on cuda:  21%|██        | 37/180 [00:01<00:05, 28.17batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:04, 29.60batch/s]

Training batches on cuda:  25%|██▌       | 45/180 [00:01<00:04, 30.27batch/s]

Training batches on cuda:  27%|██▋       | 49/180 [00:01<00:04, 30.83batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:04, 31.41batch/s]

Training batches on cuda:  32%|███▏      | 57/180 [00:01<00:03, 31.75batch/s]

Training batches on cuda:  34%|███▍      | 61/180 [00:02<00:03, 31.82batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:03, 31.99batch/s]

Training batches on cuda:  38%|███▊      | 69/180 [00:02<00:03, 32.03batch/s]

Training batches on cuda:  41%|████      | 73/180 [00:02<00:03, 32.10batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 32.48batch/s]

Training batches on cuda:  45%|████▌     | 81/180 [00:02<00:03, 32.31batch/s]

Training batches on cuda:  47%|████▋     | 85/180 [00:02<00:02, 32.29batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:02<00:02, 32.18batch/s]

Training batches on cuda:  52%|█████▏    | 93/180 [00:03<00:02, 31.49batch/s]

Training batches on cuda:  54%|█████▍    | 97/180 [00:03<00:02, 31.41batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 30.95batch/s]

Training batches on cuda:  58%|█████▊    | 105/180 [00:03<00:02, 31.06batch/s]

Training batches on cuda:  61%|██████    | 109/180 [00:03<00:02, 31.09batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:03<00:02, 31.45batch/s]

Training batches on cuda:  65%|██████▌   | 117/180 [00:03<00:01, 31.82batch/s]

Training batches on cuda:  67%|██████▋   | 121/180 [00:03<00:01, 32.09batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:04<00:01, 32.47batch/s]

Training batches on cuda:  72%|███████▏  | 129/180 [00:04<00:01, 31.63batch/s]

Training batches on cuda:  74%|███████▍  | 133/180 [00:04<00:01, 31.15batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:04<00:01, 31.03batch/s]

Training batches on cuda:  78%|███████▊  | 141/180 [00:04<00:01, 30.82batch/s]

Training batches on cuda:  81%|████████  | 145/180 [00:04<00:01, 31.91batch/s]

Training batches on cuda:  83%|████████▎ | 149/180 [00:04<00:00, 32.04batch/s]

Training batches on cuda:  85%|████████▌ | 153/180 [00:04<00:00, 31.77batch/s]

Training batches on cuda:  87%|████████▋ | 157/180 [00:05<00:00, 31.80batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:05<00:00, 31.67batch/s]

Training batches on cuda:  92%|█████████▏| 165/180 [00:05<00:00, 31.86batch/s]

Training batches on cuda:  94%|█████████▍| 169/180 [00:05<00:00, 31.77batch/s]

Training batches on cuda:  96%|█████████▌| 173/180 [00:05<00:00, 31.91batch/s]

Training batches on cuda:  98%|█████████▊| 177/180 [00:05<00:00, 31.47batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:41<00:23,  5.98s/epoch, loss=0.00038, prev_loss=0.000413]
Training epochs on cuda:  70%|███████   | 7/10 [00:41<00:17,  5.97s/epoch, loss=0.00038, prev_loss=0.000413]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:10, 17.55batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:09, 19.23batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:08, 20.83batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:07, 22.12batch/s]

Training batches on cuda:   8%|▊         | 15/180 [00:00<00:06, 24.18batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:06, 25.44batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:05, 26.76batch/s]

Training batches on cuda:  14%|█▍        | 25/180 [00:00<00:05, 26.69batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 27.76batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 28.19batch/s]

Training batches on cuda:  20%|██        | 36/180 [00:01<00:04, 28.81batch/s]

Training batches on cuda:  22%|██▏       | 40/180 [00:01<00:04, 29.33batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:04, 29.69batch/s]

Training batches on cuda:  27%|██▋       | 48/180 [00:01<00:04, 29.93batch/s]

Training batches on cuda:  29%|██▉       | 52/180 [00:01<00:04, 30.52batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:01<00:03, 31.60batch/s]

Training batches on cuda:  33%|███▎      | 60/180 [00:02<00:03, 32.50batch/s]

Training batches on cuda:  36%|███▌      | 64/180 [00:02<00:03, 32.98batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:03, 33.33batch/s]

Training batches on cuda:  40%|████      | 72/180 [00:02<00:03, 33.69batch/s]

Training batches on cuda:  42%|████▏     | 76/180 [00:02<00:03, 34.02batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:02, 34.27batch/s]

Training batches on cuda:  47%|████▋     | 84/180 [00:02<00:02, 34.52batch/s]

Training batches on cuda:  49%|████▉     | 88/180 [00:02<00:02, 34.51batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:02<00:02, 34.50batch/s]

Training batches on cuda:  53%|█████▎    | 96/180 [00:03<00:02, 34.48batch/s]

Training batches on cuda:  56%|█████▌    | 100/180 [00:03<00:02, 34.48batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:03<00:02, 34.45batch/s]

Training batches on cuda:  60%|██████    | 108/180 [00:03<00:02, 34.56batch/s]

Training batches on cuda:  62%|██████▏   | 112/180 [00:03<00:01, 34.49batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:03<00:01, 34.40batch/s]

Training batches on cuda:  67%|██████▋   | 120/180 [00:03<00:01, 34.05batch/s]

Training batches on cuda:  69%|██████▉   | 124/180 [00:03<00:01, 33.45batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:04<00:01, 32.32batch/s]

Training batches on cuda:  73%|███████▎  | 132/180 [00:04<00:01, 31.93batch/s]

Training batches on cuda:  76%|███████▌  | 136/180 [00:04<00:01, 31.94batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:04<00:01, 31.52batch/s]

Training batches on cuda:  80%|████████  | 144/180 [00:04<00:01, 31.27batch/s]

Training batches on cuda:  82%|████████▏ | 148/180 [00:04<00:01, 30.75batch/s]

Training batches on cuda:  84%|████████▍ | 152/180 [00:04<00:00, 30.88batch/s]

Training batches on cuda:  87%|████████▋ | 156/180 [00:04<00:00, 31.10batch/s]

Training batches on cuda:  89%|████████▉ | 160/180 [00:05<00:00, 31.49batch/s]

Training batches on cuda:  91%|█████████ | 164/180 [00:05<00:00, 31.69batch/s]

Training batches on cuda:  93%|█████████▎| 168/180 [00:05<00:00, 28.78batch/s]

Training batches on cuda:  96%|█████████▌| 172/180 [00:05<00:00, 29.94batch/s]

Training batches on cuda:  98%|█████████▊| 176/180 [00:05<00:00, 30.79batch/s]

Training batches on cuda: 100%|██████████| 180/180 [00:05<00:00, 32.08batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:47<00:17,  5.97s/epoch, loss=0.000364, prev_loss=0.00038]
Training epochs on cuda:  80%|████████  | 8/10 [00:47<00:11,  5.96s/epoch, loss=0.000364, prev_loss=0.00038]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:09, 18.93batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:08, 21.47batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:07, 23.93batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 26.04batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:05, 27.79batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:05, 28.98batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 29.80batch/s]

Training batches on cuda:  17%|█▋        | 30/180 [00:00<00:04, 30.64batch/s]

Training batches on cuda:  19%|█▉        | 34/180 [00:01<00:04, 31.28batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:04, 31.73batch/s]

Training batches on cuda:  23%|██▎       | 42/180 [00:01<00:04, 32.03batch/s]

Training batches on cuda:  26%|██▌       | 46/180 [00:01<00:04, 32.19batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 32.47batch/s]

Training batches on cuda:  30%|███       | 54/180 [00:01<00:03, 32.13batch/s]

Training batches on cuda:  32%|███▏      | 58/180 [00:01<00:03, 32.22batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:01<00:03, 31.92batch/s]

Training batches on cuda:  37%|███▋      | 66/180 [00:02<00:03, 31.34batch/s]

Training batches on cuda:  39%|███▉      | 70/180 [00:02<00:03, 31.66batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 31.95batch/s]

Training batches on cuda:  43%|████▎     | 78/180 [00:02<00:03, 32.22batch/s]

Training batches on cuda:  46%|████▌     | 82/180 [00:02<00:03, 32.37batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:02<00:02, 32.36batch/s]

Training batches on cuda:  50%|█████     | 90/180 [00:02<00:02, 31.72batch/s]

Training batches on cuda:  52%|█████▏    | 94/180 [00:02<00:02, 31.62batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 31.93batch/s]

Training batches on cuda:  57%|█████▋    | 102/180 [00:03<00:02, 32.11batch/s]

Training batches on cuda:  59%|█████▉    | 106/180 [00:03<00:02, 32.36batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 32.32batch/s]

Training batches on cuda:  63%|██████▎   | 114/180 [00:03<00:02, 32.44batch/s]

Training batches on cuda:  66%|██████▌   | 118/180 [00:03<00:01, 32.00batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:03<00:01, 32.03batch/s]

Training batches on cuda:  70%|███████   | 126/180 [00:03<00:01, 32.05batch/s]

Training batches on cuda:  72%|███████▏  | 130/180 [00:04<00:01, 32.38batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 31.96batch/s]

Training batches on cuda:  77%|███████▋  | 138/180 [00:04<00:01, 32.52batch/s]

Training batches on cuda:  79%|███████▉  | 142/180 [00:04<00:01, 32.37batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:04<00:01, 33.10batch/s]

Training batches on cuda:  83%|████████▎ | 150/180 [00:04<00:00, 33.49batch/s]

Training batches on cuda:  86%|████████▌ | 154/180 [00:04<00:00, 33.47batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:04<00:00, 32.59batch/s]

Training batches on cuda:  90%|█████████ | 162/180 [00:05<00:00, 32.00batch/s]

Training batches on cuda:  92%|█████████▏| 166/180 [00:05<00:00, 31.80batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:05<00:00, 32.24batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:05<00:00, 31.62batch/s]

Training batches on cuda:  99%|█████████▉| 178/180 [00:05<00:00, 32.03batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:53<00:11,  5.96s/epoch, loss=0.00037, prev_loss=0.000364]
Training epochs on cuda:  90%|█████████ | 9/10 [00:53<00:05,  5.94s/epoch, loss=0.00037, prev_loss=0.000364]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:09, 19.05batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:08, 21.58batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:07, 24.08batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 26.15batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 26.98batch/s]

Training batches on cuda:  12%|█▏        | 21/180 [00:00<00:05, 28.29batch/s]

Training batches on cuda:  14%|█▍        | 25/180 [00:00<00:05, 29.03batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:00<00:05, 29.86batch/s]

Training batches on cuda:  18%|█▊        | 33/180 [00:01<00:04, 30.65batch/s]

Training batches on cuda:  21%|██        | 37/180 [00:01<00:04, 31.48batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:04, 32.07batch/s]

Training batches on cuda:  25%|██▌       | 45/180 [00:01<00:04, 32.45batch/s]

Training batches on cuda:  27%|██▋       | 49/180 [00:01<00:04, 32.63batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:03, 32.80batch/s]

Training batches on cuda:  32%|███▏      | 57/180 [00:01<00:03, 32.80batch/s]

Training batches on cuda:  34%|███▍      | 61/180 [00:01<00:03, 32.50batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:03, 32.73batch/s]

Training batches on cuda:  38%|███▊      | 69/180 [00:02<00:03, 32.73batch/s]

Training batches on cuda:  41%|████      | 73/180 [00:02<00:03, 32.82batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 32.83batch/s]

Training batches on cuda:  45%|████▌     | 81/180 [00:02<00:03, 32.88batch/s]

Training batches on cuda:  47%|████▋     | 85/180 [00:02<00:02, 32.86batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:02<00:02, 32.92batch/s]

Training batches on cuda:  52%|█████▏    | 93/180 [00:02<00:02, 32.97batch/s]

Training batches on cuda:  54%|█████▍    | 97/180 [00:03<00:02, 32.81batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 32.92batch/s]

Training batches on cuda:  58%|█████▊    | 105/180 [00:03<00:02, 32.72batch/s]

Training batches on cuda:  61%|██████    | 109/180 [00:03<00:02, 32.49batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:03<00:02, 30.06batch/s]

Training batches on cuda:  65%|██████▌   | 117/180 [00:03<00:02, 30.46batch/s]

Training batches on cuda:  67%|██████▋   | 121/180 [00:03<00:01, 30.91batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:03<00:01, 31.58batch/s]

Training batches on cuda:  72%|███████▏  | 129/180 [00:04<00:01, 32.61batch/s]

Training batches on cuda:  74%|███████▍  | 133/180 [00:04<00:01, 33.22batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:04<00:01, 33.61batch/s]

Training batches on cuda:  78%|███████▊  | 141/180 [00:04<00:01, 33.97batch/s]

Training batches on cuda:  81%|████████  | 145/180 [00:04<00:01, 33.24batch/s]

Training batches on cuda:  83%|████████▎ | 149/180 [00:04<00:00, 33.19batch/s]

Training batches on cuda:  85%|████████▌ | 153/180 [00:04<00:00, 33.25batch/s]

Training batches on cuda:  87%|████████▋ | 157/180 [00:04<00:00, 33.21batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:04<00:00, 33.15batch/s]

Training batches on cuda:  92%|█████████▏| 165/180 [00:05<00:00, 33.22batch/s]

Training batches on cuda:  94%|█████████▍| 169/180 [00:05<00:00, 33.15batch/s]

Training batches on cuda:  96%|█████████▌| 173/180 [00:05<00:00, 33.15batch/s]

Training batches on cuda:  98%|█████████▊| 177/180 [00:05<00:00, 33.04batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:59<00:05,  5.94s/epoch, loss=0.000357, prev_loss=0.00037]2021-09-16:19:11:56,329 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:11:57,511 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:12:12,531 INFO     [evaluator.py:657] Evaluation took 15.02s seconds
2021-09-16:19:12:12,660 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00, 10.79s/epoch, loss=0.000357, prev_loss=0.00037]
Training epochs on cuda: 100%|██████████| 10/10 [01:15<00:00,  7.56s/epoch, loss=0.000357, prev_loss=0.00037]
2021-09-16:19:12:12,661 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpayd4mcx1'
2021-09-16:19:12:12,715 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpayd4mcx1' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.25ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.36ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.50ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.53ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:03, 2.56ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.58ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.59ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.61ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:19:12:27,586 INFO     [evaluator.py:657] Evaluation took 14.86s seconds
[I 2021-09-16 19:12:27,592] Trial 71 finished with value: 0.9496053291757459 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.015844182438197022, 'optimizer.lr': 0.03909789255458629, 'negative_sampler.num_negs_per_pos': 47, 'training.batch_size': 1732}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:12:27,617 WARNING  [api.py:823] No random seed is specified. Setting to 1500762067.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:09, 18.90batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:08, 21.11batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:07, 22.51batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:06, 24.15batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 25.63batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 26.60batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:05, 27.07batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 27.70batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 27.79batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 28.22batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 28.66batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:05, 28.65batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:04, 28.72batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:04, 28.70batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:04, 28.82batch/s]

Training batches on cuda:  26%|██▌       | 47/180 [00:01<00:04, 28.95batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 29.00batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:04, 29.06batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:01<00:04, 28.99batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:02<00:04, 28.80batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:02<00:04, 28.89batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:03, 29.03batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:03, 29.11batch/s]

Training batches on cuda:  39%|███▉      | 71/180 [00:02<00:03, 28.93batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 28.88batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 28.90batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:03, 28.90batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:02<00:03, 28.71batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:03<00:03, 28.74batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:03<00:03, 28.83batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:03<00:03, 28.59batch/s]

Training batches on cuda:  53%|█████▎    | 95/180 [00:03<00:02, 28.69batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 28.50batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 28.33batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:03<00:02, 28.26batch/s]

Training batches on cuda:  59%|█████▉    | 107/180 [00:03<00:02, 28.57batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 27.91batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:03<00:02, 27.99batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:04<00:02, 27.90batch/s]

Training batches on cuda:  66%|██████▌   | 119/180 [00:04<00:02, 28.20batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:04<00:02, 28.52batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:04<00:01, 28.72batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:04<00:01, 28.86batch/s]

Training batches on cuda:  73%|███████▎  | 131/180 [00:04<00:01, 29.00batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 29.07batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:04<00:01, 28.95batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:04<00:01, 28.46batch/s]

Training batches on cuda:  79%|███████▉  | 143/180 [00:05<00:01, 28.34batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:05<00:01, 28.50batch/s]

Training batches on cuda:  83%|████████▎ | 150/180 [00:05<00:01, 29.17batch/s]

Training batches on cuda:  86%|████████▌ | 154/180 [00:05<00:00, 29.61batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:05<00:00, 29.89batch/s]

Training batches on cuda:  90%|█████████ | 162/180 [00:05<00:00, 30.12batch/s]

Training batches on cuda:  92%|█████████▏| 166/180 [00:05<00:00, 30.27batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:05<00:00, 30.39batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:06<00:00, 30.48batch/s]

Training batches on cuda:  99%|█████████▉| 178/180 [00:06<00:00, 29.43batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:06<?, ?epoch/s, loss=0.000763, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:57,  6.43s/epoch, loss=0.000763, prev_loss=nan]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:10, 17.08batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:08, 19.46batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:08, 21.40batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:07, 23.19batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 24.50batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 25.29batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:06, 26.20batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 26.86batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 27.39batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 27.81batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 28.25batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:05, 28.56batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:04, 28.71batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:04, 28.90batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:04, 28.92batch/s]

Training batches on cuda:  26%|██▌       | 47/180 [00:01<00:04, 28.44batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 27.14batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:04, 26.47batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:02<00:04, 25.26batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:02<00:04, 24.86batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:02<00:04, 25.74batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:04, 26.36batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:04, 27.18batch/s]

Training batches on cuda:  39%|███▉      | 71/180 [00:02<00:03, 27.83batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 28.33batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 28.41batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:03, 27.94batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:03<00:03, 28.13batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:03<00:03, 28.32batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:03<00:03, 28.33batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:03<00:03, 27.22batch/s]

Training batches on cuda:  53%|█████▎    | 95/180 [00:03<00:03, 26.78batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:03, 26.81batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 26.90batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:03<00:02, 26.90batch/s]

Training batches on cuda:  59%|█████▉    | 107/180 [00:03<00:02, 26.81batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:04<00:02, 26.47batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:04<00:02, 26.78batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:04<00:02, 26.93batch/s]

Training batches on cuda:  66%|██████▌   | 119/180 [00:04<00:02, 27.03batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:04<00:02, 27.00batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:04<00:02, 26.93batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:04<00:02, 23.42batch/s]

Training batches on cuda:  73%|███████▎  | 131/180 [00:04<00:02, 24.31batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 25.20batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:05<00:01, 25.82batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:05<00:01, 26.47batch/s]

Training batches on cuda:  79%|███████▉  | 143/180 [00:05<00:01, 26.82batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:05<00:01, 26.81batch/s]

Training batches on cuda:  83%|████████▎ | 149/180 [00:05<00:01, 26.78batch/s]

Training batches on cuda:  84%|████████▍ | 152/180 [00:05<00:01, 26.61batch/s]

Training batches on cuda:  86%|████████▌ | 155/180 [00:05<00:00, 26.68batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:05<00:00, 27.35batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:05<00:00, 27.23batch/s]

Training batches on cuda:  91%|█████████ | 164/180 [00:06<00:00, 27.54batch/s]

Training batches on cuda:  93%|█████████▎| 167/180 [00:06<00:00, 27.76batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:06<00:00, 27.87batch/s]

Training batches on cuda:  96%|█████████▌| 173/180 [00:06<00:00, 27.55batch/s]

Training batches on cuda:  98%|█████████▊| 176/180 [00:06<00:00, 27.98batch/s]

Training batches on cuda:  99%|█████████▉| 179/180 [00:06<00:00, 27.86batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:13<00:57,  6.43s/epoch, loss=0.000762, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:52,  6.55s/epoch, loss=0.000762, prev_loss=nan]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/180 [00:00<00:25,  7.09batch/s]

Training batches on cuda:   2%|▏         | 4/180 [00:00<00:19,  9.17batch/s]

Training batches on cuda:   4%|▍         | 7/180 [00:00<00:15, 11.49batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:12, 13.84batch/s]

Training batches on cuda:   7%|▋         | 13/180 [00:00<00:10, 16.26batch/s]

Training batches on cuda:   9%|▉         | 16/180 [00:00<00:08, 18.60batch/s]

Training batches on cuda:  11%|█         | 19/180 [00:00<00:07, 20.71batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:06, 22.71batch/s]

Training batches on cuda:  14%|█▍        | 25/180 [00:00<00:06, 24.10batch/s]

Training batches on cuda:  16%|█▌        | 28/180 [00:01<00:06, 24.89batch/s]

Training batches on cuda:  17%|█▋        | 31/180 [00:01<00:05, 25.94batch/s]

Training batches on cuda:  19%|█▉        | 34/180 [00:01<00:05, 26.77batch/s]

Training batches on cuda:  21%|██        | 37/180 [00:01<00:05, 26.96batch/s]

Training batches on cuda:  22%|██▏       | 40/180 [00:01<00:05, 27.43batch/s]

Training batches on cuda:  24%|██▍       | 43/180 [00:01<00:04, 27.81batch/s]

Training batches on cuda:  26%|██▌       | 46/180 [00:01<00:04, 27.80batch/s]

Training batches on cuda:  27%|██▋       | 49/180 [00:01<00:04, 28.14batch/s]

Training batches on cuda:  29%|██▉       | 52/180 [00:01<00:04, 27.92batch/s]

Training batches on cuda:  31%|███       | 55/180 [00:02<00:04, 28.27batch/s]

Training batches on cuda:  32%|███▏      | 58/180 [00:02<00:04, 28.20batch/s]

Training batches on cuda:  34%|███▍      | 61/180 [00:02<00:04, 28.26batch/s]

Training batches on cuda:  36%|███▌      | 64/180 [00:02<00:04, 28.32batch/s]

Training batches on cuda:  37%|███▋      | 67/180 [00:02<00:04, 28.11batch/s]

Training batches on cuda:  39%|███▉      | 70/180 [00:02<00:03, 28.32batch/s]

Training batches on cuda:  41%|████      | 73/180 [00:02<00:03, 28.43batch/s]

Training batches on cuda:  42%|████▏     | 76/180 [00:02<00:03, 28.09batch/s]

Training batches on cuda:  44%|████▍     | 79/180 [00:02<00:03, 27.50batch/s]

Training batches on cuda:  46%|████▌     | 82/180 [00:03<00:03, 27.91batch/s]

Training batches on cuda:  47%|████▋     | 85/180 [00:03<00:03, 28.16batch/s]

Training batches on cuda:  49%|████▉     | 88/180 [00:03<00:03, 28.41batch/s]

Training batches on cuda:  51%|█████     | 91/180 [00:03<00:03, 28.29batch/s]

Training batches on cuda:  52%|█████▏    | 94/180 [00:03<00:03, 28.41batch/s]

Training batches on cuda:  54%|█████▍    | 97/180 [00:03<00:02, 28.54batch/s]

Training batches on cuda:  56%|█████▌    | 100/180 [00:03<00:02, 28.74batch/s]

Training batches on cuda:  57%|█████▋    | 103/180 [00:03<00:02, 28.95batch/s]

Training batches on cuda:  59%|█████▉    | 106/180 [00:03<00:02, 28.80batch/s]

Training batches on cuda:  61%|██████    | 109/180 [00:03<00:02, 28.54batch/s]

Training batches on cuda:  62%|██████▏   | 112/180 [00:04<00:02, 28.23batch/s]

Training batches on cuda:  64%|██████▍   | 115/180 [00:04<00:02, 28.08batch/s]

Training batches on cuda:  66%|██████▌   | 118/180 [00:04<00:02, 28.11batch/s]

Training batches on cuda:  67%|██████▋   | 121/180 [00:04<00:02, 27.52batch/s]

Training batches on cuda:  69%|██████▉   | 124/180 [00:04<00:02, 27.76batch/s]

Training batches on cuda:  71%|███████   | 127/180 [00:04<00:01, 27.76batch/s]

Training batches on cuda:  72%|███████▏  | 130/180 [00:04<00:01, 27.57batch/s]

Training batches on cuda:  74%|███████▍  | 133/180 [00:04<00:01, 27.71batch/s]

Training batches on cuda:  76%|███████▌  | 136/180 [00:04<00:01, 27.43batch/s]

Training batches on cuda:  77%|███████▋  | 139/180 [00:05<00:01, 27.67batch/s]

Training batches on cuda:  79%|███████▉  | 142/180 [00:05<00:01, 27.59batch/s]

Training batches on cuda:  81%|████████  | 145/180 [00:05<00:01, 27.64batch/s]

Training batches on cuda:  82%|████████▏ | 148/180 [00:05<00:01, 27.63batch/s]

Training batches on cuda:  84%|████████▍ | 151/180 [00:05<00:01, 27.54batch/s]

Training batches on cuda:  86%|████████▌ | 154/180 [00:05<00:00, 27.47batch/s]

Training batches on cuda:  87%|████████▋ | 157/180 [00:05<00:00, 27.64batch/s]

Training batches on cuda:  89%|████████▉ | 160/180 [00:05<00:00, 27.68batch/s]

Training batches on cuda:  91%|█████████ | 163/180 [00:05<00:00, 27.44batch/s]

Training batches on cuda:  92%|█████████▏| 166/180 [00:06<00:00, 27.47batch/s]

Training batches on cuda:  94%|█████████▍| 169/180 [00:06<00:00, 27.57batch/s]

Training batches on cuda:  96%|█████████▌| 172/180 [00:06<00:00, 27.66batch/s]

Training batches on cuda:  97%|█████████▋| 175/180 [00:06<00:00, 27.73batch/s]

Training batches on cuda:  99%|█████████▉| 178/180 [00:06<00:00, 27.73batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:20<00:52,  6.55s/epoch, loss=0.000761, prev_loss=0.000762]
Training epochs on cuda:  30%|███       | 3/10 [00:20<00:46,  6.62s/epoch, loss=0.000761, prev_loss=0.000762]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:10, 16.42batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:09, 18.81batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:08, 20.91batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:07, 22.69batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 24.37batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 25.54batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:06, 26.52batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 26.65batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 27.26batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 27.80batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 27.80batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:05, 27.64batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:05, 27.57batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:05, 27.50batch/s]

Training batches on cuda:  25%|██▌       | 45/180 [00:01<00:04, 28.19batch/s]

Training batches on cuda:  27%|██▋       | 48/180 [00:01<00:04, 28.22batch/s]

Training batches on cuda:  28%|██▊       | 51/180 [00:01<00:04, 28.07batch/s]

Training batches on cuda:  30%|███       | 54/180 [00:01<00:04, 28.22batch/s]

Training batches on cuda:  32%|███▏      | 57/180 [00:02<00:04, 28.31batch/s]

Training batches on cuda:  33%|███▎      | 60/180 [00:02<00:04, 28.39batch/s]

Training batches on cuda:  35%|███▌      | 63/180 [00:02<00:04, 28.53batch/s]

Training batches on cuda:  37%|███▋      | 66/180 [00:02<00:03, 28.72batch/s]

Training batches on cuda:  38%|███▊      | 69/180 [00:02<00:03, 28.66batch/s]

Training batches on cuda:  40%|████      | 72/180 [00:02<00:03, 28.85batch/s]

Training batches on cuda:  42%|████▏     | 75/180 [00:02<00:03, 27.91batch/s]

Training batches on cuda:  43%|████▎     | 78/180 [00:02<00:03, 27.81batch/s]

Training batches on cuda:  45%|████▌     | 81/180 [00:02<00:03, 27.50batch/s]

Training batches on cuda:  47%|████▋     | 84/180 [00:03<00:03, 28.00batch/s]

Training batches on cuda:  48%|████▊     | 87/180 [00:03<00:03, 27.61batch/s]

Training batches on cuda:  50%|█████     | 90/180 [00:03<00:03, 27.68batch/s]

Training batches on cuda:  52%|█████▏    | 93/180 [00:03<00:03, 27.78batch/s]

Training batches on cuda:  53%|█████▎    | 96/180 [00:03<00:03, 27.97batch/s]

Training batches on cuda:  55%|█████▌    | 99/180 [00:03<00:02, 27.94batch/s]

Training batches on cuda:  57%|█████▋    | 102/180 [00:03<00:02, 27.91batch/s]

Training batches on cuda:  58%|█████▊    | 105/180 [00:03<00:02, 28.11batch/s]

Training batches on cuda:  60%|██████    | 108/180 [00:03<00:02, 28.46batch/s]

Training batches on cuda:  62%|██████▏   | 111/180 [00:03<00:02, 28.45batch/s]

Training batches on cuda:  63%|██████▎   | 114/180 [00:04<00:02, 28.60batch/s]

Training batches on cuda:  65%|██████▌   | 117/180 [00:04<00:02, 28.87batch/s]

Training batches on cuda:  67%|██████▋   | 120/180 [00:04<00:02, 28.55batch/s]

Training batches on cuda:  68%|██████▊   | 123/180 [00:04<00:02, 28.44batch/s]

Training batches on cuda:  70%|███████   | 126/180 [00:04<00:01, 28.53batch/s]

Training batches on cuda:  72%|███████▏  | 129/180 [00:04<00:01, 28.56batch/s]

Training batches on cuda:  73%|███████▎  | 132/180 [00:04<00:01, 28.70batch/s]

Training batches on cuda:  75%|███████▌  | 135/180 [00:04<00:01, 28.07batch/s]

Training batches on cuda:  77%|███████▋  | 138/180 [00:04<00:01, 27.36batch/s]

Training batches on cuda:  78%|███████▊  | 141/180 [00:05<00:01, 27.39batch/s]

Training batches on cuda:  80%|████████  | 144/180 [00:05<00:01, 27.93batch/s]

Training batches on cuda:  82%|████████▏ | 147/180 [00:05<00:01, 28.00batch/s]

Training batches on cuda:  83%|████████▎ | 150/180 [00:05<00:01, 27.64batch/s]

Training batches on cuda:  85%|████████▌ | 153/180 [00:05<00:00, 27.83batch/s]

Training batches on cuda:  87%|████████▋ | 156/180 [00:05<00:00, 28.16batch/s]

Training batches on cuda:  88%|████████▊ | 159/180 [00:05<00:00, 28.31batch/s]

Training batches on cuda:  90%|█████████ | 162/180 [00:05<00:00, 28.53batch/s]

Training batches on cuda:  92%|█████████▏| 165/180 [00:05<00:00, 28.46batch/s]

Training batches on cuda:  93%|█████████▎| 168/180 [00:06<00:00, 28.29batch/s]

Training batches on cuda:  95%|█████████▌| 171/180 [00:06<00:00, 28.25batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:06<00:00, 28.49batch/s]

Training batches on cuda:  98%|█████████▊| 177/180 [00:06<00:00, 28.56batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:26<00:46,  6.62s/epoch, loss=0.000759, prev_loss=0.000761]
Training epochs on cuda:  40%|████      | 4/10 [00:26<00:39,  6.65s/epoch, loss=0.000759, prev_loss=0.000761]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:09, 18.94batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:08, 21.16batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:07, 22.82batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:06, 24.42batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 25.71batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 26.46batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:05, 27.14batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 27.53batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 27.40batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 27.94batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 27.88batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:05, 28.04batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:05, 28.26batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:04, 28.24batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:04, 28.18batch/s]

Training batches on cuda:  26%|██▌       | 47/180 [00:01<00:04, 28.40batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 28.23batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:04, 28.28batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:02<00:04, 27.96batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:02<00:04, 28.27batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:02<00:04, 27.77batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:04, 28.15batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:03, 28.02batch/s]

Training batches on cuda:  39%|███▉      | 71/180 [00:02<00:03, 27.72batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 27.61batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 27.69batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:03, 26.78batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:02<00:03, 26.78batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:03<00:03, 26.78batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:03<00:03, 26.12batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:03<00:03, 26.03batch/s]

Training batches on cuda:  53%|█████▎    | 95/180 [00:03<00:03, 26.62batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:03, 26.71batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 27.14batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:03<00:02, 27.34batch/s]

Training batches on cuda:  59%|█████▉    | 107/180 [00:03<00:02, 27.59batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 27.67batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:04<00:02, 27.50batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:04<00:02, 27.84batch/s]

Training batches on cuda:  66%|██████▌   | 119/180 [00:04<00:02, 28.12batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:04<00:02, 28.34batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:04<00:02, 24.37batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:04<00:02, 25.40batch/s]

Training batches on cuda:  73%|███████▎  | 131/180 [00:04<00:01, 26.34batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 26.96batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:04<00:01, 27.50batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:05<00:01, 27.91batch/s]

Training batches on cuda:  79%|███████▉  | 143/180 [00:05<00:01, 28.19batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:05<00:01, 28.29batch/s]

Training batches on cuda:  83%|████████▎ | 149/180 [00:05<00:01, 28.42batch/s]

Training batches on cuda:  84%|████████▍ | 152/180 [00:05<00:00, 28.41batch/s]

Training batches on cuda:  86%|████████▌ | 155/180 [00:05<00:00, 28.08batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:05<00:00, 28.08batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:05<00:00, 28.27batch/s]

Training batches on cuda:  91%|█████████ | 164/180 [00:05<00:00, 28.37batch/s]

Training batches on cuda:  93%|█████████▎| 167/180 [00:06<00:00, 28.41batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:06<00:00, 28.40batch/s]

Training batches on cuda:  96%|█████████▌| 173/180 [00:06<00:00, 28.61batch/s]

Training batches on cuda:  98%|█████████▊| 176/180 [00:06<00:00, 28.74batch/s]

Training batches on cuda:  99%|█████████▉| 179/180 [00:06<00:00, 28.83batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:33<00:39,  6.65s/epoch, loss=0.000748, prev_loss=0.000759]
Training epochs on cuda:  50%|█████     | 5/10 [00:33<00:33,  6.68s/epoch, loss=0.000748, prev_loss=0.000759]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:10, 16.35batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:09, 18.76batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:08, 20.51batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:07, 22.04batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:07, 23.27batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 24.50batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:06, 25.55batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 26.25batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 26.76batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 26.96batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 27.26batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:05, 27.62batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:05, 27.29batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:05, 26.48batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:05, 26.90batch/s]

Training batches on cuda:  26%|██▌       | 47/180 [00:01<00:04, 27.43batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 27.77batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:04, 27.71batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:02<00:04, 28.01batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:02<00:04, 28.35batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:02<00:04, 28.68batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:03, 28.78batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:03, 28.91batch/s]

Training batches on cuda:  39%|███▉      | 71/180 [00:02<00:03, 28.94batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 29.04batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 25.90batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:03, 26.82batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:03<00:03, 27.46batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:03<00:03, 27.95batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:03<00:03, 28.29batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:03<00:03, 28.43batch/s]

Training batches on cuda:  53%|█████▎    | 95/180 [00:03<00:02, 28.51batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 28.63batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 28.78batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:03<00:02, 28.68batch/s]

Training batches on cuda:  59%|█████▉    | 107/180 [00:03<00:02, 28.50batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 28.60batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:04<00:02, 28.77batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:04<00:02, 28.83batch/s]

Training batches on cuda:  66%|██████▌   | 119/180 [00:04<00:02, 28.23batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:04<00:02, 28.52batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:04<00:01, 28.68batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:04<00:01, 28.63batch/s]

Training batches on cuda:  73%|███████▎  | 131/180 [00:04<00:01, 28.76batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 28.59batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:04<00:01, 28.19batch/s]

Training batches on cuda:  78%|███████▊  | 141/180 [00:05<00:01, 28.86batch/s]

Training batches on cuda:  80%|████████  | 144/180 [00:05<00:01, 29.08batch/s]

Training batches on cuda:  82%|████████▏ | 148/180 [00:05<00:01, 29.60batch/s]

Training batches on cuda:  84%|████████▍ | 151/180 [00:05<00:00, 29.65batch/s]

Training batches on cuda:  86%|████████▌ | 155/180 [00:05<00:00, 29.76batch/s]

Training batches on cuda:  88%|████████▊ | 159/180 [00:05<00:00, 29.96batch/s]

Training batches on cuda:  90%|█████████ | 162/180 [00:05<00:00, 29.95batch/s]

Training batches on cuda:  92%|█████████▏| 166/180 [00:05<00:00, 30.26batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:06<00:00, 30.15batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:06<00:00, 29.33batch/s]

Training batches on cuda:  98%|█████████▊| 177/180 [00:06<00:00, 29.13batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:40<00:33,  6.68s/epoch, loss=0.000707, prev_loss=0.000748]
Training epochs on cuda:  60%|██████    | 6/10 [00:40<00:26,  6.67s/epoch, loss=0.000707, prev_loss=0.000748]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:09, 19.40batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:08, 21.53batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:07, 23.14batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:06, 24.53batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 25.48batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 26.29batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:05, 26.91batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 26.89batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 27.46batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 27.88batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 25.39batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:05, 26.10batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:05, 26.72batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:05, 27.44batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:04, 27.99batch/s]

Training batches on cuda:  26%|██▌       | 47/180 [00:01<00:04, 28.22batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 28.20batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:04, 28.20batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:02<00:04, 28.35batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:02<00:04, 28.52batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:02<00:04, 28.48batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:04, 28.44batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:03, 28.24batch/s]

Training batches on cuda:  39%|███▉      | 71/180 [00:02<00:03, 28.15batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 28.09batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 28.23batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:03, 28.31batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:02<00:03, 28.45batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:03<00:03, 27.97batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:03<00:03, 28.29batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:03<00:03, 28.23batch/s]

Training batches on cuda:  53%|█████▎    | 95/180 [00:03<00:03, 28.03batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 28.09batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 28.19batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:03<00:02, 28.25batch/s]

Training batches on cuda:  59%|█████▉    | 107/180 [00:03<00:02, 28.41batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 28.31batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:04<00:02, 28.21batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:04<00:02, 28.29batch/s]

Training batches on cuda:  66%|██████▌   | 119/180 [00:04<00:02, 28.65batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:04<00:02, 28.67batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:04<00:01, 28.52batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:04<00:01, 28.15batch/s]

Training batches on cuda:  73%|███████▎  | 131/180 [00:04<00:01, 28.02batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 27.98batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:04<00:01, 28.25batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:05<00:01, 28.42batch/s]

Training batches on cuda:  79%|███████▉  | 143/180 [00:05<00:01, 28.55batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:05<00:01, 28.27batch/s]

Training batches on cuda:  83%|████████▎ | 149/180 [00:05<00:01, 27.92batch/s]

Training batches on cuda:  84%|████████▍ | 152/180 [00:05<00:00, 28.04batch/s]

Training batches on cuda:  86%|████████▌ | 155/180 [00:05<00:00, 28.16batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:05<00:00, 28.25batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:05<00:00, 28.15batch/s]

Training batches on cuda:  91%|█████████ | 164/180 [00:05<00:00, 28.26batch/s]

Training batches on cuda:  93%|█████████▎| 167/180 [00:05<00:00, 28.14batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:06<00:00, 28.00batch/s]

Training batches on cuda:  96%|█████████▌| 173/180 [00:06<00:00, 26.50batch/s]

Training batches on cuda:  98%|█████████▊| 176/180 [00:06<00:00, 26.84batch/s]

Training batches on cuda:  99%|█████████▉| 179/180 [00:06<00:00, 27.27batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:46<00:26,  6.67s/epoch, loss=0.000586, prev_loss=0.000707]
Training epochs on cuda:  70%|███████   | 7/10 [00:46<00:20,  6.69s/epoch, loss=0.000586, prev_loss=0.000707]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:12, 14.80batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:10, 17.33batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:08, 19.54batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:07, 21.68batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:07, 23.40batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 24.84batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:06, 25.41batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 26.45batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 26.97batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 27.23batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 27.38batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:05, 27.82batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:05, 28.11batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:04, 28.21batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:04, 28.29batch/s]

Training batches on cuda:  26%|██▌       | 47/180 [00:01<00:04, 28.44batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 28.55batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:04, 28.45batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:02<00:04, 28.66batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:02<00:04, 28.63batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:02<00:04, 28.35batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:04, 28.43batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:03, 28.44batch/s]

Training batches on cuda:  39%|███▉      | 71/180 [00:02<00:03, 28.31batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:03, 28.17batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:03, 28.07batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:03, 28.17batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:02<00:03, 28.31batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:03<00:03, 28.03batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:03<00:03, 27.58batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:03<00:03, 27.71batch/s]

Training batches on cuda:  53%|█████▎    | 95/180 [00:03<00:03, 27.87batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 27.85batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 28.07batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:03<00:02, 28.42batch/s]

Training batches on cuda:  59%|█████▉    | 107/180 [00:03<00:02, 28.32batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:03<00:02, 28.09batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:04<00:02, 27.31batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:04<00:02, 27.74batch/s]

Training batches on cuda:  66%|██████▌   | 119/180 [00:04<00:02, 28.09batch/s]

Training batches on cuda:  68%|██████▊   | 122/180 [00:04<00:02, 28.32batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:04<00:02, 24.89batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:04<00:01, 26.14batch/s]

Training batches on cuda:  73%|███████▎  | 131/180 [00:04<00:01, 26.40batch/s]

Training batches on cuda:  74%|███████▍  | 134/180 [00:04<00:01, 26.76batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:04<00:01, 27.10batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:05<00:01, 27.17batch/s]

Training batches on cuda:  79%|███████▉  | 143/180 [00:05<00:01, 27.31batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:05<00:01, 27.60batch/s]

Training batches on cuda:  83%|████████▎ | 149/180 [00:05<00:01, 28.08batch/s]

Training batches on cuda:  84%|████████▍ | 152/180 [00:05<00:00, 28.16batch/s]

Training batches on cuda:  86%|████████▌ | 155/180 [00:05<00:00, 28.42batch/s]

Training batches on cuda:  88%|████████▊ | 158/180 [00:05<00:00, 28.54batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:05<00:00, 28.42batch/s]

Training batches on cuda:  91%|█████████ | 164/180 [00:05<00:00, 27.43batch/s]

Training batches on cuda:  93%|█████████▎| 167/180 [00:06<00:00, 27.22batch/s]

Training batches on cuda:  94%|█████████▍| 170/180 [00:06<00:00, 27.73batch/s]

Training batches on cuda:  96%|█████████▌| 173/180 [00:06<00:00, 27.86batch/s]

Training batches on cuda:  98%|█████████▊| 176/180 [00:06<00:00, 27.96batch/s]

Training batches on cuda:  99%|█████████▉| 179/180 [00:06<00:00, 27.29batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:53<00:20,  6.69s/epoch, loss=0.000514, prev_loss=0.000586]
Training epochs on cuda:  80%|████████  | 8/10 [00:53<00:13,  6.71s/epoch, loss=0.000514, prev_loss=0.000586]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/180 [00:00<00:10, 17.45batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:08, 19.59batch/s]

Training batches on cuda:   4%|▍         | 8/180 [00:00<00:08, 21.42batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:07, 23.13batch/s]

Training batches on cuda:   8%|▊         | 14/180 [00:00<00:06, 24.28batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:06, 25.19batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:06, 26.16batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:05, 26.90batch/s]

Training batches on cuda:  14%|█▍        | 26/180 [00:00<00:05, 27.08batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:01<00:05, 27.48batch/s]

Training batches on cuda:  18%|█▊        | 32/180 [00:01<00:05, 27.76batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:01<00:05, 27.23batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:01<00:05, 27.29batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:01<00:05, 27.11batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:01<00:05, 27.20batch/s]

Training batches on cuda:  26%|██▌       | 47/180 [00:01<00:04, 27.09batch/s]

Training batches on cuda:  28%|██▊       | 50/180 [00:01<00:04, 26.86batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:01<00:04, 26.72batch/s]

Training batches on cuda:  31%|███       | 56/180 [00:02<00:04, 26.79batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:02<00:04, 26.87batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:02<00:04, 27.03batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:02<00:04, 27.51batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:02<00:04, 27.64batch/s]

Training batches on cuda:  39%|███▉      | 71/180 [00:02<00:03, 28.07batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:02<00:04, 25.12batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:02<00:04, 25.74batch/s]

Training batches on cuda:  44%|████▍     | 80/180 [00:02<00:03, 26.01batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:03<00:03, 26.28batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:03<00:03, 26.63batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:03<00:03, 26.73batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:03<00:03, 26.98batch/s]

Training batches on cuda:  53%|█████▎    | 95/180 [00:03<00:03, 27.16batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:03<00:02, 27.60batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:03<00:02, 28.23batch/s]

Training batches on cuda:  58%|█████▊    | 105/180 [00:03<00:02, 28.95batch/s]

Training batches on cuda:  60%|██████    | 108/180 [00:03<00:02, 28.94batch/s]

Training batches on cuda:  62%|██████▏   | 111/180 [00:04<00:02, 28.67batch/s]

Training batches on cuda:  63%|██████▎   | 114/180 [00:04<00:02, 28.61batch/s]

Training batches on cuda:  65%|██████▌   | 117/180 [00:04<00:02, 28.17batch/s]

Training batches on cuda:  67%|██████▋   | 120/180 [00:04<00:02, 27.71batch/s]

Training batches on cuda:  68%|██████▊   | 123/180 [00:04<00:02, 28.11batch/s]

Training batches on cuda:  70%|███████   | 126/180 [00:04<00:01, 27.70batch/s]

Training batches on cuda:  72%|███████▏  | 129/180 [00:04<00:01, 27.24batch/s]

Training batches on cuda:  73%|███████▎  | 132/180 [00:04<00:01, 27.40batch/s]

Training batches on cuda:  75%|███████▌  | 135/180 [00:04<00:01, 27.21batch/s]

Training batches on cuda:  77%|███████▋  | 138/180 [00:05<00:01, 27.70batch/s]

Training batches on cuda:  78%|███████▊  | 141/180 [00:05<00:01, 27.63batch/s]

Training batches on cuda:  80%|████████  | 144/180 [00:05<00:01, 27.18batch/s]

Training batches on cuda:  82%|████████▏ | 147/180 [00:05<00:01, 27.14batch/s]

Training batches on cuda:  83%|████████▎ | 150/180 [00:05<00:01, 27.75batch/s]

Training batches on cuda:  85%|████████▌ | 153/180 [00:05<00:00, 27.83batch/s]

Training batches on cuda:  87%|████████▋ | 156/180 [00:05<00:00, 27.62batch/s]

Training batches on cuda:  88%|████████▊ | 159/180 [00:05<00:00, 27.60batch/s]

Training batches on cuda:  90%|█████████ | 162/180 [00:05<00:00, 27.44batch/s]

Training batches on cuda:  92%|█████████▏| 165/180 [00:06<00:00, 27.48batch/s]

Training batches on cuda:  93%|█████████▎| 168/180 [00:06<00:00, 27.23batch/s]

Training batches on cuda:  95%|█████████▌| 171/180 [00:06<00:00, 27.61batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:06<00:00, 27.32batch/s]

Training batches on cuda:  98%|█████████▊| 177/180 [00:06<00:00, 27.24batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [01:00<00:13,  6.71s/epoch, loss=0.000498, prev_loss=0.000514]
Training epochs on cuda:  90%|█████████ | 9/10 [01:00<00:06,  6.75s/epoch, loss=0.000498, prev_loss=0.000514]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/180 [00:00<00:18,  9.51batch/s]

Training batches on cuda:   2%|▏         | 4/180 [00:00<00:14, 11.89batch/s]

Training batches on cuda:   4%|▍         | 7/180 [00:00<00:12, 14.27batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:10, 16.77batch/s]

Training batches on cuda:   7%|▋         | 13/180 [00:00<00:08, 19.16batch/s]

Training batches on cuda:   9%|▉         | 16/180 [00:00<00:07, 21.22batch/s]

Training batches on cuda:  11%|█         | 19/180 [00:00<00:07, 22.83batch/s]

Training batches on cuda:  12%|█▏        | 22/180 [00:00<00:06, 24.21batch/s]

Training batches on cuda:  14%|█▍        | 25/180 [00:00<00:06, 25.39batch/s]

Training batches on cuda:  16%|█▌        | 28/180 [00:01<00:05, 25.91batch/s]

Training batches on cuda:  17%|█▋        | 31/180 [00:01<00:05, 26.79batch/s]

Training batches on cuda:  19%|█▉        | 34/180 [00:01<00:05, 27.15batch/s]

Training batches on cuda:  21%|██        | 37/180 [00:01<00:05, 27.50batch/s]

Training batches on cuda:  22%|██▏       | 40/180 [00:01<00:05, 27.34batch/s]

Training batches on cuda:  24%|██▍       | 43/180 [00:01<00:04, 27.53batch/s]

Training batches on cuda:  26%|██▌       | 46/180 [00:01<00:04, 27.82batch/s]

Training batches on cuda:  27%|██▋       | 49/180 [00:01<00:04, 27.87batch/s]

Training batches on cuda:  29%|██▉       | 52/180 [00:01<00:04, 28.10batch/s]

Training batches on cuda:  31%|███       | 55/180 [00:02<00:04, 28.39batch/s]

Training batches on cuda:  32%|███▏      | 58/180 [00:02<00:04, 28.56batch/s]

Training batches on cuda:  34%|███▍      | 61/180 [00:02<00:04, 27.79batch/s]

Training batches on cuda:  36%|███▌      | 64/180 [00:02<00:04, 27.95batch/s]

Training batches on cuda:  37%|███▋      | 67/180 [00:02<00:03, 28.26batch/s]

Training batches on cuda:  39%|███▉      | 70/180 [00:02<00:03, 28.32batch/s]

Training batches on cuda:  41%|████      | 73/180 [00:02<00:03, 28.61batch/s]

Training batches on cuda:  42%|████▏     | 76/180 [00:02<00:03, 28.64batch/s]

Training batches on cuda:  44%|████▍     | 79/180 [00:02<00:03, 28.51batch/s]

Training batches on cuda:  46%|████▌     | 82/180 [00:02<00:03, 28.48batch/s]

Training batches on cuda:  47%|████▋     | 85/180 [00:03<00:03, 27.37batch/s]

Training batches on cuda:  49%|████▉     | 88/180 [00:03<00:03, 27.72batch/s]

Training batches on cuda:  51%|█████     | 91/180 [00:03<00:03, 27.80batch/s]

Training batches on cuda:  52%|█████▏    | 94/180 [00:03<00:03, 27.84batch/s]

Training batches on cuda:  54%|█████▍    | 97/180 [00:03<00:03, 27.44batch/s]

Training batches on cuda:  56%|█████▌    | 100/180 [00:03<00:02, 27.13batch/s]

Training batches on cuda:  57%|█████▋    | 103/180 [00:03<00:02, 27.20batch/s]

Training batches on cuda:  59%|█████▉    | 106/180 [00:03<00:02, 27.64batch/s]

Training batches on cuda:  61%|██████    | 109/180 [00:03<00:02, 26.62batch/s]

Training batches on cuda:  62%|██████▏   | 112/180 [00:04<00:02, 27.09batch/s]

Training batches on cuda:  64%|██████▍   | 115/180 [00:04<00:02, 26.90batch/s]

Training batches on cuda:  66%|██████▌   | 118/180 [00:04<00:02, 27.37batch/s]

Training batches on cuda:  67%|██████▋   | 121/180 [00:04<00:02, 27.61batch/s]

Training batches on cuda:  69%|██████▉   | 124/180 [00:04<00:01, 28.22batch/s]

Training batches on cuda:  71%|███████   | 127/180 [00:04<00:01, 28.24batch/s]

Training batches on cuda:  72%|███████▏  | 130/180 [00:04<00:01, 28.42batch/s]

Training batches on cuda:  74%|███████▍  | 133/180 [00:04<00:01, 28.57batch/s]

Training batches on cuda:  76%|███████▌  | 136/180 [00:04<00:01, 28.40batch/s]

Training batches on cuda:  77%|███████▋  | 139/180 [00:05<00:01, 28.67batch/s]

Training batches on cuda:  79%|███████▉  | 142/180 [00:05<00:01, 28.24batch/s]

Training batches on cuda:  81%|████████  | 145/180 [00:05<00:01, 28.28batch/s]

Training batches on cuda:  82%|████████▏ | 148/180 [00:05<00:01, 28.22batch/s]

Training batches on cuda:  84%|████████▍ | 151/180 [00:05<00:01, 28.50batch/s]

Training batches on cuda:  86%|████████▌ | 154/180 [00:05<00:00, 28.55batch/s]

Training batches on cuda:  87%|████████▋ | 157/180 [00:05<00:00, 28.62batch/s]

Training batches on cuda:  89%|████████▉ | 160/180 [00:05<00:00, 28.86batch/s]

Training batches on cuda:  91%|█████████ | 163/180 [00:05<00:00, 28.94batch/s]

Training batches on cuda:  92%|█████████▏| 166/180 [00:05<00:00, 29.03batch/s]

Training batches on cuda:  94%|█████████▍| 169/180 [00:06<00:00, 28.90batch/s]

Training batches on cuda:  96%|█████████▌| 172/180 [00:06<00:00, 28.59batch/s]

Training batches on cuda:  97%|█████████▋| 175/180 [00:06<00:00, 28.48batch/s]

Training batches on cuda:  99%|█████████▉| 178/180 [00:06<00:00, 28.43batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [01:07<00:06,  6.75s/epoch, loss=0.000484, prev_loss=0.000498]2021-09-16:19:13:35,324 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:13:36,287 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:13:51,280 INFO     [evaluator.py:657] Evaluation took 14.99s seconds
2021-09-16:19:13:51,402 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00, 11.55s/epoch, loss=0.000484, prev_loss=0.000498]
Training epochs on cuda: 100%|██████████| 10/10 [01:23<00:00,  8.33s/epoch, loss=0.000484, prev_loss=0.000498]
2021-09-16:19:13:51,403 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpk_enymmy'
2021-09-16:19:13:51,463 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpk_enymmy' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:19, 2.02ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:17, 2.18ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:16, 2.30ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.39ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.46ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.51ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.55ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.61ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.61ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.62ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.62ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.62ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.62ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.59ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.58ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:14:06,372 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
[I 2021-09-16 19:14:06,377] Trial 72 finished with value: 0.9123631277094997 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.03650193558199926, 'optimizer.lr': 0.033582380067293624, 'negative_sampler.num_negs_per_pos': 57, 'training.batch_size': 1732}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:14:06,404 WARNING  [api.py:823] No random seed is specified. Setting to 3064723524.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/203 [00:00<00:08, 24.62batch/s]

Training batches on cuda:   4%|▍         | 8/203 [00:00<00:06, 28.00batch/s]

Training batches on cuda:   6%|▋         | 13/203 [00:00<00:06, 30.87batch/s]

Training batches on cuda:   9%|▉         | 18/203 [00:00<00:05, 33.77batch/s]

Training batches on cuda:  11%|█         | 22/203 [00:00<00:05, 33.94batch/s]

Training batches on cuda:  13%|█▎        | 27/203 [00:00<00:04, 35.91batch/s]

Training batches on cuda:  15%|█▌        | 31/203 [00:00<00:04, 35.83batch/s]

Training batches on cuda:  18%|█▊        | 36/203 [00:00<00:04, 37.78batch/s]

Training batches on cuda:  20%|██        | 41/203 [00:01<00:04, 38.88batch/s]

Training batches on cuda:  22%|██▏       | 45/203 [00:01<00:04, 38.85batch/s]

Training batches on cuda:  25%|██▍       | 50/203 [00:01<00:03, 39.76batch/s]

Training batches on cuda:  27%|██▋       | 54/203 [00:01<00:03, 39.35batch/s]

Training batches on cuda:  29%|██▊       | 58/203 [00:01<00:03, 38.77batch/s]

Training batches on cuda:  31%|███       | 62/203 [00:01<00:03, 38.21batch/s]

Training batches on cuda:  33%|███▎      | 66/203 [00:01<00:03, 36.94batch/s]

Training batches on cuda:  34%|███▍      | 70/203 [00:01<00:03, 37.71batch/s]

Training batches on cuda:  36%|███▋      | 74/203 [00:01<00:03, 37.32batch/s]

Training batches on cuda:  39%|███▉      | 79/203 [00:02<00:03, 38.16batch/s]

Training batches on cuda:  41%|████▏     | 84/203 [00:02<00:03, 38.88batch/s]

Training batches on cuda:  43%|████▎     | 88/203 [00:02<00:03, 37.08batch/s]

Training batches on cuda:  45%|████▌     | 92/203 [00:02<00:03, 35.74batch/s]

Training batches on cuda:  48%|████▊     | 97/203 [00:02<00:02, 36.78batch/s]

Training batches on cuda:  50%|█████     | 102/203 [00:02<00:02, 38.10batch/s]

Training batches on cuda:  52%|█████▏    | 106/203 [00:02<00:02, 37.93batch/s]

Training batches on cuda:  55%|█████▍    | 111/203 [00:02<00:02, 38.69batch/s]

Training batches on cuda:  57%|█████▋    | 115/203 [00:03<00:02, 35.74batch/s]

Training batches on cuda:  59%|█████▊    | 119/203 [00:03<00:02, 35.32batch/s]

Training batches on cuda:  61%|██████    | 123/203 [00:03<00:02, 34.83batch/s]

Training batches on cuda:  63%|██████▎   | 127/203 [00:03<00:02, 34.39batch/s]

Training batches on cuda:  65%|██████▍   | 131/203 [00:03<00:02, 35.28batch/s]

Training batches on cuda:  67%|██████▋   | 136/203 [00:03<00:01, 36.63batch/s]

Training batches on cuda:  69%|██████▉   | 141/203 [00:03<00:01, 37.66batch/s]

Training batches on cuda:  72%|███████▏  | 146/203 [00:03<00:01, 38.98batch/s]

Training batches on cuda:  74%|███████▍  | 151/203 [00:03<00:01, 40.23batch/s]

Training batches on cuda:  77%|███████▋  | 156/203 [00:04<00:01, 40.87batch/s]

Training batches on cuda:  79%|███████▉  | 161/203 [00:04<00:01, 41.15batch/s]

Training batches on cuda:  82%|████████▏ | 166/203 [00:04<00:00, 40.73batch/s]

Training batches on cuda:  84%|████████▍ | 171/203 [00:04<00:00, 39.76batch/s]

Training batches on cuda:  86%|████████▌ | 175/203 [00:04<00:00, 39.50batch/s]

Training batches on cuda:  88%|████████▊ | 179/203 [00:04<00:00, 38.39batch/s]

Training batches on cuda:  90%|█████████ | 183/203 [00:04<00:00, 38.70batch/s]

Training batches on cuda:  92%|█████████▏| 187/203 [00:04<00:00, 38.55batch/s]

Training batches on cuda:  95%|█████████▍| 192/203 [00:05<00:00, 38.98batch/s]

Training batches on cuda:  97%|█████████▋| 197/203 [00:05<00:00, 39.93batch/s]

Training batches on cuda: 100%|█████████▉| 202/203 [00:05<00:00, 38.05batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.000859, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:51,  5.68s/epoch, loss=0.000859, prev_loss=nan]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/203 [00:00<00:10, 18.92batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:08, 22.32batch/s]

Training batches on cuda:   5%|▍         | 10/203 [00:00<00:07, 25.44batch/s]

Training batches on cuda:   7%|▋         | 14/203 [00:00<00:06, 28.37batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:05, 31.31batch/s]

Training batches on cuda:  11%|█▏        | 23/203 [00:00<00:05, 32.90batch/s]

Training batches on cuda:  13%|█▎        | 27/203 [00:00<00:05, 34.59batch/s]

Training batches on cuda:  16%|█▌        | 32/203 [00:00<00:04, 36.31batch/s]

Training batches on cuda:  18%|█▊        | 36/203 [00:00<00:04, 37.25batch/s]

Training batches on cuda:  20%|█▉        | 40/203 [00:01<00:04, 37.12batch/s]

Training batches on cuda:  22%|██▏       | 44/203 [00:01<00:04, 37.06batch/s]

Training batches on cuda:  24%|██▎       | 48/203 [00:01<00:04, 37.29batch/s]

Training batches on cuda:  26%|██▌       | 53/203 [00:01<00:03, 37.91batch/s]

Training batches on cuda:  28%|██▊       | 57/203 [00:01<00:04, 36.25batch/s]

Training batches on cuda:  31%|███       | 62/203 [00:01<00:03, 37.71batch/s]

Training batches on cuda:  33%|███▎      | 66/203 [00:01<00:03, 38.23batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:01<00:03, 39.44batch/s]

Training batches on cuda:  37%|███▋      | 76/203 [00:02<00:03, 39.79batch/s]

Training batches on cuda:  39%|███▉      | 80/203 [00:02<00:03, 39.22batch/s]

Training batches on cuda:  42%|████▏     | 85/203 [00:02<00:02, 40.70batch/s]

Training batches on cuda:  44%|████▍     | 90/203 [00:02<00:03, 37.28batch/s]

Training batches on cuda:  46%|████▋     | 94/203 [00:02<00:02, 37.12batch/s]

Training batches on cuda:  48%|████▊     | 98/203 [00:02<00:02, 36.62batch/s]

Training batches on cuda:  50%|█████     | 102/203 [00:02<00:02, 36.35batch/s]

Training batches on cuda:  52%|█████▏    | 106/203 [00:02<00:02, 36.79batch/s]

Training batches on cuda:  54%|█████▍    | 110/203 [00:02<00:02, 37.25batch/s]

Training batches on cuda:  56%|█████▌    | 114/203 [00:03<00:02, 37.55batch/s]

Training batches on cuda:  58%|█████▊    | 118/203 [00:03<00:02, 37.29batch/s]

Training batches on cuda:  60%|██████    | 122/203 [00:03<00:02, 37.39batch/s]

Training batches on cuda:  63%|██████▎   | 127/203 [00:03<00:01, 38.48batch/s]

Training batches on cuda:  65%|██████▍   | 131/203 [00:03<00:01, 37.64batch/s]

Training batches on cuda:  67%|██████▋   | 136/203 [00:03<00:01, 38.57batch/s]

Training batches on cuda:  69%|██████▉   | 140/203 [00:03<00:01, 35.80batch/s]

Training batches on cuda:  71%|███████▏  | 145/203 [00:03<00:01, 37.19batch/s]

Training batches on cuda:  74%|███████▍  | 150/203 [00:03<00:01, 38.37batch/s]

Training batches on cuda:  76%|███████▌  | 154/203 [00:04<00:01, 38.25batch/s]

Training batches on cuda:  78%|███████▊  | 158/203 [00:04<00:01, 38.56batch/s]

Training batches on cuda:  80%|████████  | 163/203 [00:04<00:01, 39.05batch/s]

Training batches on cuda:  83%|████████▎ | 168/203 [00:04<00:00, 40.59batch/s]

Training batches on cuda:  85%|████████▌ | 173/203 [00:04<00:00, 41.26batch/s]

Training batches on cuda:  88%|████████▊ | 178/203 [00:04<00:00, 41.26batch/s]

Training batches on cuda:  90%|█████████ | 183/203 [00:04<00:00, 42.43batch/s]

Training batches on cuda:  93%|█████████▎| 188/203 [00:04<00:00, 42.63batch/s]

Training batches on cuda:  95%|█████████▌| 193/203 [00:05<00:00, 42.08batch/s]

Training batches on cuda:  98%|█████████▊| 198/203 [00:05<00:00, 40.94batch/s]

Training batches on cuda: 100%|██████████| 203/203 [00:05<00:00, 41.67batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:11<00:51,  5.68s/epoch, loss=0.000856, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:45,  5.68s/epoch, loss=0.000856, prev_loss=nan]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/203 [00:00<00:07, 27.01batch/s]

Training batches on cuda:   3%|▎         | 7/203 [00:00<00:06, 29.91batch/s]

Training batches on cuda:   5%|▌         | 11/203 [00:00<00:05, 32.01batch/s]

Training batches on cuda:   7%|▋         | 15/203 [00:00<00:05, 33.80batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:05, 34.58batch/s]

Training batches on cuda:  12%|█▏        | 24/203 [00:00<00:05, 35.68batch/s]

Training batches on cuda:  14%|█▍        | 28/203 [00:00<00:05, 34.31batch/s]

Training batches on cuda:  16%|█▌        | 32/203 [00:00<00:04, 35.01batch/s]

Training batches on cuda:  18%|█▊        | 36/203 [00:00<00:04, 36.19batch/s]

Training batches on cuda:  20%|██        | 41/203 [00:01<00:04, 37.67batch/s]

Training batches on cuda:  22%|██▏       | 45/203 [00:01<00:04, 37.95batch/s]

Training batches on cuda:  25%|██▍       | 50/203 [00:01<00:03, 39.03batch/s]

Training batches on cuda:  27%|██▋       | 54/203 [00:01<00:03, 38.68batch/s]

Training batches on cuda:  29%|██▊       | 58/203 [00:01<00:03, 38.18batch/s]

Training batches on cuda:  31%|███       | 62/203 [00:01<00:03, 36.57batch/s]

Training batches on cuda:  33%|███▎      | 66/203 [00:01<00:03, 37.09batch/s]

Training batches on cuda:  34%|███▍      | 70/203 [00:01<00:03, 37.89batch/s]

Training batches on cuda:  37%|███▋      | 75/203 [00:02<00:03, 38.43batch/s]

Training batches on cuda:  39%|███▉      | 80/203 [00:02<00:03, 38.69batch/s]

Training batches on cuda:  42%|████▏     | 85/203 [00:02<00:02, 39.50batch/s]

Training batches on cuda:  44%|████▍     | 90/203 [00:02<00:02, 41.07batch/s]

Training batches on cuda:  47%|████▋     | 95/203 [00:02<00:02, 42.12batch/s]

Training batches on cuda:  49%|████▉     | 100/203 [00:02<00:02, 41.48batch/s]

Training batches on cuda:  52%|█████▏    | 105/203 [00:02<00:02, 41.41batch/s]

Training batches on cuda:  54%|█████▍    | 110/203 [00:02<00:02, 40.84batch/s]

Training batches on cuda:  57%|█████▋    | 115/203 [00:02<00:02, 40.09batch/s]

Training batches on cuda:  59%|█████▉    | 120/203 [00:03<00:02, 41.03batch/s]

Training batches on cuda:  62%|██████▏   | 125/203 [00:03<00:01, 39.88batch/s]

Training batches on cuda:  64%|██████▍   | 130/203 [00:03<00:01, 38.68batch/s]

Training batches on cuda:  66%|██████▌   | 134/203 [00:03<00:01, 37.73batch/s]

Training batches on cuda:  68%|██████▊   | 138/203 [00:03<00:01, 38.07batch/s]

Training batches on cuda:  70%|██████▉   | 142/203 [00:03<00:01, 36.74batch/s]

Training batches on cuda:  72%|███████▏  | 146/203 [00:03<00:01, 37.15batch/s]

Training batches on cuda:  74%|███████▍  | 150/203 [00:03<00:01, 36.46batch/s]

Training batches on cuda:  76%|███████▌  | 154/203 [00:04<00:01, 36.91batch/s]

Training batches on cuda:  78%|███████▊  | 158/203 [00:04<00:01, 37.58batch/s]

Training batches on cuda:  80%|████████  | 163/203 [00:04<00:01, 38.57batch/s]

Training batches on cuda:  82%|████████▏ | 167/203 [00:04<00:00, 38.59batch/s]

Training batches on cuda:  85%|████████▍ | 172/203 [00:04<00:00, 38.74batch/s]

Training batches on cuda:  87%|████████▋ | 176/203 [00:04<00:00, 36.76batch/s]

Training batches on cuda:  89%|████████▊ | 180/203 [00:04<00:00, 37.28batch/s]

Training batches on cuda:  91%|█████████ | 185/203 [00:04<00:00, 38.37batch/s]

Training batches on cuda:  93%|█████████▎| 189/203 [00:04<00:00, 38.68batch/s]

Training batches on cuda:  96%|█████████▌| 194/203 [00:05<00:00, 40.78batch/s]

Training batches on cuda:  98%|█████████▊| 199/203 [00:05<00:00, 41.85batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:16<00:45,  5.68s/epoch, loss=0.000802, prev_loss=0.000856]
Training epochs on cuda:  30%|███       | 3/10 [00:16<00:39,  5.65s/epoch, loss=0.000802, prev_loss=0.000856]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/203 [00:00<00:10, 18.54batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:09, 21.36batch/s]

Training batches on cuda:   5%|▍         | 10/203 [00:00<00:07, 24.78batch/s]

Training batches on cuda:   7%|▋         | 15/203 [00:00<00:06, 28.55batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:05, 31.23batch/s]

Training batches on cuda:  11%|█▏        | 23/203 [00:00<00:05, 32.48batch/s]

Training batches on cuda:  13%|█▎        | 27/203 [00:00<00:05, 32.53batch/s]

Training batches on cuda:  15%|█▌        | 31/203 [00:00<00:05, 33.08batch/s]

Training batches on cuda:  17%|█▋        | 35/203 [00:01<00:04, 33.72batch/s]

Training batches on cuda:  19%|█▉        | 39/203 [00:01<00:04, 33.87batch/s]

Training batches on cuda:  21%|██        | 43/203 [00:01<00:04, 33.43batch/s]

Training batches on cuda:  23%|██▎       | 47/203 [00:01<00:04, 34.45batch/s]

Training batches on cuda:  25%|██▌       | 51/203 [00:01<00:04, 34.91batch/s]

Training batches on cuda:  27%|██▋       | 55/203 [00:01<00:04, 34.87batch/s]

Training batches on cuda:  29%|██▉       | 59/203 [00:01<00:03, 36.09batch/s]

Training batches on cuda:  31%|███       | 63/203 [00:01<00:03, 37.15batch/s]

Training batches on cuda:  33%|███▎      | 67/203 [00:01<00:03, 37.44batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:01<00:03, 37.22batch/s]

Training batches on cuda:  37%|███▋      | 75/203 [00:02<00:03, 35.30batch/s]

Training batches on cuda:  39%|███▉      | 79/203 [00:02<00:03, 36.02batch/s]

Training batches on cuda:  41%|████      | 83/203 [00:02<00:03, 36.72batch/s]

Training batches on cuda:  43%|████▎     | 87/203 [00:02<00:03, 37.00batch/s]

Training batches on cuda:  45%|████▍     | 91/203 [00:02<00:03, 36.18batch/s]

Training batches on cuda:  47%|████▋     | 95/203 [00:02<00:03, 35.33batch/s]

Training batches on cuda:  49%|████▉     | 99/203 [00:02<00:03, 33.82batch/s]

Training batches on cuda:  51%|█████     | 103/203 [00:02<00:02, 35.08batch/s]

Training batches on cuda:  53%|█████▎    | 107/203 [00:03<00:02, 35.13batch/s]

Training batches on cuda:  55%|█████▍    | 111/203 [00:03<00:02, 36.24batch/s]

Training batches on cuda:  57%|█████▋    | 115/203 [00:03<00:02, 37.02batch/s]

Training batches on cuda:  59%|█████▊    | 119/203 [00:03<00:02, 36.41batch/s]

Training batches on cuda:  61%|██████    | 123/203 [00:03<00:02, 37.05batch/s]

Training batches on cuda:  63%|██████▎   | 127/203 [00:03<00:02, 37.09batch/s]

Training batches on cuda:  65%|██████▍   | 131/203 [00:03<00:01, 37.10batch/s]

Training batches on cuda:  67%|██████▋   | 135/203 [00:03<00:01, 36.79batch/s]

Training batches on cuda:  69%|██████▉   | 140/203 [00:03<00:01, 38.24batch/s]

Training batches on cuda:  71%|███████   | 144/203 [00:03<00:01, 38.11batch/s]

Training batches on cuda:  73%|███████▎  | 148/203 [00:04<00:01, 38.22batch/s]

Training batches on cuda:  75%|███████▍  | 152/203 [00:04<00:01, 38.25batch/s]

Training batches on cuda:  77%|███████▋  | 156/203 [00:04<00:01, 38.43batch/s]

Training batches on cuda:  79%|███████▉  | 161/203 [00:04<00:01, 38.78batch/s]

Training batches on cuda:  81%|████████▏ | 165/203 [00:04<00:01, 37.45batch/s]

Training batches on cuda:  83%|████████▎ | 169/203 [00:04<00:00, 37.92batch/s]

Training batches on cuda:  86%|████████▌ | 174/203 [00:04<00:00, 38.86batch/s]

Training batches on cuda:  88%|████████▊ | 178/203 [00:04<00:00, 38.79batch/s]

Training batches on cuda:  90%|████████▉ | 182/203 [00:04<00:00, 39.13batch/s]

Training batches on cuda:  92%|█████████▏| 186/203 [00:05<00:00, 39.39batch/s]

Training batches on cuda:  94%|█████████▎| 190/203 [00:05<00:00, 38.79batch/s]

Training batches on cuda:  96%|█████████▌| 194/203 [00:05<00:00, 37.71batch/s]

Training batches on cuda:  98%|█████████▊| 198/203 [00:05<00:00, 37.70batch/s]

Training batches on cuda: 100%|█████████▉| 202/203 [00:05<00:00, 38.15batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:22<00:39,  5.65s/epoch, loss=0.000591, prev_loss=0.000802]
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:34,  5.71s/epoch, loss=0.000591, prev_loss=0.000802]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/203 [00:00<00:07, 26.36batch/s]

Training batches on cuda:   3%|▎         | 7/203 [00:00<00:06, 29.22batch/s]

Training batches on cuda:   5%|▌         | 11/203 [00:00<00:06, 30.74batch/s]

Training batches on cuda:   8%|▊         | 16/203 [00:00<00:05, 33.61batch/s]

Training batches on cuda:  10%|█         | 21/203 [00:00<00:05, 35.54batch/s]

Training batches on cuda:  12%|█▏        | 25/203 [00:00<00:04, 36.48batch/s]

Training batches on cuda:  15%|█▍        | 30/203 [00:00<00:04, 37.69batch/s]

Training batches on cuda:  17%|█▋        | 34/203 [00:00<00:04, 37.43batch/s]

Training batches on cuda:  19%|█▊        | 38/203 [00:01<00:04, 35.78batch/s]

Training batches on cuda:  21%|██        | 43/203 [00:01<00:04, 37.13batch/s]

Training batches on cuda:  23%|██▎       | 47/203 [00:01<00:04, 37.28batch/s]

Training batches on cuda:  25%|██▌       | 51/203 [00:01<00:03, 38.02batch/s]

Training batches on cuda:  28%|██▊       | 56/203 [00:01<00:03, 38.97batch/s]

Training batches on cuda:  30%|███       | 61/203 [00:01<00:03, 39.48batch/s]

Training batches on cuda:  33%|███▎      | 66/203 [00:01<00:03, 39.93batch/s]

Training batches on cuda:  34%|███▍      | 70/203 [00:01<00:03, 36.99batch/s]

Training batches on cuda:  36%|███▋      | 74/203 [00:01<00:03, 36.54batch/s]

Training batches on cuda:  39%|███▉      | 79/203 [00:02<00:03, 38.19batch/s]

Training batches on cuda:  41%|████▏     | 84/203 [00:02<00:03, 38.92batch/s]

Training batches on cuda:  44%|████▍     | 89/203 [00:02<00:02, 39.65batch/s]

Training batches on cuda:  46%|████▌     | 93/203 [00:02<00:02, 38.63batch/s]

Training batches on cuda:  48%|████▊     | 97/203 [00:02<00:02, 38.08batch/s]

Training batches on cuda:  50%|████▉     | 101/203 [00:02<00:02, 36.52batch/s]

Training batches on cuda:  52%|█████▏    | 105/203 [00:02<00:02, 36.24batch/s]

Training batches on cuda:  54%|█████▍    | 110/203 [00:02<00:02, 37.41batch/s]

Training batches on cuda:  56%|█████▌    | 114/203 [00:02<00:02, 37.72batch/s]

Training batches on cuda:  58%|█████▊    | 118/203 [00:03<00:02, 38.15batch/s]

Training batches on cuda:  60%|██████    | 122/203 [00:03<00:02, 38.55batch/s]

Training batches on cuda:  62%|██████▏   | 126/203 [00:03<00:02, 37.54batch/s]

Training batches on cuda:  64%|██████▍   | 130/203 [00:03<00:01, 37.60batch/s]

Training batches on cuda:  66%|██████▌   | 134/203 [00:03<00:01, 36.84batch/s]

Training batches on cuda:  68%|██████▊   | 139/203 [00:03<00:01, 37.96batch/s]

Training batches on cuda:  71%|███████   | 144/203 [00:03<00:01, 38.61batch/s]

Training batches on cuda:  73%|███████▎  | 148/203 [00:03<00:01, 38.82batch/s]

Training batches on cuda:  75%|███████▍  | 152/203 [00:03<00:01, 39.01batch/s]

Training batches on cuda:  77%|███████▋  | 157/203 [00:04<00:01, 39.75batch/s]

Training batches on cuda:  80%|███████▉  | 162/203 [00:04<00:01, 40.86batch/s]

Training batches on cuda:  82%|████████▏ | 167/203 [00:04<00:00, 40.95batch/s]

Training batches on cuda:  85%|████████▍ | 172/203 [00:04<00:00, 39.97batch/s]

Training batches on cuda:  87%|████████▋ | 177/203 [00:04<00:00, 39.25batch/s]

Training batches on cuda:  90%|████████▉ | 182/203 [00:04<00:00, 39.64batch/s]

Training batches on cuda:  92%|█████████▏| 186/203 [00:04<00:00, 38.94batch/s]

Training batches on cuda:  94%|█████████▎| 190/203 [00:04<00:00, 38.82batch/s]

Training batches on cuda:  96%|█████████▌| 194/203 [00:05<00:00, 38.43batch/s]

Training batches on cuda:  98%|█████████▊| 198/203 [00:05<00:00, 37.41batch/s]

Training batches on cuda: 100%|█████████▉| 202/203 [00:05<00:00, 34.93batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:28<00:34,  5.71s/epoch, loss=0.000481, prev_loss=0.000591]
Training epochs on cuda:  50%|█████     | 5/10 [00:28<00:28,  5.70s/epoch, loss=0.000481, prev_loss=0.000591]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/203 [00:00<00:10, 18.34batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:09, 21.58batch/s]

Training batches on cuda:   5%|▍         | 10/203 [00:00<00:07, 24.33batch/s]

Training batches on cuda:   7%|▋         | 15/203 [00:00<00:06, 27.80batch/s]

Training batches on cuda:  10%|▉         | 20/203 [00:00<00:05, 30.87batch/s]

Training batches on cuda:  12%|█▏        | 24/203 [00:00<00:05, 30.57batch/s]

Training batches on cuda:  14%|█▍        | 28/203 [00:00<00:05, 32.38batch/s]

Training batches on cuda:  16%|█▋        | 33/203 [00:00<00:04, 34.87batch/s]

Training batches on cuda:  19%|█▊        | 38/203 [00:01<00:04, 35.95batch/s]

Training batches on cuda:  21%|██        | 42/203 [00:01<00:04, 34.85batch/s]

Training batches on cuda:  23%|██▎       | 47/203 [00:01<00:04, 37.01batch/s]

Training batches on cuda:  26%|██▌       | 52/203 [00:01<00:03, 38.49batch/s]

Training batches on cuda:  28%|██▊       | 56/203 [00:01<00:03, 37.70batch/s]

Training batches on cuda:  30%|██▉       | 60/203 [00:01<00:03, 38.08batch/s]

Training batches on cuda:  32%|███▏      | 64/203 [00:01<00:03, 37.85batch/s]

Training batches on cuda:  34%|███▍      | 69/203 [00:01<00:03, 39.31batch/s]

Training batches on cuda:  36%|███▌      | 73/203 [00:01<00:03, 39.05batch/s]

Training batches on cuda:  38%|███▊      | 77/203 [00:02<00:03, 39.24batch/s]

Training batches on cuda:  40%|████      | 82/203 [00:02<00:03, 39.76batch/s]

Training batches on cuda:  42%|████▏     | 86/203 [00:02<00:02, 39.83batch/s]

Training batches on cuda:  44%|████▍     | 90/203 [00:02<00:02, 38.54batch/s]

Training batches on cuda:  46%|████▋     | 94/203 [00:02<00:02, 37.79batch/s]

Training batches on cuda:  49%|████▉     | 99/203 [00:02<00:02, 38.84batch/s]

Training batches on cuda:  51%|█████     | 103/203 [00:02<00:02, 39.13batch/s]

Training batches on cuda:  53%|█████▎    | 107/203 [00:02<00:02, 39.21batch/s]

Training batches on cuda:  55%|█████▍    | 111/203 [00:02<00:02, 38.32batch/s]

Training batches on cuda:  57%|█████▋    | 116/203 [00:03<00:02, 39.05batch/s]

Training batches on cuda:  60%|█████▉    | 121/203 [00:03<00:02, 39.35batch/s]

Training batches on cuda:  62%|██████▏   | 125/203 [00:03<00:02, 37.98batch/s]

Training batches on cuda:  64%|██████▍   | 130/203 [00:03<00:01, 39.32batch/s]

Training batches on cuda:  67%|██████▋   | 135/203 [00:03<00:01, 39.76batch/s]

Training batches on cuda:  69%|██████▉   | 140/203 [00:03<00:01, 40.41batch/s]

Training batches on cuda:  71%|███████▏  | 145/203 [00:03<00:01, 39.82batch/s]

Training batches on cuda:  74%|███████▍  | 150/203 [00:03<00:01, 40.87batch/s]

Training batches on cuda:  76%|███████▋  | 155/203 [00:04<00:01, 42.36batch/s]

Training batches on cuda:  79%|███████▉  | 160/203 [00:04<00:01, 42.81batch/s]

Training batches on cuda:  81%|████████▏ | 165/203 [00:04<00:00, 42.95batch/s]

Training batches on cuda:  84%|████████▎ | 170/203 [00:04<00:00, 41.79batch/s]

Training batches on cuda:  86%|████████▌ | 175/203 [00:04<00:00, 41.32batch/s]

Training batches on cuda:  89%|████████▊ | 180/203 [00:04<00:00, 42.18batch/s]

Training batches on cuda:  91%|█████████ | 185/203 [00:04<00:00, 41.37batch/s]

Training batches on cuda:  94%|█████████▎| 190/203 [00:04<00:00, 40.50batch/s]

Training batches on cuda:  96%|█████████▌| 195/203 [00:04<00:00, 40.98batch/s]

Training batches on cuda:  99%|█████████▊| 200/203 [00:05<00:00, 39.43batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:33<00:28,  5.70s/epoch, loss=0.000449, prev_loss=0.000481]
Training epochs on cuda:  60%|██████    | 6/10 [00:33<00:22,  5.64s/epoch, loss=0.000449, prev_loss=0.000481]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/203 [00:00<00:11, 17.45batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:09, 20.40batch/s]

Training batches on cuda:   5%|▍         | 10/203 [00:00<00:08, 23.64batch/s]

Training batches on cuda:   7%|▋         | 14/203 [00:00<00:07, 26.25batch/s]

Training batches on cuda:   9%|▉         | 18/203 [00:00<00:06, 28.73batch/s]

Training batches on cuda:  11%|█         | 22/203 [00:00<00:06, 29.62batch/s]

Training batches on cuda:  12%|█▏        | 25/203 [00:00<00:05, 29.68batch/s]

Training batches on cuda:  14%|█▍        | 29/203 [00:00<00:05, 31.81batch/s]

Training batches on cuda:  17%|█▋        | 34/203 [00:01<00:04, 34.13batch/s]

Training batches on cuda:  19%|█▉        | 39/203 [00:01<00:04, 35.95batch/s]

Training batches on cuda:  22%|██▏       | 44/203 [00:01<00:04, 38.16batch/s]

Training batches on cuda:  24%|██▍       | 49/203 [00:01<00:03, 39.27batch/s]

Training batches on cuda:  27%|██▋       | 54/203 [00:01<00:03, 39.11batch/s]

Training batches on cuda:  29%|██▊       | 58/203 [00:01<00:03, 37.74batch/s]

Training batches on cuda:  31%|███       | 63/203 [00:01<00:03, 39.77batch/s]

Training batches on cuda:  33%|███▎      | 68/203 [00:01<00:03, 40.83batch/s]

Training batches on cuda:  36%|███▌      | 73/203 [00:01<00:03, 40.86batch/s]

Training batches on cuda:  38%|███▊      | 78/203 [00:02<00:03, 41.31batch/s]

Training batches on cuda:  41%|████      | 83/203 [00:02<00:02, 40.80batch/s]

Training batches on cuda:  43%|████▎     | 88/203 [00:02<00:02, 41.13batch/s]

Training batches on cuda:  46%|████▌     | 93/203 [00:02<00:02, 41.31batch/s]

Training batches on cuda:  48%|████▊     | 98/203 [00:02<00:02, 41.46batch/s]

Training batches on cuda:  51%|█████     | 103/203 [00:02<00:02, 41.37batch/s]

Training batches on cuda:  53%|█████▎    | 108/203 [00:02<00:02, 41.01batch/s]

Training batches on cuda:  56%|█████▌    | 113/203 [00:02<00:02, 40.92batch/s]

Training batches on cuda:  58%|█████▊    | 118/203 [00:03<00:02, 40.78batch/s]

Training batches on cuda:  61%|██████    | 123/203 [00:03<00:01, 41.49batch/s]

Training batches on cuda:  63%|██████▎   | 128/203 [00:03<00:01, 41.44batch/s]

Training batches on cuda:  66%|██████▌   | 133/203 [00:03<00:01, 41.43batch/s]

Training batches on cuda:  68%|██████▊   | 138/203 [00:03<00:01, 41.36batch/s]

Training batches on cuda:  70%|███████   | 143/203 [00:03<00:01, 41.96batch/s]

Training batches on cuda:  73%|███████▎  | 148/203 [00:03<00:01, 41.77batch/s]

Training batches on cuda:  75%|███████▌  | 153/203 [00:03<00:01, 40.81batch/s]

Training batches on cuda:  78%|███████▊  | 158/203 [00:04<00:01, 40.78batch/s]

Training batches on cuda:  80%|████████  | 163/203 [00:04<00:00, 40.60batch/s]

Training batches on cuda:  83%|████████▎ | 168/203 [00:04<00:00, 37.77batch/s]

Training batches on cuda:  85%|████████▍ | 172/203 [00:04<00:00, 37.44batch/s]

Training batches on cuda:  87%|████████▋ | 176/203 [00:04<00:00, 36.00batch/s]

Training batches on cuda:  89%|████████▊ | 180/203 [00:04<00:00, 34.83batch/s]

Training batches on cuda:  91%|█████████ | 184/203 [00:04<00:00, 35.29batch/s]

Training batches on cuda:  93%|█████████▎| 188/203 [00:04<00:00, 36.17batch/s]

Training batches on cuda:  95%|█████████▌| 193/203 [00:04<00:00, 37.43batch/s]

Training batches on cuda:  98%|█████████▊| 198/203 [00:05<00:00, 38.75batch/s]

Training batches on cuda: 100%|█████████▉| 202/203 [00:05<00:00, 39.02batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:39<00:22,  5.64s/epoch, loss=0.000434, prev_loss=0.000449]
Training epochs on cuda:  70%|███████   | 7/10 [00:39<00:16,  5.61s/epoch, loss=0.000434, prev_loss=0.000449]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/203 [00:00<00:07, 26.78batch/s]

Training batches on cuda:   3%|▎         | 7/203 [00:00<00:06, 29.42batch/s]

Training batches on cuda:   6%|▌         | 12/203 [00:00<00:05, 32.09batch/s]

Training batches on cuda:   8%|▊         | 17/203 [00:00<00:05, 34.51batch/s]

Training batches on cuda:  11%|█         | 22/203 [00:00<00:04, 36.36batch/s]

Training batches on cuda:  13%|█▎        | 27/203 [00:00<00:04, 38.14batch/s]

Training batches on cuda:  16%|█▌        | 32/203 [00:00<00:04, 39.47batch/s]

Training batches on cuda:  18%|█▊        | 36/203 [00:00<00:04, 39.46batch/s]

Training batches on cuda:  20%|█▉        | 40/203 [00:01<00:04, 38.88batch/s]

Training batches on cuda:  22%|██▏       | 45/203 [00:01<00:03, 39.76batch/s]

Training batches on cuda:  25%|██▍       | 50/203 [00:01<00:03, 40.92batch/s]

Training batches on cuda:  27%|██▋       | 55/203 [00:01<00:03, 40.98batch/s]

Training batches on cuda:  30%|██▉       | 60/203 [00:01<00:03, 41.72batch/s]

Training batches on cuda:  32%|███▏      | 65/203 [00:01<00:03, 42.24batch/s]

Training batches on cuda:  34%|███▍      | 70/203 [00:01<00:03, 42.76batch/s]

Training batches on cuda:  37%|███▋      | 75/203 [00:01<00:02, 42.85batch/s]

Training batches on cuda:  39%|███▉      | 80/203 [00:01<00:03, 40.86batch/s]

Training batches on cuda:  42%|████▏     | 85/203 [00:02<00:02, 41.35batch/s]

Training batches on cuda:  44%|████▍     | 90/203 [00:02<00:02, 41.45batch/s]

Training batches on cuda:  47%|████▋     | 95/203 [00:02<00:02, 41.95batch/s]

Training batches on cuda:  49%|████▉     | 100/203 [00:02<00:02, 42.36batch/s]

Training batches on cuda:  52%|█████▏    | 105/203 [00:02<00:02, 42.56batch/s]

Training batches on cuda:  54%|█████▍    | 110/203 [00:02<00:02, 43.03batch/s]

Training batches on cuda:  57%|█████▋    | 115/203 [00:02<00:02, 42.95batch/s]

Training batches on cuda:  59%|█████▉    | 120/203 [00:02<00:01, 43.14batch/s]

Training batches on cuda:  62%|██████▏   | 125/203 [00:03<00:01, 43.00batch/s]

Training batches on cuda:  64%|██████▍   | 130/203 [00:03<00:01, 43.19batch/s]

Training batches on cuda:  67%|██████▋   | 135/203 [00:03<00:01, 43.44batch/s]

Training batches on cuda:  69%|██████▉   | 140/203 [00:03<00:01, 43.56batch/s]

Training batches on cuda:  71%|███████▏  | 145/203 [00:03<00:01, 42.19batch/s]

Training batches on cuda:  74%|███████▍  | 150/203 [00:03<00:01, 42.93batch/s]

Training batches on cuda:  76%|███████▋  | 155/203 [00:03<00:01, 43.10batch/s]

Training batches on cuda:  79%|███████▉  | 160/203 [00:03<00:00, 43.33batch/s]

Training batches on cuda:  81%|████████▏ | 165/203 [00:03<00:00, 43.37batch/s]

Training batches on cuda:  84%|████████▎ | 170/203 [00:04<00:00, 43.51batch/s]

Training batches on cuda:  86%|████████▌ | 175/203 [00:04<00:00, 42.49batch/s]

Training batches on cuda:  89%|████████▊ | 180/203 [00:04<00:00, 42.73batch/s]

Training batches on cuda:  91%|█████████ | 185/203 [00:04<00:00, 42.24batch/s]

Training batches on cuda:  94%|█████████▎| 190/203 [00:04<00:00, 41.65batch/s]

Training batches on cuda:  96%|█████████▌| 195/203 [00:04<00:00, 41.80batch/s]

Training batches on cuda:  99%|█████████▊| 200/203 [00:04<00:00, 40.69batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:44<00:16,  5.61s/epoch, loss=0.000417, prev_loss=0.000434]
Training epochs on cuda:  80%|████████  | 8/10 [00:44<00:10,  5.47s/epoch, loss=0.000417, prev_loss=0.000434]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   0%|          | 1/203 [00:00<00:22,  9.16batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:16, 11.92batch/s]

Training batches on cuda:   5%|▌         | 11/203 [00:00<00:12, 15.18batch/s]

Training batches on cuda:   8%|▊         | 16/203 [00:00<00:10, 18.66batch/s]

Training batches on cuda:  10%|▉         | 20/203 [00:00<00:08, 22.05batch/s]

Training batches on cuda:  12%|█▏        | 25/203 [00:00<00:06, 25.90batch/s]

Training batches on cuda:  15%|█▍        | 30/203 [00:00<00:05, 29.39batch/s]

Training batches on cuda:  17%|█▋        | 35/203 [00:00<00:05, 32.50batch/s]

Training batches on cuda:  20%|█▉        | 40/203 [00:01<00:04, 34.70batch/s]

Training batches on cuda:  22%|██▏       | 45/203 [00:01<00:04, 36.72batch/s]

Training batches on cuda:  25%|██▍       | 50/203 [00:01<00:03, 38.42batch/s]

Training batches on cuda:  27%|██▋       | 55/203 [00:01<00:03, 38.77batch/s]

Training batches on cuda:  30%|██▉       | 60/203 [00:01<00:03, 39.08batch/s]

Training batches on cuda:  32%|███▏      | 65/203 [00:01<00:03, 39.08batch/s]

Training batches on cuda:  34%|███▍      | 70/203 [00:01<00:03, 38.17batch/s]

Training batches on cuda:  36%|███▋      | 74/203 [00:01<00:03, 37.68batch/s]

Training batches on cuda:  38%|███▊      | 78/203 [00:02<00:03, 37.46batch/s]

Training batches on cuda:  40%|████      | 82/203 [00:02<00:03, 37.42batch/s]

Training batches on cuda:  42%|████▏     | 86/203 [00:02<00:03, 37.88batch/s]

Training batches on cuda:  44%|████▍     | 90/203 [00:02<00:02, 38.17batch/s]

Training batches on cuda:  46%|████▋     | 94/203 [00:02<00:02, 38.39batch/s]

Training batches on cuda:  48%|████▊     | 98/203 [00:02<00:02, 38.53batch/s]

Training batches on cuda:  50%|█████     | 102/203 [00:02<00:02, 38.15batch/s]

Training batches on cuda:  52%|█████▏    | 106/203 [00:02<00:02, 38.09batch/s]

Training batches on cuda:  54%|█████▍    | 110/203 [00:02<00:02, 37.70batch/s]

Training batches on cuda:  56%|█████▌    | 114/203 [00:02<00:02, 37.93batch/s]

Training batches on cuda:  58%|█████▊    | 118/203 [00:03<00:02, 37.82batch/s]

Training batches on cuda:  60%|██████    | 122/203 [00:03<00:02, 37.92batch/s]

Training batches on cuda:  62%|██████▏   | 126/203 [00:03<00:02, 38.11batch/s]

Training batches on cuda:  64%|██████▍   | 130/203 [00:03<00:01, 38.35batch/s]

Training batches on cuda:  67%|██████▋   | 135/203 [00:03<00:01, 39.08batch/s]

Training batches on cuda:  68%|██████▊   | 139/203 [00:03<00:01, 36.19batch/s]

Training batches on cuda:  71%|███████   | 144/203 [00:03<00:01, 37.87batch/s]

Training batches on cuda:  73%|███████▎  | 149/203 [00:03<00:01, 39.56batch/s]

Training batches on cuda:  76%|███████▌  | 154/203 [00:03<00:01, 40.39batch/s]

Training batches on cuda:  78%|███████▊  | 159/203 [00:04<00:01, 40.93batch/s]

Training batches on cuda:  81%|████████  | 164/203 [00:04<00:00, 41.33batch/s]

Training batches on cuda:  83%|████████▎ | 169/203 [00:04<00:00, 41.60batch/s]

Training batches on cuda:  86%|████████▌ | 174/203 [00:04<00:00, 42.18batch/s]

Training batches on cuda:  88%|████████▊ | 179/203 [00:04<00:00, 42.21batch/s]

Training batches on cuda:  91%|█████████ | 184/203 [00:04<00:00, 40.77batch/s]

Training batches on cuda:  93%|█████████▎| 189/203 [00:04<00:00, 41.85batch/s]

Training batches on cuda:  96%|█████████▌| 194/203 [00:04<00:00, 41.67batch/s]

Training batches on cuda:  98%|█████████▊| 199/203 [00:05<00:00, 40.83batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:50<00:10,  5.47s/epoch, loss=0.000405, prev_loss=0.000417]
Training epochs on cuda:  90%|█████████ | 9/10 [00:50<00:05,  5.48s/epoch, loss=0.000405, prev_loss=0.000417]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/203 [00:00<00:10, 18.46batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:08, 21.98batch/s]

Training batches on cuda:   5%|▌         | 11/203 [00:00<00:07, 25.71batch/s]

Training batches on cuda:   8%|▊         | 16/203 [00:00<00:06, 29.35batch/s]

Training batches on cuda:  10%|▉         | 20/203 [00:00<00:05, 31.87batch/s]

Training batches on cuda:  12%|█▏        | 25/203 [00:00<00:05, 34.27batch/s]

Training batches on cuda:  15%|█▍        | 30/203 [00:00<00:04, 36.45batch/s]

Training batches on cuda:  17%|█▋        | 35/203 [00:00<00:04, 37.89batch/s]

Training batches on cuda:  20%|█▉        | 40/203 [00:01<00:04, 38.44batch/s]

Training batches on cuda:  22%|██▏       | 45/203 [00:01<00:04, 39.31batch/s]

Training batches on cuda:  25%|██▍       | 50/203 [00:01<00:03, 40.54batch/s]

Training batches on cuda:  27%|██▋       | 55/203 [00:01<00:03, 41.56batch/s]

Training batches on cuda:  30%|██▉       | 60/203 [00:01<00:03, 41.97batch/s]

Training batches on cuda:  32%|███▏      | 65/203 [00:01<00:03, 42.37batch/s]

Training batches on cuda:  34%|███▍      | 70/203 [00:01<00:03, 42.11batch/s]

Training batches on cuda:  37%|███▋      | 75/203 [00:01<00:03, 42.00batch/s]

Training batches on cuda:  39%|███▉      | 80/203 [00:01<00:02, 42.37batch/s]

Training batches on cuda:  42%|████▏     | 85/203 [00:02<00:02, 41.79batch/s]

Training batches on cuda:  44%|████▍     | 90/203 [00:02<00:02, 41.25batch/s]

Training batches on cuda:  47%|████▋     | 95/203 [00:02<00:02, 41.72batch/s]

Training batches on cuda:  49%|████▉     | 100/203 [00:02<00:02, 41.54batch/s]

Training batches on cuda:  52%|█████▏    | 105/203 [00:02<00:02, 40.99batch/s]

Training batches on cuda:  54%|█████▍    | 110/203 [00:02<00:02, 40.48batch/s]

Training batches on cuda:  57%|█████▋    | 115/203 [00:02<00:02, 39.86batch/s]

Training batches on cuda:  59%|█████▊    | 119/203 [00:02<00:02, 38.63batch/s]

Training batches on cuda:  61%|██████    | 123/203 [00:03<00:02, 38.84batch/s]

Training batches on cuda:  63%|██████▎   | 127/203 [00:03<00:02, 36.21batch/s]

Training batches on cuda:  65%|██████▌   | 132/203 [00:03<00:01, 37.41batch/s]

Training batches on cuda:  67%|██████▋   | 136/203 [00:03<00:01, 38.02batch/s]

Training batches on cuda:  69%|██████▉   | 140/203 [00:03<00:01, 38.36batch/s]

Training batches on cuda:  71%|███████   | 144/203 [00:03<00:01, 38.70batch/s]

Training batches on cuda:  73%|███████▎  | 148/203 [00:03<00:01, 38.72batch/s]

Training batches on cuda:  75%|███████▍  | 152/203 [00:03<00:01, 38.91batch/s]

Training batches on cuda:  77%|███████▋  | 156/203 [00:03<00:01, 38.71batch/s]

Training batches on cuda:  79%|███████▉  | 160/203 [00:04<00:01, 38.70batch/s]

Training batches on cuda:  81%|████████  | 164/203 [00:04<00:01, 38.59batch/s]

Training batches on cuda:  83%|████████▎ | 169/203 [00:04<00:00, 39.32batch/s]

Training batches on cuda:  85%|████████▌ | 173/203 [00:04<00:00, 39.45batch/s]

Training batches on cuda:  87%|████████▋ | 177/203 [00:04<00:00, 38.55batch/s]

Training batches on cuda:  89%|████████▉ | 181/203 [00:04<00:00, 38.57batch/s]

Training batches on cuda:  91%|█████████ | 185/203 [00:04<00:00, 38.24batch/s]

Training batches on cuda:  93%|█████████▎| 189/203 [00:04<00:00, 38.72batch/s]

Training batches on cuda:  95%|█████████▌| 193/203 [00:04<00:00, 37.46batch/s]

Training batches on cuda:  97%|█████████▋| 197/203 [00:04<00:00, 37.70batch/s]

Training batches on cuda:  99%|█████████▉| 201/203 [00:05<00:00, 38.00batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:55<00:05,  5.48s/epoch, loss=0.0004, prev_loss=0.000405]  2021-09-16:19:15:02,464 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:15:03,497 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:15:18,493 INFO     [evaluator.py:657] Evaluation took 15.00s seconds
2021-09-16:19:15:18,637 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00, 10.30s/epoch, loss=0.0004, prev_loss=0.000405]
Training epochs on cuda: 100%|██████████| 10/10 [01:11<00:00,  7.17s/epoch, loss=0.0004, prev_loss=0.000405]
2021-09-16:19:15:18,638 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpbn9z831o'
2021-09-16:19:15:18,689 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbn9z831o' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:19, 2.01ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:17, 2.16ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:16, 2.29ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.38ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.46ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.50ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.55ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.62ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:03, 2.51ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:12<00:02, 2.55ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.57ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.59ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.61ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.62ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:15:33,605 INFO     [evaluator.py:657] Evaluation took 14.91s seconds
[I 2021-09-16 19:15:33,611] Trial 73 finished with value: 0.9466842593442085 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.016853991586199664, 'optimizer.lr': 0.04835243663396181, 'negative_sampler.num_negs_per_pos': 40, 'training.batch_size': 1532}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:15:33,635 WARNING  [api.py:823] No random seed is specified. Setting to 1830872200.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/253 [00:00<00:09, 26.88batch/s]

Training batches on cuda:   3%|▎         | 8/253 [00:00<00:08, 30.62batch/s]

Training batches on cuda:   5%|▌         | 13/253 [00:00<00:07, 34.03batch/s]

Training batches on cuda:   7%|▋         | 18/253 [00:00<00:06, 37.20batch/s]

Training batches on cuda:   9%|▉         | 23/253 [00:00<00:05, 40.12batch/s]

Training batches on cuda:  11%|█         | 28/253 [00:00<00:05, 42.21batch/s]

Training batches on cuda:  13%|█▎        | 34/253 [00:00<00:04, 44.41batch/s]

Training batches on cuda:  15%|█▌        | 39/253 [00:00<00:04, 45.59batch/s]

Training batches on cuda:  18%|█▊        | 45/253 [00:00<00:04, 47.07batch/s]

Training batches on cuda:  20%|█▉        | 50/253 [00:01<00:04, 47.79batch/s]

Training batches on cuda:  22%|██▏       | 56/253 [00:01<00:04, 48.65batch/s]

Training batches on cuda:  25%|██▍       | 62/253 [00:01<00:03, 49.45batch/s]

Training batches on cuda:  27%|██▋       | 68/253 [00:01<00:03, 49.89batch/s]

Training batches on cuda:  29%|██▉       | 74/253 [00:01<00:03, 50.01batch/s]

Training batches on cuda:  32%|███▏      | 80/253 [00:01<00:03, 49.93batch/s]

Training batches on cuda:  34%|███▍      | 86/253 [00:01<00:03, 50.27batch/s]

Training batches on cuda:  36%|███▋      | 92/253 [00:01<00:03, 50.21batch/s]

Training batches on cuda:  39%|███▊      | 98/253 [00:02<00:03, 50.43batch/s]

Training batches on cuda:  41%|████      | 104/253 [00:02<00:02, 50.69batch/s]

Training batches on cuda:  43%|████▎     | 110/253 [00:02<00:02, 50.59batch/s]

Training batches on cuda:  46%|████▌     | 116/253 [00:02<00:02, 50.57batch/s]

Training batches on cuda:  48%|████▊     | 122/253 [00:02<00:02, 50.96batch/s]

Training batches on cuda:  51%|█████     | 128/253 [00:02<00:02, 50.95batch/s]

Training batches on cuda:  53%|█████▎    | 134/253 [00:02<00:02, 50.92batch/s]

Training batches on cuda:  55%|█████▌    | 140/253 [00:02<00:02, 51.13batch/s]

Training batches on cuda:  58%|█████▊    | 146/253 [00:02<00:02, 51.08batch/s]

Training batches on cuda:  60%|██████    | 152/253 [00:03<00:01, 51.10batch/s]

Training batches on cuda:  62%|██████▏   | 158/253 [00:03<00:01, 50.44batch/s]

Training batches on cuda:  65%|██████▍   | 164/253 [00:03<00:01, 50.24batch/s]

Training batches on cuda:  67%|██████▋   | 170/253 [00:03<00:01, 50.05batch/s]

Training batches on cuda:  70%|██████▉   | 176/253 [00:03<00:01, 50.26batch/s]

Training batches on cuda:  72%|███████▏  | 182/253 [00:03<00:01, 49.25batch/s]

Training batches on cuda:  74%|███████▍  | 188/253 [00:03<00:01, 49.58batch/s]

Training batches on cuda:  77%|███████▋  | 194/253 [00:03<00:01, 49.77batch/s]

Training batches on cuda:  79%|███████▉  | 200/253 [00:04<00:01, 49.78batch/s]

Training batches on cuda:  81%|████████  | 205/253 [00:04<00:00, 49.76batch/s]

Training batches on cuda:  83%|████████▎ | 211/253 [00:04<00:00, 49.95batch/s]

Training batches on cuda:  86%|████████▌ | 217/253 [00:04<00:00, 50.33batch/s]

Training batches on cuda:  88%|████████▊ | 223/253 [00:04<00:00, 49.44batch/s]

Training batches on cuda:  91%|█████████ | 229/253 [00:04<00:00, 49.96batch/s]

Training batches on cuda:  93%|█████████▎| 235/253 [00:04<00:00, 50.26batch/s]

Training batches on cuda:  95%|█████████▌| 241/253 [00:04<00:00, 50.54batch/s]

Training batches on cuda:  98%|█████████▊| 247/253 [00:04<00:00, 50.60batch/s]

Training batches on cuda: 100%|██████████| 253/253 [00:05<00:00, 51.34batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.00108, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:48,  5.37s/epoch, loss=0.00108, prev_loss=nan]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/253 [00:00<00:09, 25.38batch/s]

Training batches on cuda:   4%|▎         | 9/253 [00:00<00:08, 30.28batch/s]

Training batches on cuda:   6%|▌         | 15/253 [00:00<00:06, 34.80batch/s]

Training batches on cuda:   8%|▊         | 21/253 [00:00<00:05, 39.25batch/s]

Training batches on cuda:  11%|█         | 27/253 [00:00<00:05, 43.75batch/s]

Training batches on cuda:  13%|█▎        | 34/253 [00:00<00:04, 47.83batch/s]

Training batches on cuda:  16%|█▌        | 40/253 [00:00<00:04, 50.78batch/s]

Training batches on cuda:  18%|█▊        | 46/253 [00:00<00:04, 51.33batch/s]

Training batches on cuda:  21%|██        | 52/253 [00:00<00:03, 52.32batch/s]

Training batches on cuda:  23%|██▎       | 58/253 [00:01<00:03, 51.85batch/s]

Training batches on cuda:  25%|██▌       | 64/253 [00:01<00:03, 50.39batch/s]

Training batches on cuda:  28%|██▊       | 70/253 [00:01<00:03, 49.34batch/s]

Training batches on cuda:  30%|███       | 76/253 [00:01<00:03, 50.11batch/s]

Training batches on cuda:  32%|███▏      | 82/253 [00:01<00:03, 50.86batch/s]

Training batches on cuda:  35%|███▍      | 88/253 [00:01<00:03, 51.92batch/s]

Training batches on cuda:  37%|███▋      | 94/253 [00:01<00:03, 52.75batch/s]

Training batches on cuda:  40%|███▉      | 100/253 [00:01<00:02, 53.64batch/s]

Training batches on cuda:  42%|████▏     | 106/253 [00:02<00:02, 54.07batch/s]

Training batches on cuda:  44%|████▍     | 112/253 [00:02<00:02, 55.15batch/s]

Training batches on cuda:  47%|████▋     | 118/253 [00:02<00:02, 54.19batch/s]

Training batches on cuda:  49%|████▉     | 124/253 [00:02<00:02, 55.55batch/s]

Training batches on cuda:  51%|█████▏    | 130/253 [00:02<00:02, 54.23batch/s]

Training batches on cuda:  54%|█████▍    | 136/253 [00:02<00:02, 55.52batch/s]

Training batches on cuda:  56%|█████▌    | 142/253 [00:02<00:02, 54.01batch/s]

Training batches on cuda:  58%|█████▊    | 148/253 [00:02<00:01, 53.96batch/s]

Training batches on cuda:  61%|██████    | 154/253 [00:02<00:01, 54.27batch/s]

Training batches on cuda:  63%|██████▎   | 160/253 [00:03<00:01, 54.01batch/s]

Training batches on cuda:  66%|██████▌   | 166/253 [00:03<00:01, 53.74batch/s]

Training batches on cuda:  68%|██████▊   | 172/253 [00:03<00:01, 54.08batch/s]

Training batches on cuda:  70%|███████   | 178/253 [00:03<00:01, 53.23batch/s]

Training batches on cuda:  73%|███████▎  | 184/253 [00:03<00:01, 53.09batch/s]

Training batches on cuda:  75%|███████▌  | 190/253 [00:03<00:01, 53.08batch/s]

Training batches on cuda:  77%|███████▋  | 196/253 [00:03<00:01, 53.34batch/s]

Training batches on cuda:  80%|███████▉  | 202/253 [00:03<00:00, 53.68batch/s]

Training batches on cuda:  82%|████████▏ | 208/253 [00:03<00:00, 52.78batch/s]

Training batches on cuda:  85%|████████▍ | 214/253 [00:04<00:00, 53.10batch/s]

Training batches on cuda:  87%|████████▋ | 220/253 [00:04<00:00, 53.61batch/s]

Training batches on cuda:  89%|████████▉ | 226/253 [00:04<00:00, 52.79batch/s]

Training batches on cuda:  92%|█████████▏| 232/253 [00:04<00:00, 52.17batch/s]

Training batches on cuda:  94%|█████████▍| 238/253 [00:04<00:00, 52.81batch/s]

Training batches on cuda:  96%|█████████▋| 244/253 [00:04<00:00, 54.18batch/s]

Training batches on cuda:  99%|█████████▉| 250/253 [00:04<00:00, 55.25batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:10<00:48,  5.37s/epoch, loss=0.00107, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:10<00:41,  5.24s/epoch, loss=0.00107, prev_loss=nan]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/253 [00:00<00:06, 38.80batch/s]

Training batches on cuda:   4%|▍         | 10/253 [00:00<00:05, 42.45batch/s]

Training batches on cuda:   6%|▋         | 16/253 [00:00<00:05, 45.04batch/s]

Training batches on cuda:   9%|▊         | 22/253 [00:00<00:04, 48.43batch/s]

Training batches on cuda:  11%|█         | 28/253 [00:00<00:04, 50.79batch/s]

Training batches on cuda:  13%|█▎        | 34/253 [00:00<00:04, 52.50batch/s]

Training batches on cuda:  16%|█▌        | 40/253 [00:00<00:03, 53.48batch/s]

Training batches on cuda:  18%|█▊        | 46/253 [00:00<00:03, 54.55batch/s]

Training batches on cuda:  21%|██        | 52/253 [00:00<00:03, 50.70batch/s]

Training batches on cuda:  23%|██▎       | 58/253 [00:01<00:03, 52.29batch/s]

Training batches on cuda:  25%|██▌       | 64/253 [00:01<00:03, 52.96batch/s]

Training batches on cuda:  28%|██▊       | 70/253 [00:01<00:03, 53.39batch/s]

Training batches on cuda:  30%|███       | 76/253 [00:01<00:03, 54.06batch/s]

Training batches on cuda:  32%|███▏      | 82/253 [00:01<00:03, 54.25batch/s]

Training batches on cuda:  35%|███▍      | 88/253 [00:01<00:03, 54.37batch/s]

Training batches on cuda:  37%|███▋      | 94/253 [00:01<00:02, 54.53batch/s]

Training batches on cuda:  40%|███▉      | 100/253 [00:01<00:02, 53.94batch/s]

Training batches on cuda:  42%|████▏     | 106/253 [00:01<00:02, 54.22batch/s]

Training batches on cuda:  44%|████▍     | 112/253 [00:02<00:02, 54.70batch/s]

Training batches on cuda:  47%|████▋     | 118/253 [00:02<00:02, 54.71batch/s]

Training batches on cuda:  49%|████▉     | 124/253 [00:02<00:02, 54.24batch/s]

Training batches on cuda:  51%|█████▏    | 130/253 [00:02<00:02, 54.65batch/s]

Training batches on cuda:  54%|█████▍    | 136/253 [00:02<00:02, 54.72batch/s]

Training batches on cuda:  56%|█████▌    | 142/253 [00:02<00:02, 54.61batch/s]

Training batches on cuda:  58%|█████▊    | 148/253 [00:02<00:01, 54.44batch/s]

Training batches on cuda:  61%|██████    | 154/253 [00:02<00:01, 53.57batch/s]

Training batches on cuda:  63%|██████▎   | 160/253 [00:02<00:01, 55.28batch/s]

Training batches on cuda:  66%|██████▌   | 166/253 [00:03<00:01, 55.12batch/s]

Training batches on cuda:  68%|██████▊   | 172/253 [00:03<00:01, 55.75batch/s]

Training batches on cuda:  70%|███████   | 178/253 [00:03<00:01, 55.59batch/s]

Training batches on cuda:  73%|███████▎  | 184/253 [00:03<00:01, 55.20batch/s]

Training batches on cuda:  75%|███████▌  | 190/253 [00:03<00:01, 54.60batch/s]

Training batches on cuda:  77%|███████▋  | 196/253 [00:03<00:01, 54.48batch/s]

Training batches on cuda:  80%|███████▉  | 202/253 [00:03<00:00, 54.16batch/s]

Training batches on cuda:  82%|████████▏ | 208/253 [00:03<00:00, 53.78batch/s]

Training batches on cuda:  85%|████████▍ | 214/253 [00:03<00:00, 53.97batch/s]

Training batches on cuda:  87%|████████▋ | 220/253 [00:04<00:00, 54.60batch/s]

Training batches on cuda:  89%|████████▉ | 226/253 [00:04<00:00, 54.42batch/s]

Training batches on cuda:  92%|█████████▏| 232/253 [00:04<00:00, 54.21batch/s]

Training batches on cuda:  94%|█████████▍| 238/253 [00:04<00:00, 53.99batch/s]

Training batches on cuda:  96%|█████████▋| 244/253 [00:04<00:00, 54.43batch/s]

Training batches on cuda:  99%|█████████▉| 250/253 [00:04<00:00, 54.47batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:15<00:41,  5.24s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:35,  5.13s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/253 [00:00<00:07, 34.44batch/s]

Training batches on cuda:   4%|▍         | 10/253 [00:00<00:06, 38.42batch/s]

Training batches on cuda:   6%|▋         | 16/253 [00:00<00:05, 41.34batch/s]

Training batches on cuda:   9%|▊         | 22/253 [00:00<00:05, 44.03batch/s]

Training batches on cuda:  11%|█         | 28/253 [00:00<00:04, 47.00batch/s]

Training batches on cuda:  13%|█▎        | 34/253 [00:00<00:04, 49.37batch/s]

Training batches on cuda:  15%|█▌        | 39/253 [00:00<00:04, 49.50batch/s]

Training batches on cuda:  17%|█▋        | 44/253 [00:00<00:04, 49.48batch/s]

Training batches on cuda:  20%|█▉        | 50/253 [00:01<00:03, 50.78batch/s]

Training batches on cuda:  22%|██▏       | 55/253 [00:01<00:04, 46.69batch/s]

Training batches on cuda:  24%|██▍       | 61/253 [00:01<00:03, 48.11batch/s]

Training batches on cuda:  26%|██▋       | 67/253 [00:01<00:03, 49.99batch/s]

Training batches on cuda:  29%|██▉       | 73/253 [00:01<00:03, 51.04batch/s]

Training batches on cuda:  31%|███       | 79/253 [00:01<00:03, 50.55batch/s]

Training batches on cuda:  34%|███▎      | 85/253 [00:01<00:03, 50.74batch/s]

Training batches on cuda:  36%|███▌      | 91/253 [00:01<00:03, 50.26batch/s]

Training batches on cuda:  38%|███▊      | 97/253 [00:01<00:03, 50.30batch/s]

Training batches on cuda:  41%|████      | 103/253 [00:02<00:02, 51.16batch/s]

Training batches on cuda:  43%|████▎     | 109/253 [00:02<00:02, 51.35batch/s]

Training batches on cuda:  45%|████▌     | 115/253 [00:02<00:02, 51.32batch/s]

Training batches on cuda:  48%|████▊     | 121/253 [00:02<00:02, 51.35batch/s]

Training batches on cuda:  50%|█████     | 127/253 [00:02<00:02, 50.01batch/s]

Training batches on cuda:  53%|█████▎    | 133/253 [00:02<00:02, 50.57batch/s]

Training batches on cuda:  55%|█████▍    | 139/253 [00:02<00:02, 51.09batch/s]

Training batches on cuda:  57%|█████▋    | 145/253 [00:02<00:02, 51.26batch/s]

Training batches on cuda:  60%|█████▉    | 151/253 [00:02<00:01, 51.77batch/s]

Training batches on cuda:  62%|██████▏   | 157/253 [00:03<00:01, 52.62batch/s]

Training batches on cuda:  64%|██████▍   | 163/253 [00:03<00:01, 52.81batch/s]

Training batches on cuda:  67%|██████▋   | 169/253 [00:03<00:01, 53.00batch/s]

Training batches on cuda:  69%|██████▉   | 175/253 [00:03<00:01, 52.77batch/s]

Training batches on cuda:  72%|███████▏  | 181/253 [00:03<00:01, 53.35batch/s]

Training batches on cuda:  74%|███████▍  | 187/253 [00:03<00:01, 52.98batch/s]

Training batches on cuda:  76%|███████▋  | 193/253 [00:03<00:01, 52.41batch/s]

Training batches on cuda:  79%|███████▊  | 199/253 [00:03<00:01, 52.78batch/s]

Training batches on cuda:  81%|████████  | 205/253 [00:03<00:00, 53.40batch/s]

Training batches on cuda:  83%|████████▎ | 211/253 [00:04<00:00, 53.94batch/s]

Training batches on cuda:  86%|████████▌ | 217/253 [00:04<00:00, 53.44batch/s]

Training batches on cuda:  88%|████████▊ | 223/253 [00:04<00:00, 53.50batch/s]

Training batches on cuda:  91%|█████████ | 229/253 [00:04<00:00, 52.52batch/s]

Training batches on cuda:  93%|█████████▎| 235/253 [00:04<00:00, 52.15batch/s]

Training batches on cuda:  95%|█████████▌| 241/253 [00:04<00:00, 52.72batch/s]

Training batches on cuda:  98%|█████████▊| 247/253 [00:04<00:00, 52.92batch/s]

Training batches on cuda: 100%|██████████| 253/253 [00:04<00:00, 53.61batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:20<00:35,  5.13s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:30,  5.12s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/253 [00:00<00:06, 35.62batch/s]

Training batches on cuda:   4%|▍         | 10/253 [00:00<00:06, 39.59batch/s]

Training batches on cuda:   6%|▌         | 15/253 [00:00<00:05, 42.03batch/s]

Training batches on cuda:   8%|▊         | 21/253 [00:00<00:05, 45.25batch/s]

Training batches on cuda:  11%|█         | 27/253 [00:00<00:04, 47.38batch/s]

Training batches on cuda:  13%|█▎        | 33/253 [00:00<00:04, 48.80batch/s]

Training batches on cuda:  15%|█▌        | 38/253 [00:00<00:04, 48.48batch/s]

Training batches on cuda:  17%|█▋        | 44/253 [00:00<00:04, 49.46batch/s]

Training batches on cuda:  20%|█▉        | 50/253 [00:01<00:04, 50.50batch/s]

Training batches on cuda:  22%|██▏       | 55/253 [00:01<00:04, 47.22batch/s]

Training batches on cuda:  24%|██▍       | 61/253 [00:01<00:03, 48.94batch/s]

Training batches on cuda:  26%|██▋       | 67/253 [00:01<00:03, 50.42batch/s]

Training batches on cuda:  29%|██▉       | 73/253 [00:01<00:03, 51.62batch/s]

Training batches on cuda:  31%|███       | 79/253 [00:01<00:03, 52.34batch/s]

Training batches on cuda:  34%|███▎      | 85/253 [00:01<00:03, 53.31batch/s]

Training batches on cuda:  36%|███▌      | 91/253 [00:01<00:03, 53.76batch/s]

Training batches on cuda:  38%|███▊      | 97/253 [00:01<00:02, 54.21batch/s]

Training batches on cuda:  41%|████      | 103/253 [00:01<00:02, 54.48batch/s]

Training batches on cuda:  43%|████▎     | 109/253 [00:02<00:02, 54.80batch/s]

Training batches on cuda:  45%|████▌     | 115/253 [00:02<00:02, 54.71batch/s]

Training batches on cuda:  48%|████▊     | 121/253 [00:02<00:02, 53.99batch/s]

Training batches on cuda:  50%|█████     | 127/253 [00:02<00:02, 53.71batch/s]

Training batches on cuda:  53%|█████▎    | 133/253 [00:02<00:02, 53.51batch/s]

Training batches on cuda:  55%|█████▍    | 139/253 [00:02<00:02, 53.24batch/s]

Training batches on cuda:  57%|█████▋    | 145/253 [00:02<00:02, 52.12batch/s]

Training batches on cuda:  60%|█████▉    | 151/253 [00:02<00:01, 51.74batch/s]

Training batches on cuda:  62%|██████▏   | 157/253 [00:03<00:01, 52.02batch/s]

Training batches on cuda:  64%|██████▍   | 163/253 [00:03<00:01, 52.51batch/s]

Training batches on cuda:  67%|██████▋   | 169/253 [00:03<00:01, 52.16batch/s]

Training batches on cuda:  69%|██████▉   | 175/253 [00:03<00:01, 51.34batch/s]

Training batches on cuda:  72%|███████▏  | 181/253 [00:03<00:01, 50.95batch/s]

Training batches on cuda:  74%|███████▍  | 187/253 [00:03<00:01, 50.08batch/s]

Training batches on cuda:  76%|███████▋  | 193/253 [00:03<00:01, 51.43batch/s]

Training batches on cuda:  79%|███████▊  | 199/253 [00:03<00:01, 50.33batch/s]

Training batches on cuda:  81%|████████  | 205/253 [00:03<00:00, 51.24batch/s]

Training batches on cuda:  83%|████████▎ | 211/253 [00:04<00:00, 51.87batch/s]

Training batches on cuda:  86%|████████▌ | 217/253 [00:04<00:00, 52.33batch/s]

Training batches on cuda:  88%|████████▊ | 223/253 [00:04<00:00, 52.19batch/s]

Training batches on cuda:  91%|█████████ | 229/253 [00:04<00:00, 52.86batch/s]

Training batches on cuda:  93%|█████████▎| 235/253 [00:04<00:00, 51.66batch/s]

Training batches on cuda:  95%|█████████▌| 241/253 [00:04<00:00, 51.15batch/s]

Training batches on cuda:  98%|█████████▊| 247/253 [00:04<00:00, 50.97batch/s]

Training batches on cuda: 100%|██████████| 253/253 [00:04<00:00, 52.57batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:25<00:30,  5.12s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  50%|█████     | 5/10 [00:25<00:25,  5.14s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/253 [00:00<00:08, 28.79batch/s]

Training batches on cuda:   4%|▎         | 9/253 [00:00<00:07, 33.38batch/s]

Training batches on cuda:   6%|▌         | 15/253 [00:00<00:06, 37.23batch/s]

Training batches on cuda:   8%|▊         | 21/253 [00:00<00:05, 41.12batch/s]

Training batches on cuda:  11%|█         | 27/253 [00:00<00:05, 44.26batch/s]

Training batches on cuda:  13%|█▎        | 33/253 [00:00<00:04, 47.08batch/s]

Training batches on cuda:  15%|█▌        | 39/253 [00:00<00:04, 49.02batch/s]

Training batches on cuda:  18%|█▊        | 45/253 [00:00<00:04, 50.53batch/s]

Training batches on cuda:  20%|██        | 51/253 [00:01<00:04, 50.43batch/s]

Training batches on cuda:  23%|██▎       | 57/253 [00:01<00:03, 51.53batch/s]

Training batches on cuda:  25%|██▍       | 63/253 [00:01<00:03, 52.05batch/s]

Training batches on cuda:  27%|██▋       | 69/253 [00:01<00:03, 53.04batch/s]

Training batches on cuda:  30%|██▉       | 75/253 [00:01<00:03, 53.78batch/s]

Training batches on cuda:  32%|███▏      | 81/253 [00:01<00:03, 53.89batch/s]

Training batches on cuda:  34%|███▍      | 87/253 [00:01<00:03, 51.56batch/s]

Training batches on cuda:  37%|███▋      | 93/253 [00:01<00:03, 52.40batch/s]

Training batches on cuda:  39%|███▉      | 99/253 [00:01<00:02, 52.30batch/s]

Training batches on cuda:  42%|████▏     | 105/253 [00:02<00:02, 52.96batch/s]

Training batches on cuda:  44%|████▍     | 111/253 [00:02<00:02, 53.12batch/s]

Training batches on cuda:  46%|████▌     | 117/253 [00:02<00:02, 53.51batch/s]

Training batches on cuda:  49%|████▊     | 123/253 [00:02<00:02, 53.53batch/s]

Training batches on cuda:  51%|█████     | 129/253 [00:02<00:02, 53.61batch/s]

Training batches on cuda:  53%|█████▎    | 135/253 [00:02<00:02, 53.89batch/s]

Training batches on cuda:  56%|█████▌    | 141/253 [00:02<00:02, 53.86batch/s]

Training batches on cuda:  58%|█████▊    | 147/253 [00:02<00:01, 53.75batch/s]

Training batches on cuda:  60%|██████    | 153/253 [00:02<00:01, 53.75batch/s]

Training batches on cuda:  63%|██████▎   | 159/253 [00:03<00:01, 53.69batch/s]

Training batches on cuda:  65%|██████▌   | 165/253 [00:03<00:01, 52.54batch/s]

Training batches on cuda:  68%|██████▊   | 171/253 [00:03<00:01, 51.64batch/s]

Training batches on cuda:  70%|██████▉   | 177/253 [00:03<00:01, 51.65batch/s]

Training batches on cuda:  72%|███████▏  | 183/253 [00:03<00:01, 52.36batch/s]

Training batches on cuda:  75%|███████▍  | 189/253 [00:03<00:01, 53.11batch/s]

Training batches on cuda:  77%|███████▋  | 195/253 [00:03<00:01, 53.46batch/s]

Training batches on cuda:  79%|███████▉  | 201/253 [00:03<00:00, 52.63batch/s]

Training batches on cuda:  82%|████████▏ | 207/253 [00:03<00:00, 53.11batch/s]

Training batches on cuda:  84%|████████▍ | 213/253 [00:04<00:00, 53.02batch/s]

Training batches on cuda:  87%|████████▋ | 219/253 [00:04<00:00, 53.18batch/s]

Training batches on cuda:  89%|████████▉ | 225/253 [00:04<00:00, 52.67batch/s]

Training batches on cuda:  91%|█████████▏| 231/253 [00:04<00:00, 52.86batch/s]

Training batches on cuda:  94%|█████████▎| 237/253 [00:04<00:00, 53.75batch/s]

Training batches on cuda:  96%|█████████▌| 243/253 [00:04<00:00, 53.80batch/s]

Training batches on cuda:  98%|█████████▊| 249/253 [00:04<00:00, 53.74batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:25,  5.14s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:20,  5.09s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/253 [00:00<00:09, 26.43batch/s]

Training batches on cuda:   4%|▎         | 9/253 [00:00<00:07, 31.21batch/s]

Training batches on cuda:   6%|▌         | 15/253 [00:00<00:06, 35.52batch/s]

Training batches on cuda:   8%|▊         | 21/253 [00:00<00:05, 39.40batch/s]

Training batches on cuda:  11%|█         | 27/253 [00:00<00:05, 42.74batch/s]

Training batches on cuda:  13%|█▎        | 33/253 [00:00<00:04, 45.56batch/s]

Training batches on cuda:  15%|█▌        | 39/253 [00:00<00:04, 47.66batch/s]

Training batches on cuda:  18%|█▊        | 45/253 [00:00<00:04, 49.11batch/s]

Training batches on cuda:  20%|█▉        | 50/253 [00:01<00:04, 45.39batch/s]

Training batches on cuda:  22%|██▏       | 56/253 [00:01<00:04, 47.52batch/s]

Training batches on cuda:  25%|██▍       | 62/253 [00:01<00:03, 48.39batch/s]

Training batches on cuda:  27%|██▋       | 68/253 [00:01<00:03, 49.28batch/s]

Training batches on cuda:  29%|██▉       | 74/253 [00:01<00:03, 49.07batch/s]

Training batches on cuda:  32%|███▏      | 80/253 [00:01<00:03, 49.77batch/s]

Training batches on cuda:  34%|███▍      | 86/253 [00:01<00:03, 49.77batch/s]

Training batches on cuda:  36%|███▌      | 91/253 [00:01<00:03, 48.57batch/s]

Training batches on cuda:  38%|███▊      | 97/253 [00:01<00:03, 49.61batch/s]

Training batches on cuda:  41%|████      | 103/253 [00:02<00:02, 50.62batch/s]

Training batches on cuda:  43%|████▎     | 109/253 [00:02<00:02, 51.21batch/s]

Training batches on cuda:  45%|████▌     | 115/253 [00:02<00:02, 51.89batch/s]

Training batches on cuda:  48%|████▊     | 121/253 [00:02<00:02, 51.85batch/s]

Training batches on cuda:  50%|█████     | 127/253 [00:02<00:02, 51.69batch/s]

Training batches on cuda:  53%|█████▎    | 133/253 [00:02<00:02, 51.44batch/s]

Training batches on cuda:  55%|█████▍    | 139/253 [00:02<00:02, 50.29batch/s]

Training batches on cuda:  57%|█████▋    | 145/253 [00:02<00:02, 51.42batch/s]

Training batches on cuda:  60%|█████▉    | 151/253 [00:03<00:01, 51.69batch/s]

Training batches on cuda:  62%|██████▏   | 157/253 [00:03<00:01, 51.17batch/s]

Training batches on cuda:  64%|██████▍   | 163/253 [00:03<00:01, 52.01batch/s]

Training batches on cuda:  67%|██████▋   | 169/253 [00:03<00:01, 52.37batch/s]

Training batches on cuda:  69%|██████▉   | 175/253 [00:03<00:01, 53.11batch/s]

Training batches on cuda:  72%|███████▏  | 181/253 [00:03<00:01, 53.94batch/s]

Training batches on cuda:  74%|███████▍  | 187/253 [00:03<00:01, 54.50batch/s]

Training batches on cuda:  76%|███████▋  | 193/253 [00:03<00:01, 54.40batch/s]

Training batches on cuda:  79%|███████▊  | 199/253 [00:03<00:00, 54.51batch/s]

Training batches on cuda:  81%|████████  | 205/253 [00:04<00:00, 54.67batch/s]

Training batches on cuda:  83%|████████▎ | 211/253 [00:04<00:00, 54.32batch/s]

Training batches on cuda:  86%|████████▌ | 217/253 [00:04<00:00, 54.58batch/s]

Training batches on cuda:  88%|████████▊ | 223/253 [00:04<00:00, 53.99batch/s]

Training batches on cuda:  91%|█████████ | 229/253 [00:04<00:00, 53.52batch/s]

Training batches on cuda:  93%|█████████▎| 235/253 [00:04<00:00, 53.76batch/s]

Training batches on cuda:  95%|█████████▌| 241/253 [00:04<00:00, 53.23batch/s]

Training batches on cuda:  98%|█████████▊| 247/253 [00:04<00:00, 53.69batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:35<00:20,  5.09s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:15,  5.11s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/253 [00:00<00:07, 34.06batch/s]

Training batches on cuda:   4%|▍         | 10/253 [00:00<00:06, 38.29batch/s]

Training batches on cuda:   6%|▋         | 16/253 [00:00<00:05, 41.41batch/s]

Training batches on cuda:   9%|▊         | 22/253 [00:00<00:05, 44.37batch/s]

Training batches on cuda:  11%|█         | 28/253 [00:00<00:04, 46.30batch/s]

Training batches on cuda:  13%|█▎        | 34/253 [00:00<00:04, 47.85batch/s]

Training batches on cuda:  16%|█▌        | 40/253 [00:00<00:04, 49.55batch/s]

Training batches on cuda:  18%|█▊        | 45/253 [00:00<00:04, 45.67batch/s]

Training batches on cuda:  20%|█▉        | 50/253 [00:01<00:04, 46.40batch/s]

Training batches on cuda:  22%|██▏       | 56/253 [00:01<00:04, 48.65batch/s]

Training batches on cuda:  25%|██▍       | 62/253 [00:01<00:03, 50.04batch/s]

Training batches on cuda:  27%|██▋       | 68/253 [00:01<00:03, 50.90batch/s]

Training batches on cuda:  29%|██▉       | 74/253 [00:01<00:03, 50.19batch/s]

Training batches on cuda:  32%|███▏      | 80/253 [00:01<00:03, 48.43batch/s]

Training batches on cuda:  34%|███▍      | 86/253 [00:01<00:03, 49.06batch/s]

Training batches on cuda:  36%|███▋      | 92/253 [00:01<00:03, 50.63batch/s]

Training batches on cuda:  39%|███▊      | 98/253 [00:01<00:03, 51.58batch/s]

Training batches on cuda:  41%|████      | 104/253 [00:02<00:02, 51.29batch/s]

Training batches on cuda:  43%|████▎     | 110/253 [00:02<00:02, 51.19batch/s]

Training batches on cuda:  46%|████▌     | 116/253 [00:02<00:02, 50.94batch/s]

Training batches on cuda:  48%|████▊     | 122/253 [00:02<00:02, 51.04batch/s]

Training batches on cuda:  51%|█████     | 128/253 [00:02<00:02, 50.95batch/s]

Training batches on cuda:  53%|█████▎    | 134/253 [00:02<00:02, 50.96batch/s]

Training batches on cuda:  55%|█████▌    | 140/253 [00:02<00:02, 51.05batch/s]

Training batches on cuda:  58%|█████▊    | 146/253 [00:02<00:02, 50.81batch/s]

Training batches on cuda:  60%|██████    | 152/253 [00:03<00:01, 50.66batch/s]

Training batches on cuda:  62%|██████▏   | 158/253 [00:03<00:01, 50.13batch/s]

Training batches on cuda:  65%|██████▍   | 164/253 [00:03<00:01, 50.39batch/s]

Training batches on cuda:  67%|██████▋   | 170/253 [00:03<00:01, 50.25batch/s]

Training batches on cuda:  70%|██████▉   | 176/253 [00:03<00:01, 50.38batch/s]

Training batches on cuda:  72%|███████▏  | 182/253 [00:03<00:01, 49.68batch/s]

Training batches on cuda:  74%|███████▍  | 187/253 [00:03<00:01, 49.48batch/s]

Training batches on cuda:  76%|███████▋  | 193/253 [00:03<00:01, 49.71batch/s]

Training batches on cuda:  79%|███████▊  | 199/253 [00:03<00:01, 50.25batch/s]

Training batches on cuda:  81%|████████  | 205/253 [00:04<00:00, 51.37batch/s]

Training batches on cuda:  83%|████████▎ | 211/253 [00:04<00:00, 52.13batch/s]

Training batches on cuda:  86%|████████▌ | 217/253 [00:04<00:00, 51.95batch/s]

Training batches on cuda:  88%|████████▊ | 223/253 [00:04<00:00, 50.47batch/s]

Training batches on cuda:  91%|█████████ | 229/253 [00:04<00:00, 50.68batch/s]

Training batches on cuda:  93%|█████████▎| 235/253 [00:04<00:00, 52.59batch/s]

Training batches on cuda:  95%|█████████▌| 241/253 [00:04<00:00, 54.53batch/s]

Training batches on cuda:  98%|█████████▊| 247/253 [00:04<00:00, 55.41batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:40<00:15,  5.11s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  80%|████████  | 8/10 [00:40<00:10,  5.12s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/253 [00:00<00:09, 27.55batch/s]

Training batches on cuda:   4%|▍         | 10/253 [00:00<00:07, 32.95batch/s]

Training batches on cuda:   6%|▋         | 16/253 [00:00<00:06, 37.50batch/s]

Training batches on cuda:   9%|▊         | 22/253 [00:00<00:05, 41.07batch/s]

Training batches on cuda:  11%|█         | 28/253 [00:00<00:05, 44.30batch/s]

Training batches on cuda:  13%|█▎        | 34/253 [00:00<00:04, 47.13batch/s]

Training batches on cuda:  15%|█▌        | 39/253 [00:00<00:04, 46.40batch/s]

Training batches on cuda:  18%|█▊        | 45/253 [00:00<00:04, 48.70batch/s]

Training batches on cuda:  20%|██        | 51/253 [00:00<00:04, 50.36batch/s]

Training batches on cuda:  23%|██▎       | 57/253 [00:01<00:03, 51.90batch/s]

Training batches on cuda:  25%|██▍       | 63/253 [00:01<00:03, 52.32batch/s]

Training batches on cuda:  27%|██▋       | 69/253 [00:01<00:03, 50.06batch/s]

Training batches on cuda:  30%|██▉       | 75/253 [00:01<00:03, 50.24batch/s]

Training batches on cuda:  32%|███▏      | 81/253 [00:01<00:03, 51.29batch/s]

Training batches on cuda:  34%|███▍      | 87/253 [00:01<00:03, 51.66batch/s]

Training batches on cuda:  37%|███▋      | 93/253 [00:01<00:03, 52.38batch/s]

Training batches on cuda:  39%|███▉      | 99/253 [00:01<00:02, 52.37batch/s]

Training batches on cuda:  42%|████▏     | 105/253 [00:02<00:02, 52.52batch/s]

Training batches on cuda:  44%|████▍     | 111/253 [00:02<00:02, 52.95batch/s]

Training batches on cuda:  46%|████▌     | 117/253 [00:02<00:02, 51.39batch/s]

Training batches on cuda:  49%|████▊     | 123/253 [00:02<00:02, 52.31batch/s]

Training batches on cuda:  51%|█████     | 129/253 [00:02<00:02, 52.72batch/s]

Training batches on cuda:  53%|█████▎    | 135/253 [00:02<00:02, 53.39batch/s]

Training batches on cuda:  56%|█████▌    | 141/253 [00:02<00:02, 52.80batch/s]

Training batches on cuda:  58%|█████▊    | 147/253 [00:02<00:02, 52.67batch/s]

Training batches on cuda:  60%|██████    | 153/253 [00:02<00:01, 52.54batch/s]

Training batches on cuda:  63%|██████▎   | 159/253 [00:03<00:01, 52.52batch/s]

Training batches on cuda:  65%|██████▌   | 165/253 [00:03<00:01, 53.01batch/s]

Training batches on cuda:  68%|██████▊   | 171/253 [00:03<00:01, 52.47batch/s]

Training batches on cuda:  70%|██████▉   | 177/253 [00:03<00:01, 51.96batch/s]

Training batches on cuda:  72%|███████▏  | 183/253 [00:03<00:01, 51.41batch/s]

Training batches on cuda:  75%|███████▍  | 189/253 [00:03<00:01, 50.97batch/s]

Training batches on cuda:  77%|███████▋  | 195/253 [00:03<00:01, 50.58batch/s]

Training batches on cuda:  79%|███████▉  | 201/253 [00:03<00:01, 50.11batch/s]

Training batches on cuda:  82%|████████▏ | 207/253 [00:04<00:00, 49.81batch/s]

Training batches on cuda:  84%|████████▍ | 212/253 [00:04<00:00, 49.46batch/s]

Training batches on cuda:  86%|████████▌ | 217/253 [00:04<00:00, 49.10batch/s]

Training batches on cuda:  88%|████████▊ | 222/253 [00:04<00:00, 49.16batch/s]

Training batches on cuda:  90%|█████████ | 228/253 [00:04<00:00, 49.71batch/s]

Training batches on cuda:  92%|█████████▏| 234/253 [00:04<00:00, 50.86batch/s]

Training batches on cuda:  95%|█████████▍| 240/253 [00:04<00:00, 51.82batch/s]

Training batches on cuda:  97%|█████████▋| 246/253 [00:04<00:00, 52.30batch/s]

Training batches on cuda: 100%|█████████▉| 252/253 [00:04<00:00, 52.70batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:45<00:10,  5.12s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda:  90%|█████████ | 9/10 [00:45<00:05,  5.13s/epoch, loss=0.00107, prev_loss=0.00107]

Training batches on cuda:   0%|          | 0/253 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/253 [00:00<00:06, 35.97batch/s]

Training batches on cuda:   4%|▍         | 10/253 [00:00<00:06, 39.37batch/s]

Training batches on cuda:   6%|▌         | 15/253 [00:00<00:05, 41.79batch/s]

Training batches on cuda:   8%|▊         | 21/253 [00:00<00:05, 44.30batch/s]

Training batches on cuda:  10%|█         | 26/253 [00:00<00:05, 45.19batch/s]

Training batches on cuda:  12%|█▏        | 31/253 [00:00<00:04, 44.74batch/s]

Training batches on cuda:  15%|█▍        | 37/253 [00:00<00:04, 47.07batch/s]

Training batches on cuda:  17%|█▋        | 42/253 [00:00<00:04, 47.16batch/s]

Training batches on cuda:  19%|█▉        | 48/253 [00:01<00:04, 48.09batch/s]

Training batches on cuda:  21%|██        | 53/253 [00:01<00:04, 48.65batch/s]

Training batches on cuda:  23%|██▎       | 59/253 [00:01<00:03, 50.30batch/s]

Training batches on cuda:  26%|██▌       | 65/253 [00:01<00:03, 50.72batch/s]

Training batches on cuda:  28%|██▊       | 71/253 [00:01<00:03, 52.04batch/s]

Training batches on cuda:  30%|███       | 77/253 [00:01<00:03, 52.84batch/s]

Training batches on cuda:  33%|███▎      | 83/253 [00:01<00:03, 52.84batch/s]

Training batches on cuda:  35%|███▌      | 89/253 [00:01<00:03, 52.68batch/s]

Training batches on cuda:  38%|███▊      | 95/253 [00:01<00:02, 52.72batch/s]

Training batches on cuda:  40%|███▉      | 101/253 [00:01<00:02, 53.66batch/s]

Training batches on cuda:  42%|████▏     | 107/253 [00:02<00:02, 54.03batch/s]

Training batches on cuda:  45%|████▍     | 113/253 [00:02<00:02, 53.66batch/s]

Training batches on cuda:  47%|████▋     | 119/253 [00:02<00:02, 53.74batch/s]

Training batches on cuda:  49%|████▉     | 125/253 [00:02<00:02, 54.02batch/s]

Training batches on cuda:  52%|█████▏    | 131/253 [00:02<00:02, 53.86batch/s]

Training batches on cuda:  54%|█████▍    | 137/253 [00:02<00:02, 53.84batch/s]

Training batches on cuda:  57%|█████▋    | 143/253 [00:02<00:02, 53.23batch/s]

Training batches on cuda:  59%|█████▉    | 149/253 [00:02<00:01, 53.50batch/s]

Training batches on cuda:  61%|██████▏   | 155/253 [00:03<00:01, 53.22batch/s]

Training batches on cuda:  64%|██████▎   | 161/253 [00:03<00:01, 53.32batch/s]

Training batches on cuda:  66%|██████▌   | 167/253 [00:03<00:01, 53.09batch/s]

Training batches on cuda:  68%|██████▊   | 173/253 [00:03<00:01, 52.44batch/s]

Training batches on cuda:  71%|███████   | 179/253 [00:03<00:01, 52.68batch/s]

Training batches on cuda:  73%|███████▎  | 185/253 [00:03<00:01, 51.88batch/s]

Training batches on cuda:  75%|███████▌  | 191/253 [00:03<00:01, 51.77batch/s]

Training batches on cuda:  78%|███████▊  | 197/253 [00:03<00:01, 50.03batch/s]

Training batches on cuda:  80%|████████  | 203/253 [00:03<00:00, 51.32batch/s]

Training batches on cuda:  83%|████████▎ | 209/253 [00:04<00:00, 52.31batch/s]

Training batches on cuda:  85%|████████▍ | 215/253 [00:04<00:00, 52.39batch/s]

Training batches on cuda:  87%|████████▋ | 221/253 [00:04<00:00, 52.68batch/s]

Training batches on cuda:  90%|████████▉ | 227/253 [00:04<00:00, 53.13batch/s]

Training batches on cuda:  92%|█████████▏| 233/253 [00:04<00:00, 53.10batch/s]

Training batches on cuda:  94%|█████████▍| 239/253 [00:04<00:00, 53.16batch/s]

Training batches on cuda:  97%|█████████▋| 245/253 [00:04<00:00, 52.28batch/s]

Training batches on cuda:  99%|█████████▉| 251/253 [00:04<00:00, 52.06batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:51<00:05,  5.13s/epoch, loss=0.00107, prev_loss=0.00107]2021-09-16:19:16:25,181 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:16:26,192 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:16:41,236 INFO     [evaluator.py:657] Evaluation took 15.04s seconds
2021-09-16:19:16:41,404 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00, 10.00s/epoch, loss=0.00107, prev_loss=0.00107]
Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00,  6.72s/epoch, loss=0.00107, prev_loss=0.00107]
2021-09-16:19:16:41,405 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp83idqwnh'
2021-09-16:19:16:41,443 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp83idqwnh' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.25ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.35ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.49ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.61ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.60ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.61ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.62ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:16:56,331 INFO     [evaluator.py:657] Evaluation took 14.88s seconds
[I 2021-09-16 19:16:56,336] Trial 74 finished with value: 0.0008719394937195535 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.10636016110129308, 'optimizer.lr': 0.04247291034265411, 'negative_sampler.num_negs_per_pos': 45, 'training.batch_size': 1232}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:16:56,359 WARNING  [api.py:823] No random seed is specified. Setting to 1375387354.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/492 [00:00<00:16, 29.32batch/s]

Training batches on cuda:   2%|▏         | 12/492 [00:00<00:13, 36.60batch/s]

Training batches on cuda:   4%|▍         | 22/492 [00:00<00:10, 44.88batch/s]

Training batches on cuda:   7%|▋         | 32/492 [00:00<00:08, 53.01batch/s]

Training batches on cuda:   9%|▊         | 42/492 [00:00<00:07, 61.15batch/s]

Training batches on cuda:  10%|█         | 51/492 [00:00<00:06, 66.24batch/s]

Training batches on cuda:  12%|█▏        | 61/492 [00:00<00:05, 72.70batch/s]

Training batches on cuda:  14%|█▍        | 70/492 [00:00<00:05, 76.26batch/s]

Training batches on cuda:  16%|█▌        | 79/492 [00:00<00:05, 79.31batch/s]

Training batches on cuda:  18%|█▊        | 88/492 [00:01<00:04, 81.03batch/s]

Training batches on cuda:  20%|█▉        | 98/492 [00:01<00:04, 83.91batch/s]

Training batches on cuda:  22%|██▏       | 107/492 [00:01<00:04, 85.40batch/s]

Training batches on cuda:  24%|██▍       | 117/492 [00:01<00:04, 87.02batch/s]

Training batches on cuda:  26%|██▌       | 127/492 [00:01<00:04, 87.93batch/s]

Training batches on cuda:  28%|██▊       | 136/492 [00:01<00:04, 88.42batch/s]

Training batches on cuda:  29%|██▉       | 145/492 [00:01<00:03, 88.38batch/s]

Training batches on cuda:  31%|███▏      | 154/492 [00:01<00:03, 86.02batch/s]

Training batches on cuda:  33%|███▎      | 164/492 [00:01<00:03, 87.72batch/s]

Training batches on cuda:  35%|███▌      | 173/492 [00:02<00:03, 85.22batch/s]

Training batches on cuda:  37%|███▋      | 182/492 [00:02<00:03, 86.13batch/s]

Training batches on cuda:  39%|███▉      | 191/492 [00:02<00:03, 86.15batch/s]

Training batches on cuda:  41%|████      | 200/492 [00:02<00:03, 86.65batch/s]

Training batches on cuda:  42%|████▏     | 209/492 [00:02<00:03, 86.87batch/s]

Training batches on cuda:  44%|████▍     | 218/492 [00:02<00:03, 87.23batch/s]

Training batches on cuda:  46%|████▌     | 227/492 [00:02<00:03, 87.52batch/s]

Training batches on cuda:  48%|████▊     | 236/492 [00:02<00:02, 87.64batch/s]

Training batches on cuda:  50%|████▉     | 245/492 [00:02<00:02, 87.54batch/s]

Training batches on cuda:  52%|█████▏    | 254/492 [00:02<00:02, 87.87batch/s]

Training batches on cuda:  53%|█████▎    | 263/492 [00:03<00:02, 87.82batch/s]

Training batches on cuda:  55%|█████▌    | 272/492 [00:03<00:02, 87.55batch/s]

Training batches on cuda:  57%|█████▋    | 281/492 [00:03<00:02, 87.23batch/s]

Training batches on cuda:  59%|█████▉    | 290/492 [00:03<00:02, 87.11batch/s]

Training batches on cuda:  61%|██████    | 300/492 [00:03<00:02, 88.17batch/s]

Training batches on cuda:  63%|██████▎   | 309/492 [00:03<00:02, 88.12batch/s]

Training batches on cuda:  65%|██████▍   | 318/492 [00:03<00:02, 85.00batch/s]

Training batches on cuda:  66%|██████▋   | 327/492 [00:03<00:02, 80.37batch/s]

Training batches on cuda:  68%|██████▊   | 336/492 [00:03<00:02, 77.29batch/s]

Training batches on cuda:  70%|██████▉   | 344/492 [00:04<00:01, 77.34batch/s]

Training batches on cuda:  72%|███████▏  | 352/492 [00:04<00:01, 73.98batch/s]

Training batches on cuda:  73%|███████▎  | 360/492 [00:04<00:01, 75.65batch/s]

Training batches on cuda:  75%|███████▌  | 369/492 [00:04<00:01, 77.50batch/s]

Training batches on cuda:  77%|███████▋  | 378/492 [00:04<00:01, 78.94batch/s]

Training batches on cuda:  79%|███████▊  | 387/492 [00:04<00:01, 79.83batch/s]

Training batches on cuda:  80%|████████  | 396/492 [00:04<00:01, 81.96batch/s]

Training batches on cuda:  82%|████████▏ | 405/492 [00:04<00:01, 83.57batch/s]

Training batches on cuda:  84%|████████▍ | 414/492 [00:04<00:00, 83.49batch/s]

Training batches on cuda:  86%|████████▌ | 423/492 [00:05<00:00, 84.83batch/s]

Training batches on cuda:  88%|████████▊ | 432/492 [00:05<00:00, 85.22batch/s]

Training batches on cuda:  90%|████████▉ | 441/492 [00:05<00:00, 83.19batch/s]

Training batches on cuda:  91%|█████████▏| 450/492 [00:05<00:00, 83.62batch/s]

Training batches on cuda:  93%|█████████▎| 459/492 [00:05<00:00, 82.58batch/s]

Training batches on cuda:  95%|█████████▌| 468/492 [00:05<00:00, 82.63batch/s]

Training batches on cuda:  97%|█████████▋| 477/492 [00:05<00:00, 81.77batch/s]

Training batches on cuda:  99%|█████████▉| 486/492 [00:05<00:00, 82.28batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:06<?, ?epoch/s, loss=0.00338, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:54,  6.07s/epoch, loss=0.00338, prev_loss=nan]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 5/492 [00:00<00:09, 48.91batch/s]

Training batches on cuda:   3%|▎         | 14/492 [00:00<00:08, 56.29batch/s]

Training batches on cuda:   5%|▍         | 23/492 [00:00<00:07, 63.02batch/s]

Training batches on cuda:   7%|▋         | 32/492 [00:00<00:06, 68.18batch/s]

Training batches on cuda:   8%|▊         | 41/492 [00:00<00:06, 72.65batch/s]

Training batches on cuda:  10%|█         | 51/492 [00:00<00:05, 77.48batch/s]

Training batches on cuda:  12%|█▏        | 60/492 [00:00<00:05, 80.77batch/s]

Training batches on cuda:  14%|█▍        | 69/492 [00:00<00:05, 83.27batch/s]

Training batches on cuda:  16%|█▌        | 78/492 [00:00<00:04, 83.67batch/s]

Training batches on cuda:  18%|█▊        | 87/492 [00:01<00:04, 85.05batch/s]

Training batches on cuda:  20%|█▉        | 96/492 [00:01<00:04, 86.06batch/s]

Training batches on cuda:  22%|██▏       | 106/492 [00:01<00:04, 87.39batch/s]

Training batches on cuda:  23%|██▎       | 115/492 [00:01<00:04, 87.84batch/s]

Training batches on cuda:  25%|██▌       | 124/492 [00:01<00:04, 88.29batch/s]

Training batches on cuda:  27%|██▋       | 134/492 [00:01<00:04, 89.20batch/s]

Training batches on cuda:  29%|██▉       | 143/492 [00:01<00:03, 88.56batch/s]

Training batches on cuda:  31%|███       | 152/492 [00:01<00:03, 88.58batch/s]

Training batches on cuda:  33%|███▎      | 161/492 [00:01<00:03, 87.84batch/s]

Training batches on cuda:  35%|███▍      | 171/492 [00:01<00:03, 88.71batch/s]

Training batches on cuda:  37%|███▋      | 180/492 [00:02<00:03, 88.95batch/s]

Training batches on cuda:  39%|███▊      | 190/492 [00:02<00:03, 89.49batch/s]

Training batches on cuda:  40%|████      | 199/492 [00:02<00:03, 89.01batch/s]

Training batches on cuda:  42%|████▏     | 208/492 [00:02<00:03, 88.89batch/s]

Training batches on cuda:  44%|████▍     | 217/492 [00:02<00:03, 88.99batch/s]

Training batches on cuda:  46%|████▌     | 226/492 [00:02<00:03, 88.09batch/s]

Training batches on cuda:  48%|████▊     | 235/492 [00:02<00:02, 88.45batch/s]

Training batches on cuda:  50%|████▉     | 244/492 [00:02<00:02, 87.49batch/s]

Training batches on cuda:  51%|█████▏    | 253/492 [00:02<00:02, 88.06batch/s]

Training batches on cuda:  53%|█████▎    | 262/492 [00:03<00:02, 87.60batch/s]

Training batches on cuda:  55%|█████▌    | 271/492 [00:03<00:02, 85.35batch/s]

Training batches on cuda:  57%|█████▋    | 280/492 [00:03<00:02, 85.62batch/s]

Training batches on cuda:  59%|█████▊    | 289/492 [00:03<00:02, 82.20batch/s]

Training batches on cuda:  61%|██████    | 299/492 [00:03<00:02, 84.90batch/s]

Training batches on cuda:  63%|██████▎   | 309/492 [00:03<00:02, 88.09batch/s]

Training batches on cuda:  65%|██████▍   | 319/492 [00:03<00:01, 89.42batch/s]

Training batches on cuda:  67%|██████▋   | 328/492 [00:03<00:01, 85.82batch/s]

Training batches on cuda:  68%|██████▊   | 337/492 [00:03<00:01, 84.15batch/s]

Training batches on cuda:  70%|███████   | 346/492 [00:04<00:01, 83.87batch/s]

Training batches on cuda:  72%|███████▏  | 355/492 [00:04<00:01, 83.25batch/s]

Training batches on cuda:  74%|███████▍  | 364/492 [00:04<00:01, 83.19batch/s]

Training batches on cuda:  76%|███████▌  | 373/492 [00:04<00:01, 81.77batch/s]

Training batches on cuda:  78%|███████▊  | 382/492 [00:04<00:01, 81.05batch/s]

Training batches on cuda:  79%|███████▉  | 391/492 [00:04<00:01, 81.25batch/s]

Training batches on cuda:  81%|████████▏ | 400/492 [00:04<00:01, 80.29batch/s]

Training batches on cuda:  83%|████████▎ | 409/492 [00:04<00:01, 80.86batch/s]

Training batches on cuda:  85%|████████▍ | 418/492 [00:04<00:00, 79.55batch/s]

Training batches on cuda:  87%|████████▋ | 426/492 [00:05<00:00, 79.47batch/s]

Training batches on cuda:  88%|████████▊ | 435/492 [00:05<00:00, 81.97batch/s]

Training batches on cuda:  90%|█████████ | 444/492 [00:05<00:00, 82.74batch/s]

Training batches on cuda:  92%|█████████▏| 453/492 [00:05<00:00, 83.27batch/s]

Training batches on cuda:  94%|█████████▍| 462/492 [00:05<00:00, 84.52batch/s]

Training batches on cuda:  96%|█████████▌| 471/492 [00:05<00:00, 85.39batch/s]

Training batches on cuda:  98%|█████████▊| 480/492 [00:05<00:00, 86.03batch/s]

Training batches on cuda: 100%|█████████▉| 490/492 [00:05<00:00, 87.32batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:12<00:54,  6.07s/epoch, loss=0.00337, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:48,  6.05s/epoch, loss=0.00337, prev_loss=nan]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 5/492 [00:00<00:09, 48.95batch/s]

Training batches on cuda:   3%|▎         | 14/492 [00:00<00:08, 55.75batch/s]

Training batches on cuda:   5%|▍         | 23/492 [00:00<00:07, 61.66batch/s]

Training batches on cuda:   7%|▋         | 32/492 [00:00<00:06, 67.66batch/s]

Training batches on cuda:   8%|▊         | 41/492 [00:00<00:06, 72.56batch/s]

Training batches on cuda:  10%|█         | 50/492 [00:00<00:05, 75.30batch/s]

Training batches on cuda:  12%|█▏        | 59/492 [00:00<00:05, 77.20batch/s]

Training batches on cuda:  14%|█▍        | 68/492 [00:00<00:05, 77.87batch/s]

Training batches on cuda:  16%|█▌        | 77/492 [00:00<00:05, 80.14batch/s]

Training batches on cuda:  17%|█▋        | 86/492 [00:01<00:04, 81.72batch/s]

Training batches on cuda:  19%|█▉        | 95/492 [00:01<00:04, 83.47batch/s]

Training batches on cuda:  21%|██        | 104/492 [00:01<00:04, 84.69batch/s]

Training batches on cuda:  23%|██▎       | 113/492 [00:01<00:04, 85.50batch/s]

Training batches on cuda:  25%|██▍       | 122/492 [00:01<00:04, 84.22batch/s]

Training batches on cuda:  27%|██▋       | 131/492 [00:01<00:04, 84.00batch/s]

Training batches on cuda:  28%|██▊       | 140/492 [00:01<00:04, 83.40batch/s]

Training batches on cuda:  30%|███       | 149/492 [00:01<00:04, 84.81batch/s]

Training batches on cuda:  32%|███▏      | 158/492 [00:01<00:03, 84.46batch/s]

Training batches on cuda:  34%|███▍      | 167/492 [00:02<00:04, 80.89batch/s]

Training batches on cuda:  36%|███▌      | 176/492 [00:02<00:03, 82.95batch/s]

Training batches on cuda:  38%|███▊      | 185/492 [00:02<00:03, 83.97batch/s]

Training batches on cuda:  39%|███▉      | 194/492 [00:02<00:03, 80.97batch/s]

Training batches on cuda:  41%|████▏     | 203/492 [00:02<00:03, 79.40batch/s]

Training batches on cuda:  43%|████▎     | 213/492 [00:02<00:03, 82.58batch/s]

Training batches on cuda:  45%|████▌     | 222/492 [00:02<00:03, 84.13batch/s]

Training batches on cuda:  47%|████▋     | 232/492 [00:02<00:03, 86.19batch/s]

Training batches on cuda:  49%|████▉     | 241/492 [00:02<00:02, 86.77batch/s]

Training batches on cuda:  51%|█████     | 250/492 [00:03<00:02, 83.31batch/s]

Training batches on cuda:  53%|█████▎    | 259/492 [00:03<00:02, 83.55batch/s]

Training batches on cuda:  54%|█████▍    | 268/492 [00:03<00:02, 83.66batch/s]

Training batches on cuda:  56%|█████▋    | 277/492 [00:03<00:02, 82.70batch/s]

Training batches on cuda:  58%|█████▊    | 286/492 [00:03<00:02, 83.70batch/s]

Training batches on cuda:  60%|█████▉    | 295/492 [00:03<00:02, 82.29batch/s]

Training batches on cuda:  62%|██████▏   | 304/492 [00:03<00:02, 82.53batch/s]

Training batches on cuda:  64%|██████▎   | 313/492 [00:03<00:02, 81.34batch/s]

Training batches on cuda:  65%|██████▌   | 322/492 [00:03<00:02, 76.87batch/s]

Training batches on cuda:  67%|██████▋   | 330/492 [00:04<00:02, 77.23batch/s]

Training batches on cuda:  69%|██████▉   | 339/492 [00:04<00:01, 79.40batch/s]

Training batches on cuda:  71%|███████   | 348/492 [00:04<00:01, 81.28batch/s]

Training batches on cuda:  73%|███████▎  | 357/492 [00:04<00:01, 80.42batch/s]

Training batches on cuda:  74%|███████▍  | 366/492 [00:04<00:01, 78.21batch/s]

Training batches on cuda:  76%|███████▌  | 375/492 [00:04<00:01, 79.51batch/s]

Training batches on cuda:  78%|███████▊  | 384/492 [00:04<00:01, 81.89batch/s]

Training batches on cuda:  80%|███████▉  | 393/492 [00:04<00:01, 77.84batch/s]

Training batches on cuda:  82%|████████▏ | 402/492 [00:04<00:01, 78.78batch/s]

Training batches on cuda:  84%|████████▎ | 412/492 [00:05<00:00, 81.92batch/s]

Training batches on cuda:  86%|████████▌ | 421/492 [00:05<00:00, 84.02batch/s]

Training batches on cuda:  87%|████████▋ | 430/492 [00:05<00:00, 83.75batch/s]

Training batches on cuda:  89%|████████▉ | 439/492 [00:05<00:00, 81.85batch/s]

Training batches on cuda:  91%|█████████ | 448/492 [00:05<00:00, 83.22batch/s]

Training batches on cuda:  93%|█████████▎| 458/492 [00:05<00:00, 84.92batch/s]

Training batches on cuda:  95%|█████████▍| 467/492 [00:05<00:00, 85.42batch/s]

Training batches on cuda:  97%|█████████▋| 476/492 [00:05<00:00, 84.86batch/s]

Training batches on cuda:  99%|█████████▊| 485/492 [00:05<00:00, 85.61batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:18<00:48,  6.05s/epoch, loss=0.00338, prev_loss=0.00337]
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:42,  6.12s/epoch, loss=0.00338, prev_loss=0.00337]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 4/492 [00:00<00:12, 39.18batch/s]

Training batches on cuda:   3%|▎         | 14/492 [00:00<00:10, 47.61batch/s]

Training batches on cuda:   5%|▍         | 24/492 [00:00<00:08, 56.04batch/s]

Training batches on cuda:   7%|▋         | 34/492 [00:00<00:07, 64.02batch/s]

Training batches on cuda:   9%|▉         | 44/492 [00:00<00:06, 71.03batch/s]

Training batches on cuda:  11%|█         | 54/492 [00:00<00:05, 76.95batch/s]

Training batches on cuda:  13%|█▎        | 64/492 [00:00<00:05, 81.59batch/s]

Training batches on cuda:  15%|█▍        | 73/492 [00:00<00:05, 77.68batch/s]

Training batches on cuda:  17%|█▋        | 83/492 [00:00<00:04, 82.31batch/s]

Training batches on cuda:  19%|█▉        | 93/492 [00:01<00:04, 85.93batch/s]

Training batches on cuda:  21%|██        | 103/492 [00:01<00:04, 89.20batch/s]

Training batches on cuda:  23%|██▎       | 114/492 [00:01<00:04, 92.98batch/s]

Training batches on cuda:  25%|██▌       | 125/492 [00:01<00:03, 95.13batch/s]

Training batches on cuda:  28%|██▊       | 136/492 [00:01<00:03, 96.82batch/s]

Training batches on cuda:  30%|██▉       | 147/492 [00:01<00:03, 98.00batch/s]

Training batches on cuda:  32%|███▏      | 157/492 [00:01<00:03, 98.28batch/s]

Training batches on cuda:  34%|███▍      | 168/492 [00:01<00:03, 99.24batch/s]

Training batches on cuda:  36%|███▋      | 179/492 [00:01<00:03, 99.98batch/s]

Training batches on cuda:  39%|███▊      | 190/492 [00:02<00:03, 99.51batch/s]

Training batches on cuda:  41%|████      | 200/492 [00:02<00:02, 99.58batch/s]

Training batches on cuda:  43%|████▎     | 210/492 [00:02<00:02, 99.49batch/s]

Training batches on cuda:  45%|████▍     | 221/492 [00:02<00:02, 99.91batch/s]

Training batches on cuda:  47%|████▋     | 231/492 [00:02<00:02, 99.75batch/s]

Training batches on cuda:  49%|████▉     | 242/492 [00:02<00:02, 99.79batch/s]

Training batches on cuda:  51%|█████     | 252/492 [00:02<00:02, 99.30batch/s]

Training batches on cuda:  53%|█████▎    | 262/492 [00:02<00:02, 98.92batch/s]

Training batches on cuda:  55%|█████▌    | 272/492 [00:02<00:02, 98.74batch/s]

Training batches on cuda:  57%|█████▋    | 282/492 [00:02<00:02, 98.85batch/s]

Training batches on cuda:  60%|█████▉    | 293/492 [00:03<00:02, 99.18batch/s]

Training batches on cuda:  62%|██████▏   | 304/492 [00:03<00:01, 99.60batch/s]

Training batches on cuda:  64%|██████▍   | 314/492 [00:03<00:01, 95.47batch/s]

Training batches on cuda:  66%|██████▌   | 324/492 [00:03<00:01, 91.89batch/s]

Training batches on cuda:  68%|██████▊   | 334/492 [00:03<00:01, 90.70batch/s]

Training batches on cuda:  70%|██████▉   | 344/492 [00:03<00:01, 89.36batch/s]

Training batches on cuda:  72%|███████▏  | 354/492 [00:03<00:01, 90.54batch/s]

Training batches on cuda:  74%|███████▍  | 364/492 [00:03<00:01, 90.40batch/s]

Training batches on cuda:  76%|███████▌  | 374/492 [00:03<00:01, 89.64batch/s]

Training batches on cuda:  78%|███████▊  | 383/492 [00:04<00:01, 87.87batch/s]

Training batches on cuda:  80%|███████▉  | 392/492 [00:04<00:01, 84.49batch/s]

Training batches on cuda:  82%|████████▏ | 402/492 [00:04<00:01, 86.62batch/s]

Training batches on cuda:  84%|████████▎ | 411/492 [00:04<00:00, 83.23batch/s]

Training batches on cuda:  85%|████████▌ | 420/492 [00:04<00:00, 84.90batch/s]

Training batches on cuda:  87%|████████▋ | 429/492 [00:04<00:00, 85.47batch/s]

Training batches on cuda:  89%|████████▉ | 438/492 [00:04<00:00, 86.05batch/s]

Training batches on cuda:  91%|█████████ | 447/492 [00:04<00:00, 84.59batch/s]

Training batches on cuda:  93%|█████████▎| 456/492 [00:04<00:00, 85.45batch/s]

Training batches on cuda:  95%|█████████▍| 466/492 [00:05<00:00, 86.92batch/s]

Training batches on cuda:  97%|█████████▋| 475/492 [00:05<00:00, 86.89batch/s]

Training batches on cuda:  98%|█████████▊| 484/492 [00:05<00:00, 85.37batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:23<00:42,  6.12s/epoch, loss=0.00338, prev_loss=0.00338]
Training epochs on cuda:  40%|████      | 4/10 [00:23<00:35,  5.96s/epoch, loss=0.00338, prev_loss=0.00338]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 5/492 [00:00<00:10, 48.07batch/s]

Training batches on cuda:   3%|▎         | 13/492 [00:00<00:08, 54.13batch/s]

Training batches on cuda:   4%|▍         | 19/492 [00:00<00:08, 55.47batch/s]

Training batches on cuda:   6%|▌         | 28/492 [00:00<00:07, 62.18batch/s]

Training batches on cuda:   8%|▊         | 37/492 [00:00<00:06, 67.74batch/s]

Training batches on cuda:   9%|▉         | 45/492 [00:00<00:06, 69.90batch/s]

Training batches on cuda:  11%|█         | 54/492 [00:00<00:06, 72.39batch/s]

Training batches on cuda:  13%|█▎        | 63/492 [00:00<00:05, 75.09batch/s]

Training batches on cuda:  14%|█▍        | 71/492 [00:00<00:05, 74.88batch/s]

Training batches on cuda:  16%|█▋        | 80/492 [00:01<00:05, 76.82batch/s]

Training batches on cuda:  18%|█▊        | 88/492 [00:01<00:05, 76.94batch/s]

Training batches on cuda:  20%|█▉        | 97/492 [00:01<00:04, 79.80batch/s]

Training batches on cuda:  22%|██▏       | 106/492 [00:01<00:04, 81.19batch/s]

Training batches on cuda:  23%|██▎       | 115/492 [00:01<00:04, 83.60batch/s]

Training batches on cuda:  25%|██▌       | 124/492 [00:01<00:04, 85.15batch/s]

Training batches on cuda:  27%|██▋       | 133/492 [00:01<00:04, 84.45batch/s]

Training batches on cuda:  29%|██▉       | 142/492 [00:01<00:04, 84.04batch/s]

Training batches on cuda:  31%|███       | 152/492 [00:01<00:03, 86.37batch/s]

Training batches on cuda:  33%|███▎      | 161/492 [00:02<00:03, 84.90batch/s]

Training batches on cuda:  35%|███▍      | 170/492 [00:02<00:03, 84.56batch/s]

Training batches on cuda:  36%|███▋      | 179/492 [00:02<00:03, 84.11batch/s]

Training batches on cuda:  38%|███▊      | 188/492 [00:02<00:03, 83.62batch/s]

Training batches on cuda:  40%|████      | 197/492 [00:02<00:03, 81.37batch/s]

Training batches on cuda:  42%|████▏     | 206/492 [00:02<00:03, 81.30batch/s]

Training batches on cuda:  44%|████▎     | 215/492 [00:02<00:03, 77.67batch/s]

Training batches on cuda:  45%|████▌     | 223/492 [00:02<00:03, 76.61batch/s]

Training batches on cuda:  47%|████▋     | 232/492 [00:02<00:03, 78.48batch/s]

Training batches on cuda:  49%|████▉     | 241/492 [00:03<00:03, 79.13batch/s]

Training batches on cuda:  51%|█████     | 249/492 [00:03<00:03, 79.20batch/s]

Training batches on cuda:  52%|█████▏    | 258/492 [00:03<00:02, 78.99batch/s]

Training batches on cuda:  54%|█████▍    | 266/492 [00:03<00:02, 78.71batch/s]

Training batches on cuda:  56%|█████▌    | 275/492 [00:03<00:02, 81.26batch/s]

Training batches on cuda:  58%|█████▊    | 285/492 [00:03<00:02, 83.66batch/s]

Training batches on cuda:  60%|█████▉    | 294/492 [00:03<00:02, 84.36batch/s]

Training batches on cuda:  62%|██████▏   | 303/492 [00:03<00:02, 85.85batch/s]

Training batches on cuda:  64%|██████▎   | 313/492 [00:03<00:02, 87.15batch/s]

Training batches on cuda:  65%|██████▌   | 322/492 [00:03<00:01, 87.39batch/s]

Training batches on cuda:  67%|██████▋   | 332/492 [00:04<00:01, 88.23batch/s]

Training batches on cuda:  69%|██████▉   | 341/492 [00:04<00:01, 88.25batch/s]

Training batches on cuda:  71%|███████   | 350/492 [00:04<00:01, 88.48batch/s]

Training batches on cuda:  73%|███████▎  | 359/492 [00:04<00:01, 87.71batch/s]

Training batches on cuda:  75%|███████▌  | 369/492 [00:04<00:01, 88.35batch/s]

Training batches on cuda:  77%|███████▋  | 379/492 [00:04<00:01, 89.17batch/s]

Training batches on cuda:  79%|███████▉  | 388/492 [00:04<00:01, 88.65batch/s]

Training batches on cuda:  81%|████████  | 398/492 [00:04<00:01, 89.31batch/s]

Training batches on cuda:  83%|████████▎ | 407/492 [00:04<00:00, 86.88batch/s]

Training batches on cuda:  85%|████████▍ | 416/492 [00:05<00:00, 85.23batch/s]

Training batches on cuda:  86%|████████▋ | 425/492 [00:05<00:00, 84.95batch/s]

Training batches on cuda:  88%|████████▊ | 434/492 [00:05<00:00, 84.63batch/s]

Training batches on cuda:  90%|█████████ | 443/492 [00:05<00:00, 83.38batch/s]

Training batches on cuda:  92%|█████████▏| 452/492 [00:05<00:00, 83.84batch/s]

Training batches on cuda:  94%|█████████▎| 461/492 [00:05<00:00, 85.04batch/s]

Training batches on cuda:  96%|█████████▌| 471/492 [00:05<00:00, 86.82batch/s]

Training batches on cuda:  98%|█████████▊| 480/492 [00:05<00:00, 85.48batch/s]

Training batches on cuda:  99%|█████████▉| 489/492 [00:05<00:00, 86.56batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:30<00:35,  5.96s/epoch, loss=0.00338, prev_loss=0.00338]
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:30,  6.03s/epoch, loss=0.00338, prev_loss=0.00338]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 5/492 [00:00<00:09, 49.79batch/s]

Training batches on cuda:   3%|▎         | 15/492 [00:00<00:08, 57.57batch/s]

Training batches on cuda:   5%|▍         | 24/492 [00:00<00:07, 64.54batch/s]

Training batches on cuda:   7%|▋         | 34/492 [00:00<00:06, 70.64batch/s]

Training batches on cuda:   9%|▉         | 44/492 [00:00<00:05, 76.36batch/s]

Training batches on cuda:  11%|█         | 54/492 [00:00<00:05, 81.56batch/s]

Training batches on cuda:  13%|█▎        | 64/492 [00:00<00:05, 85.59batch/s]

Training batches on cuda:  15%|█▌        | 74/492 [00:00<00:04, 88.68batch/s]

Training batches on cuda:  17%|█▋        | 84/492 [00:00<00:04, 90.96batch/s]

Training batches on cuda:  19%|█▉        | 94/492 [00:01<00:04, 91.35batch/s]

Training batches on cuda:  21%|██        | 104/492 [00:01<00:04, 91.33batch/s]

Training batches on cuda:  23%|██▎       | 114/492 [00:01<00:04, 91.34batch/s]

Training batches on cuda:  25%|██▌       | 124/492 [00:01<00:04, 89.35batch/s]

Training batches on cuda:  27%|██▋       | 133/492 [00:01<00:04, 89.49batch/s]

Training batches on cuda:  29%|██▉       | 143/492 [00:01<00:03, 89.75batch/s]

Training batches on cuda:  31%|███       | 152/492 [00:01<00:03, 89.72batch/s]

Training batches on cuda:  33%|███▎      | 161/492 [00:01<00:03, 89.48batch/s]

Training batches on cuda:  35%|███▍      | 171/492 [00:01<00:03, 89.78batch/s]

Training batches on cuda:  37%|███▋      | 181/492 [00:02<00:03, 89.91batch/s]

Training batches on cuda:  39%|███▊      | 190/492 [00:02<00:03, 89.80batch/s]

Training batches on cuda:  41%|████      | 200/492 [00:02<00:03, 90.08batch/s]

Training batches on cuda:  43%|████▎     | 210/492 [00:02<00:03, 88.26batch/s]

Training batches on cuda:  45%|████▍     | 219/492 [00:02<00:03, 86.86batch/s]

Training batches on cuda:  46%|████▋     | 228/492 [00:02<00:03, 86.32batch/s]

Training batches on cuda:  48%|████▊     | 237/492 [00:02<00:02, 86.51batch/s]

Training batches on cuda:  50%|█████     | 246/492 [00:02<00:02, 87.33batch/s]

Training batches on cuda:  52%|█████▏    | 256/492 [00:02<00:02, 88.00batch/s]

Training batches on cuda:  54%|█████▍    | 265/492 [00:02<00:02, 87.76batch/s]

Training batches on cuda:  56%|█████▌    | 274/492 [00:03<00:02, 87.97batch/s]

Training batches on cuda:  58%|█████▊    | 284/492 [00:03<00:02, 88.80batch/s]

Training batches on cuda:  60%|█████▉    | 293/492 [00:03<00:02, 85.73batch/s]

Training batches on cuda:  61%|██████▏   | 302/492 [00:03<00:02, 86.42batch/s]

Training batches on cuda:  63%|██████▎   | 312/492 [00:03<00:02, 87.64batch/s]

Training batches on cuda:  65%|██████▌   | 321/492 [00:03<00:01, 87.09batch/s]

Training batches on cuda:  67%|██████▋   | 330/492 [00:03<00:01, 87.38batch/s]

Training batches on cuda:  69%|██████▉   | 340/492 [00:03<00:01, 89.16batch/s]

Training batches on cuda:  71%|███████   | 350/492 [00:03<00:01, 89.51batch/s]

Training batches on cuda:  73%|███████▎  | 359/492 [00:04<00:01, 89.57batch/s]

Training batches on cuda:  75%|███████▍  | 368/492 [00:04<00:01, 87.57batch/s]

Training batches on cuda:  77%|███████▋  | 377/492 [00:04<00:01, 85.88batch/s]

Training batches on cuda:  78%|███████▊  | 386/492 [00:04<00:01, 85.54batch/s]

Training batches on cuda:  80%|████████  | 395/492 [00:04<00:01, 86.17batch/s]

Training batches on cuda:  82%|████████▏ | 404/492 [00:04<00:01, 86.32batch/s]

Training batches on cuda:  84%|████████▍ | 413/492 [00:04<00:00, 86.62batch/s]

Training batches on cuda:  86%|████████▌ | 422/492 [00:04<00:00, 87.17batch/s]

Training batches on cuda:  88%|████████▊ | 432/492 [00:04<00:00, 88.34batch/s]

Training batches on cuda:  90%|████████▉ | 442/492 [00:04<00:00, 89.10batch/s]

Training batches on cuda:  92%|█████████▏| 452/492 [00:05<00:00, 89.55batch/s]

Training batches on cuda:  94%|█████████▎| 461/492 [00:05<00:00, 89.57batch/s]

Training batches on cuda:  96%|█████████▌| 470/492 [00:05<00:00, 87.77batch/s]

Training batches on cuda:  97%|█████████▋| 479/492 [00:05<00:00, 87.20batch/s]

Training batches on cuda:  99%|█████████▉| 488/492 [00:05<00:00, 85.79batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:36<00:30,  6.03s/epoch, loss=0.00338, prev_loss=0.00338]
Training epochs on cuda:  60%|██████    | 6/10 [00:36<00:23,  5.98s/epoch, loss=0.00338, prev_loss=0.00338]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/492 [00:00<00:17, 27.97batch/s]

Training batches on cuda:   2%|▏         | 12/492 [00:00<00:13, 34.87batch/s]

Training batches on cuda:   4%|▍         | 21/492 [00:00<00:11, 42.25batch/s]

Training batches on cuda:   6%|▌         | 30/492 [00:00<00:09, 49.58batch/s]

Training batches on cuda:   8%|▊         | 39/492 [00:00<00:08, 56.55batch/s]

Training batches on cuda:  10%|▉         | 48/492 [00:00<00:07, 62.58batch/s]

Training batches on cuda:  12%|█▏        | 57/492 [00:00<00:06, 67.21batch/s]

Training batches on cuda:  13%|█▎        | 66/492 [00:00<00:05, 71.82batch/s]

Training batches on cuda:  15%|█▌        | 75/492 [00:00<00:05, 75.29batch/s]

Training batches on cuda:  17%|█▋        | 83/492 [00:01<00:05, 76.48batch/s]

Training batches on cuda:  18%|█▊        | 91/492 [00:01<00:05, 76.26batch/s]

Training batches on cuda:  20%|██        | 100/492 [00:01<00:05, 76.86batch/s]

Training batches on cuda:  22%|██▏       | 109/492 [00:01<00:04, 78.14batch/s]

Training batches on cuda:  24%|██▍       | 118/492 [00:01<00:04, 79.83batch/s]

Training batches on cuda:  26%|██▌       | 127/492 [00:01<00:04, 81.24batch/s]

Training batches on cuda:  28%|██▊       | 136/492 [00:01<00:04, 83.31batch/s]

Training batches on cuda:  30%|██▉       | 146/492 [00:01<00:04, 85.33batch/s]

Training batches on cuda:  32%|███▏      | 155/492 [00:01<00:03, 86.55batch/s]

Training batches on cuda:  33%|███▎      | 164/492 [00:02<00:03, 86.66batch/s]

Training batches on cuda:  35%|███▌      | 173/492 [00:02<00:03, 87.38batch/s]

Training batches on cuda:  37%|███▋      | 183/492 [00:02<00:03, 89.10batch/s]

Training batches on cuda:  39%|███▉      | 193/492 [00:02<00:03, 89.37batch/s]

Training batches on cuda:  41%|████      | 202/492 [00:02<00:03, 88.05batch/s]

Training batches on cuda:  43%|████▎     | 212/492 [00:02<00:03, 88.88batch/s]

Training batches on cuda:  45%|████▍     | 221/492 [00:02<00:03, 89.15batch/s]

Training batches on cuda:  47%|████▋     | 230/492 [00:02<00:03, 86.27batch/s]

Training batches on cuda:  49%|████▊     | 239/492 [00:02<00:02, 85.00batch/s]

Training batches on cuda:  50%|█████     | 248/492 [00:02<00:02, 84.48batch/s]

Training batches on cuda:  52%|█████▏    | 257/492 [00:03<00:02, 85.18batch/s]

Training batches on cuda:  54%|█████▍    | 266/492 [00:03<00:02, 85.70batch/s]

Training batches on cuda:  56%|█████▌    | 275/492 [00:03<00:02, 76.92batch/s]

Training batches on cuda:  58%|█████▊    | 283/492 [00:03<00:02, 77.55batch/s]

Training batches on cuda:  59%|█████▉    | 292/492 [00:03<00:02, 79.15batch/s]

Training batches on cuda:  61%|██████    | 301/492 [00:03<00:02, 80.44batch/s]

Training batches on cuda:  63%|██████▎   | 310/492 [00:03<00:02, 81.08batch/s]

Training batches on cuda:  65%|██████▍   | 319/492 [00:03<00:02, 82.90batch/s]

Training batches on cuda:  67%|██████▋   | 328/492 [00:03<00:01, 84.33batch/s]

Training batches on cuda:  68%|██████▊   | 337/492 [00:04<00:01, 83.75batch/s]

Training batches on cuda:  70%|███████   | 346/492 [00:04<00:01, 85.09batch/s]

Training batches on cuda:  72%|███████▏  | 355/492 [00:04<00:01, 86.39batch/s]

Training batches on cuda:  74%|███████▍  | 364/492 [00:04<00:01, 86.88batch/s]

Training batches on cuda:  76%|███████▌  | 373/492 [00:04<00:01, 86.71batch/s]

Training batches on cuda:  78%|███████▊  | 382/492 [00:04<00:01, 87.25batch/s]

Training batches on cuda:  79%|███████▉  | 391/492 [00:04<00:01, 87.93batch/s]

Training batches on cuda:  81%|████████▏ | 400/492 [00:04<00:01, 87.19batch/s]

Training batches on cuda:  83%|████████▎ | 409/492 [00:04<00:00, 87.86batch/s]

Training batches on cuda:  85%|████████▍ | 418/492 [00:05<00:00, 88.43batch/s]

Training batches on cuda:  87%|████████▋ | 427/492 [00:05<00:00, 88.25batch/s]

Training batches on cuda:  89%|████████▊ | 436/492 [00:05<00:00, 87.20batch/s]

Training batches on cuda:  90%|█████████ | 445/492 [00:05<00:00, 87.21batch/s]

Training batches on cuda:  92%|█████████▏| 455/492 [00:05<00:00, 89.34batch/s]

Training batches on cuda:  94%|█████████▍| 464/492 [00:05<00:00, 89.41batch/s]

Training batches on cuda:  96%|█████████▌| 473/492 [00:05<00:00, 89.14batch/s]

Training batches on cuda:  98%|█████████▊| 482/492 [00:05<00:00, 87.62batch/s]

Training batches on cuda: 100%|█████████▉| 491/492 [00:05<00:00, 86.21batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:42<00:23,  5.98s/epoch, loss=0.00338, prev_loss=0.00338]
Training epochs on cuda:  70%|███████   | 7/10 [00:42<00:17,  6.00s/epoch, loss=0.00338, prev_loss=0.00338]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 6/492 [00:00<00:08, 56.44batch/s]

Training batches on cuda:   3%|▎         | 14/492 [00:00<00:07, 61.22batch/s]

Training batches on cuda:   4%|▍         | 21/492 [00:00<00:07, 63.37batch/s]

Training batches on cuda:   6%|▋         | 31/492 [00:00<00:06, 70.80batch/s]

Training batches on cuda:   8%|▊         | 41/492 [00:00<00:05, 77.22batch/s]

Training batches on cuda:  10%|█         | 50/492 [00:00<00:05, 79.78batch/s]

Training batches on cuda:  12%|█▏        | 59/492 [00:00<00:05, 81.22batch/s]

Training batches on cuda:  14%|█▍        | 69/492 [00:00<00:05, 84.43batch/s]

Training batches on cuda:  16%|█▌        | 78/492 [00:00<00:05, 81.66batch/s]

Training batches on cuda:  18%|█▊        | 87/492 [00:01<00:04, 82.54batch/s]

Training batches on cuda:  20%|█▉        | 96/492 [00:01<00:04, 83.86batch/s]

Training batches on cuda:  21%|██▏       | 105/492 [00:01<00:04, 85.13batch/s]

Training batches on cuda:  23%|██▎       | 115/492 [00:01<00:04, 87.01batch/s]

Training batches on cuda:  25%|██▌       | 125/492 [00:01<00:04, 89.74batch/s]

Training batches on cuda:  27%|██▋       | 135/492 [00:01<00:03, 89.98batch/s]

Training batches on cuda:  29%|██▉       | 145/492 [00:01<00:03, 90.92batch/s]

Training batches on cuda:  32%|███▏      | 155/492 [00:01<00:03, 89.58batch/s]

Training batches on cuda:  33%|███▎      | 164/492 [00:01<00:03, 88.36batch/s]

Training batches on cuda:  35%|███▌      | 173/492 [00:02<00:03, 88.56batch/s]

Training batches on cuda:  37%|███▋      | 182/492 [00:02<00:03, 87.52batch/s]

Training batches on cuda:  39%|███▉      | 191/492 [00:02<00:03, 84.26batch/s]

Training batches on cuda:  41%|████      | 200/492 [00:02<00:03, 82.48batch/s]

Training batches on cuda:  42%|████▏     | 209/492 [00:02<00:03, 83.79batch/s]

Training batches on cuda:  44%|████▍     | 218/492 [00:02<00:03, 84.15batch/s]

Training batches on cuda:  46%|████▌     | 227/492 [00:02<00:03, 82.90batch/s]

Training batches on cuda:  48%|████▊     | 236/492 [00:02<00:03, 83.57batch/s]

Training batches on cuda:  50%|████▉     | 245/492 [00:02<00:03, 81.94batch/s]

Training batches on cuda:  52%|█████▏    | 254/492 [00:03<00:02, 82.77batch/s]

Training batches on cuda:  53%|█████▎    | 263/492 [00:03<00:02, 84.69batch/s]

Training batches on cuda:  55%|█████▌    | 272/492 [00:03<00:02, 85.78batch/s]

Training batches on cuda:  57%|█████▋    | 281/492 [00:03<00:02, 86.12batch/s]

Training batches on cuda:  59%|█████▉    | 290/492 [00:03<00:02, 85.25batch/s]

Training batches on cuda:  61%|██████    | 299/492 [00:03<00:02, 86.53batch/s]

Training batches on cuda:  63%|██████▎   | 308/492 [00:03<00:02, 87.13batch/s]

Training batches on cuda:  64%|██████▍   | 317/492 [00:03<00:01, 87.52batch/s]

Training batches on cuda:  66%|██████▋   | 326/492 [00:03<00:01, 87.61batch/s]

Training batches on cuda:  68%|██████▊   | 335/492 [00:03<00:01, 87.50batch/s]

Training batches on cuda:  70%|██████▉   | 344/492 [00:04<00:01, 87.86batch/s]

Training batches on cuda:  72%|███████▏  | 353/492 [00:04<00:01, 87.36batch/s]

Training batches on cuda:  74%|███████▎  | 362/492 [00:04<00:01, 86.17batch/s]

Training batches on cuda:  75%|███████▌  | 371/492 [00:04<00:01, 87.00batch/s]

Training batches on cuda:  77%|███████▋  | 381/492 [00:04<00:01, 87.85batch/s]

Training batches on cuda:  79%|███████▉  | 390/492 [00:04<00:01, 85.73batch/s]

Training batches on cuda:  81%|████████  | 399/492 [00:04<00:01, 84.75batch/s]

Training batches on cuda:  83%|████████▎ | 408/492 [00:04<00:01, 83.15batch/s]

Training batches on cuda:  85%|████████▍ | 417/492 [00:04<00:00, 83.96batch/s]

Training batches on cuda:  87%|████████▋ | 426/492 [00:04<00:00, 84.95batch/s]

Training batches on cuda:  89%|████████▊ | 436/492 [00:05<00:00, 85.65batch/s]

Training batches on cuda:  90%|█████████ | 445/492 [00:05<00:00, 84.59batch/s]

Training batches on cuda:  92%|█████████▏| 454/492 [00:05<00:00, 82.82batch/s]

Training batches on cuda:  94%|█████████▍| 463/492 [00:05<00:00, 84.78batch/s]

Training batches on cuda:  96%|█████████▌| 472/492 [00:05<00:00, 86.00batch/s]

Training batches on cuda:  98%|█████████▊| 482/492 [00:05<00:00, 87.26batch/s]

Training batches on cuda: 100%|█████████▉| 491/492 [00:05<00:00, 87.96batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:48<00:17,  6.00s/epoch, loss=0.00338, prev_loss=0.00338]
Training epochs on cuda:  80%|████████  | 8/10 [00:48<00:12,  6.02s/epoch, loss=0.00338, prev_loss=0.00338]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 6/492 [00:00<00:08, 59.19batch/s]

Training batches on cuda:   3%|▎         | 16/492 [00:00<00:07, 66.08batch/s]

Training batches on cuda:   5%|▌         | 26/492 [00:00<00:06, 71.84batch/s]

Training batches on cuda:   7%|▋         | 36/492 [00:00<00:05, 76.92batch/s]

Training batches on cuda:   9%|▉         | 46/492 [00:00<00:05, 81.34batch/s]

Training batches on cuda:  11%|█▏        | 56/492 [00:00<00:05, 85.27batch/s]

Training batches on cuda:  13%|█▎        | 66/492 [00:00<00:04, 87.92batch/s]

Training batches on cuda:  15%|█▌        | 75/492 [00:00<00:04, 85.00batch/s]

Training batches on cuda:  17%|█▋        | 84/492 [00:00<00:04, 84.45batch/s]

Training batches on cuda:  19%|█▉        | 93/492 [00:01<00:04, 85.29batch/s]

Training batches on cuda:  21%|██        | 102/492 [00:01<00:04, 86.18batch/s]

Training batches on cuda:  23%|██▎       | 111/492 [00:01<00:04, 86.92batch/s]

Training batches on cuda:  24%|██▍       | 120/492 [00:01<00:04, 87.57batch/s]

Training batches on cuda:  26%|██▌       | 129/492 [00:01<00:04, 87.43batch/s]

Training batches on cuda:  28%|██▊       | 138/492 [00:01<00:04, 85.61batch/s]

Training batches on cuda:  30%|██▉       | 147/492 [00:01<00:04, 82.39batch/s]

Training batches on cuda:  32%|███▏      | 157/492 [00:01<00:03, 84.44batch/s]

Training batches on cuda:  34%|███▎      | 166/492 [00:01<00:03, 84.05batch/s]

Training batches on cuda:  36%|███▌      | 175/492 [00:02<00:03, 85.74batch/s]

Training batches on cuda:  38%|███▊      | 185/492 [00:02<00:03, 87.88batch/s]

Training batches on cuda:  39%|███▉      | 194/492 [00:02<00:03, 88.23batch/s]

Training batches on cuda:  41%|████▏     | 203/492 [00:02<00:03, 86.55batch/s]

Training batches on cuda:  43%|████▎     | 212/492 [00:02<00:03, 85.67batch/s]

Training batches on cuda:  45%|████▌     | 222/492 [00:02<00:03, 88.05batch/s]

Training batches on cuda:  47%|████▋     | 232/492 [00:02<00:02, 89.31batch/s]

Training batches on cuda:  49%|████▉     | 242/492 [00:02<00:02, 90.04batch/s]

Training batches on cuda:  51%|█████     | 252/492 [00:02<00:02, 89.07batch/s]

Training batches on cuda:  53%|█████▎    | 262/492 [00:03<00:02, 89.39batch/s]

Training batches on cuda:  55%|█████▌    | 272/492 [00:03<00:02, 89.98batch/s]

Training batches on cuda:  57%|█████▋    | 282/492 [00:03<00:02, 89.88batch/s]

Training batches on cuda:  59%|█████▉    | 292/492 [00:03<00:02, 90.20batch/s]

Training batches on cuda:  61%|██████▏   | 302/492 [00:03<00:02, 89.98batch/s]

Training batches on cuda:  63%|██████▎   | 312/492 [00:03<00:02, 88.05batch/s]

Training batches on cuda:  65%|██████▌   | 321/492 [00:03<00:01, 88.59batch/s]

Training batches on cuda:  67%|██████▋   | 330/492 [00:03<00:01, 87.67batch/s]

Training batches on cuda:  69%|██████▉   | 340/492 [00:03<00:01, 88.42batch/s]

Training batches on cuda:  71%|███████   | 349/492 [00:03<00:01, 87.86batch/s]

Training batches on cuda:  73%|███████▎  | 359/492 [00:04<00:01, 90.08batch/s]

Training batches on cuda:  75%|███████▌  | 369/492 [00:04<00:01, 92.12batch/s]

Training batches on cuda:  77%|███████▋  | 379/492 [00:04<00:01, 93.10batch/s]

Training batches on cuda:  79%|███████▉  | 389/492 [00:04<00:01, 94.23batch/s]

Training batches on cuda:  81%|████████  | 399/492 [00:04<00:01, 91.90batch/s]

Training batches on cuda:  83%|████████▎ | 409/492 [00:04<00:00, 91.85batch/s]

Training batches on cuda:  85%|████████▌ | 419/492 [00:04<00:00, 91.49batch/s]

Training batches on cuda:  87%|████████▋ | 429/492 [00:04<00:00, 90.04batch/s]

Training batches on cuda:  89%|████████▉ | 439/492 [00:04<00:00, 90.26batch/s]

Training batches on cuda:  91%|█████████▏| 449/492 [00:05<00:00, 90.43batch/s]

Training batches on cuda:  93%|█████████▎| 459/492 [00:05<00:00, 89.62batch/s]

Training batches on cuda:  95%|█████████▌| 468/492 [00:05<00:00, 88.38batch/s]

Training batches on cuda:  97%|█████████▋| 478/492 [00:05<00:00, 89.22batch/s]

Training batches on cuda:  99%|█████████▉| 487/492 [00:05<00:00, 89.29batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:53<00:12,  6.02s/epoch, loss=0.00338, prev_loss=0.00338]
Training epochs on cuda:  90%|█████████ | 9/10 [00:53<00:05,  5.94s/epoch, loss=0.00338, prev_loss=0.00338]

Training batches on cuda:   0%|          | 0/492 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 3/492 [00:00<00:16, 28.81batch/s]

Training batches on cuda:   3%|▎         | 13/492 [00:00<00:13, 36.25batch/s]

Training batches on cuda:   5%|▍         | 23/492 [00:00<00:10, 44.29batch/s]

Training batches on cuda:   7%|▋         | 33/492 [00:00<00:08, 52.74batch/s]

Training batches on cuda:   9%|▊         | 43/492 [00:00<00:07, 60.63batch/s]

Training batches on cuda:  11%|█         | 52/492 [00:00<00:06, 67.18batch/s]

Training batches on cuda:  13%|█▎        | 62/492 [00:00<00:05, 72.91batch/s]

Training batches on cuda:  14%|█▍        | 71/492 [00:00<00:05, 77.19batch/s]

Training batches on cuda:  16%|█▋        | 81/492 [00:00<00:05, 81.63batch/s]

Training batches on cuda:  18%|█▊        | 90/492 [00:01<00:04, 81.55batch/s]

Training batches on cuda:  20%|██        | 99/492 [00:01<00:04, 81.49batch/s]

Training batches on cuda:  22%|██▏       | 108/492 [00:01<00:04, 79.33batch/s]

Training batches on cuda:  24%|██▍       | 117/492 [00:01<00:04, 80.51batch/s]

Training batches on cuda:  26%|██▌       | 127/492 [00:01<00:04, 82.93batch/s]

Training batches on cuda:  28%|██▊       | 136/492 [00:01<00:04, 82.38batch/s]

Training batches on cuda:  29%|██▉       | 145/492 [00:01<00:04, 83.93batch/s]

Training batches on cuda:  32%|███▏      | 155/492 [00:01<00:03, 86.43batch/s]

Training batches on cuda:  33%|███▎      | 164/492 [00:01<00:03, 87.25batch/s]

Training batches on cuda:  35%|███▌      | 174/492 [00:02<00:03, 89.22batch/s]

Training batches on cuda:  37%|███▋      | 184/492 [00:02<00:03, 89.87batch/s]

Training batches on cuda:  39%|███▉      | 194/492 [00:02<00:03, 88.43batch/s]

Training batches on cuda:  41%|████▏     | 203/492 [00:02<00:03, 88.70batch/s]

Training batches on cuda:  43%|████▎     | 213/492 [00:02<00:03, 89.29batch/s]

Training batches on cuda:  45%|████▌     | 222/492 [00:02<00:03, 87.46batch/s]

Training batches on cuda:  47%|████▋     | 231/492 [00:02<00:02, 87.17batch/s]

Training batches on cuda:  49%|████▉     | 240/492 [00:02<00:02, 87.02batch/s]

Training batches on cuda:  51%|█████     | 249/492 [00:02<00:02, 86.88batch/s]

Training batches on cuda:  52%|█████▏    | 258/492 [00:03<00:02, 87.28batch/s]

Training batches on cuda:  54%|█████▍    | 267/492 [00:03<00:02, 86.70batch/s]

Training batches on cuda:  56%|█████▌    | 276/492 [00:03<00:02, 86.87batch/s]

Training batches on cuda:  58%|█████▊    | 285/492 [00:03<00:02, 86.84batch/s]

Training batches on cuda:  60%|█████▉    | 294/492 [00:03<00:02, 87.16batch/s]

Training batches on cuda:  62%|██████▏   | 303/492 [00:03<00:02, 87.21batch/s]

Training batches on cuda:  63%|██████▎   | 312/492 [00:03<00:02, 87.38batch/s]

Training batches on cuda:  65%|██████▌   | 321/492 [00:03<00:01, 87.40batch/s]

Training batches on cuda:  67%|██████▋   | 330/492 [00:03<00:01, 88.12batch/s]

Training batches on cuda:  69%|██████▉   | 339/492 [00:03<00:01, 86.92batch/s]

Training batches on cuda:  71%|███████   | 348/492 [00:04<00:01, 86.44batch/s]

Training batches on cuda:  73%|███████▎  | 357/492 [00:04<00:01, 85.87batch/s]

Training batches on cuda:  74%|███████▍  | 366/492 [00:04<00:01, 84.95batch/s]

Training batches on cuda:  76%|███████▌  | 375/492 [00:04<00:01, 84.43batch/s]

Training batches on cuda:  78%|███████▊  | 384/492 [00:04<00:01, 82.47batch/s]

Training batches on cuda:  80%|███████▉  | 393/492 [00:04<00:01, 81.65batch/s]

Training batches on cuda:  82%|████████▏ | 402/492 [00:04<00:01, 83.35batch/s]

Training batches on cuda:  84%|████████▎ | 412/492 [00:04<00:00, 85.69batch/s]

Training batches on cuda:  86%|████████▌ | 421/492 [00:04<00:00, 84.65batch/s]

Training batches on cuda:  87%|████████▋ | 430/492 [00:05<00:00, 84.56batch/s]

Training batches on cuda:  89%|████████▉ | 439/492 [00:05<00:00, 83.04batch/s]

Training batches on cuda:  91%|█████████ | 448/492 [00:05<00:00, 83.17batch/s]

Training batches on cuda:  93%|█████████▎| 457/492 [00:05<00:00, 84.30batch/s]

Training batches on cuda:  95%|█████████▍| 466/492 [00:05<00:00, 84.69batch/s]

Training batches on cuda:  97%|█████████▋| 475/492 [00:05<00:00, 81.19batch/s]

Training batches on cuda:  98%|█████████▊| 484/492 [00:05<00:00, 82.83batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:59<00:05,  5.94s/epoch, loss=0.00338, prev_loss=0.00338]2021-09-16:19:17:56,600 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:17:58,0 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:18:12,966 INFO     [evaluator.py:657] Evaluation took 14.97s seconds
2021-09-16:19:18:13,117 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00, 10.90s/epoch, loss=0.00338, prev_loss=0.00338]
Training epochs on cuda: 100%|██████████| 10/10 [01:16<00:00,  7.63s/epoch, loss=0.00338, prev_loss=0.00338]
2021-09-16:19:18:13,118 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmppc1h_qva'
2021-09-16:19:18:13,180 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmppc1h_qva' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.28ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.37ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.49ktriple/s]
Evaluating on cuda:   5%|▌         | 2.05k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   7%|▋         | 2.56k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   8%|▊         | 3.07k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:   9%|▉         | 3.58k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  11%|█         | 4.10k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.62ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.62ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.63ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:18:28,88 INFO     [evaluator.py:657] Evaluation took 14.89s seconds
[I 2021-09-16 19:18:28,093] Trial 75 finished with value: -0.0002545656388568407 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'softplus', 'regularizer.weight': 0.08142485533563946, 'optimizer.lr': 0.03878810112739917, 'negative_sampler.num_negs_per_pos': 75, 'training.batch_size': 632}. Best is trial 47 with value: 0.9731190935614269.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:18:28,117 WARNING  [api.py:823] No random seed is specified. Setting to 2898650025.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 43.61batch/s]

Training batches on cuda:   7%|▋         | 11/161 [00:00<00:03, 47.25batch/s]

Training batches on cuda:  11%|█         | 18/161 [00:00<00:02, 50.54batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:02, 53.30batch/s]

Training batches on cuda:  20%|█▉        | 32/161 [00:00<00:02, 55.35batch/s]

Training batches on cuda:  24%|██▍       | 39/161 [00:00<00:02, 57.35batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:01, 59.80batch/s]

Training batches on cuda:  33%|███▎      | 53/161 [00:00<00:01, 61.56batch/s]

Training batches on cuda:  37%|███▋      | 60/161 [00:00<00:01, 62.50batch/s]

Training batches on cuda:  42%|████▏     | 67/161 [00:01<00:01, 63.58batch/s]

Training batches on cuda:  46%|████▌     | 74/161 [00:01<00:01, 64.08batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:01<00:01, 64.91batch/s]

Training batches on cuda:  55%|█████▍    | 88/161 [00:01<00:01, 65.77batch/s]

Training batches on cuda:  60%|█████▉    | 96/161 [00:01<00:00, 67.02batch/s]

Training batches on cuda:  64%|██████▍   | 103/161 [00:01<00:00, 67.39batch/s]

Training batches on cuda:  68%|██████▊   | 110/161 [00:01<00:00, 67.95batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:01<00:00, 68.05batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:01<00:00, 68.40batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:02<00:00, 68.28batch/s]

Training batches on cuda:  86%|████████▌ | 138/161 [00:02<00:00, 68.25batch/s]

Training batches on cuda:  90%|█████████ | 145/161 [00:02<00:00, 66.17batch/s]

Training batches on cuda:  94%|█████████▍| 152/161 [00:02<00:00, 63.70batch/s]

Training batches on cuda:  99%|█████████▉| 159/161 [00:02<00:00, 62.36batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000517, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:24,  2.73s/epoch, loss=0.000517, prev_loss=nan]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/161 [00:00<00:04, 37.84batch/s]

Training batches on cuda:   6%|▌         | 10/161 [00:00<00:03, 42.32batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:03, 45.91batch/s]

Training batches on cuda:  14%|█▎        | 22/161 [00:00<00:02, 49.06batch/s]

Training batches on cuda:  17%|█▋        | 28/161 [00:00<00:02, 51.75batch/s]

Training batches on cuda:  21%|██        | 34/161 [00:00<00:02, 53.77batch/s]

Training batches on cuda:  25%|██▍       | 40/161 [00:00<00:02, 55.12batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:02, 56.32batch/s]

Training batches on cuda:  32%|███▏      | 52/161 [00:00<00:01, 57.19batch/s]

Training batches on cuda:  36%|███▌      | 58/161 [00:01<00:01, 57.72batch/s]

Training batches on cuda:  40%|███▉      | 64/161 [00:01<00:01, 58.06batch/s]

Training batches on cuda:  44%|████▍     | 71/161 [00:01<00:01, 58.57batch/s]

Training batches on cuda:  48%|████▊     | 78/161 [00:01<00:01, 59.09batch/s]

Training batches on cuda:  53%|█████▎    | 85/161 [00:01<00:01, 59.71batch/s]

Training batches on cuda:  57%|█████▋    | 92/161 [00:01<00:01, 60.41batch/s]

Training batches on cuda:  61%|██████▏   | 99/161 [00:01<00:01, 60.23batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 59.71batch/s]

Training batches on cuda:  70%|██████▉   | 112/161 [00:01<00:00, 59.61batch/s]

Training batches on cuda:  73%|███████▎  | 118/161 [00:02<00:00, 59.14batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:02<00:00, 58.71batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:02<00:00, 59.38batch/s]

Training batches on cuda:  86%|████████▌ | 138/161 [00:02<00:00, 59.62batch/s]

Training batches on cuda:  90%|█████████ | 145/161 [00:02<00:00, 59.82batch/s]

Training batches on cuda:  94%|█████████▍| 151/161 [00:02<00:00, 59.73batch/s]

Training batches on cuda:  98%|█████████▊| 158/161 [00:02<00:00, 59.93batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:24,  2.73s/epoch, loss=0.000505, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:22,  2.81s/epoch, loss=0.000505, prev_loss=nan]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 46.11batch/s]

Training batches on cuda:   7%|▋         | 12/161 [00:00<00:02, 49.84batch/s]

Training batches on cuda:  12%|█▏        | 19/161 [00:00<00:02, 53.68batch/s]

Training batches on cuda:  16%|█▌        | 26/161 [00:00<00:02, 57.13batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:02, 59.57batch/s]

Training batches on cuda:  25%|██▍       | 40/161 [00:00<00:01, 61.24batch/s]

Training batches on cuda:  29%|██▉       | 47/161 [00:00<00:01, 62.54batch/s]

Training batches on cuda:  34%|███▎      | 54/161 [00:00<00:01, 63.56batch/s]

Training batches on cuda:  38%|███▊      | 61/161 [00:00<00:01, 63.98batch/s]

Training batches on cuda:  42%|████▏     | 68/161 [00:01<00:01, 63.03batch/s]

Training batches on cuda:  47%|████▋     | 75/161 [00:01<00:01, 62.11batch/s]

Training batches on cuda:  51%|█████     | 82/161 [00:01<00:01, 61.98batch/s]

Training batches on cuda:  55%|█████▌    | 89/161 [00:01<00:01, 61.95batch/s]

Training batches on cuda:  60%|█████▉    | 96/161 [00:01<00:01, 61.93batch/s]

Training batches on cuda:  64%|██████▍   | 103/161 [00:01<00:00, 60.38batch/s]

Training batches on cuda:  68%|██████▊   | 110/161 [00:01<00:00, 60.70batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:01<00:00, 60.85batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:02<00:00, 60.18batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:02<00:00, 59.46batch/s]

Training batches on cuda:  85%|████████▌ | 137/161 [00:02<00:00, 58.40batch/s]

Training batches on cuda:  89%|████████▉ | 143/161 [00:02<00:00, 52.44batch/s]

Training batches on cuda:  93%|█████████▎| 149/161 [00:02<00:00, 53.78batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:02<00:00, 54.86batch/s]

Training batches on cuda: 100%|██████████| 161/161 [00:02<00:00, 52.65batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:22,  2.81s/epoch, loss=0.000471, prev_loss=0.000505]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:19,  2.85s/epoch, loss=0.000471, prev_loss=0.000505]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/161 [00:00<00:18,  8.82batch/s]

Training batches on cuda:   4%|▍         | 7/161 [00:00<00:13, 11.82batch/s]

Training batches on cuda:   8%|▊         | 13/161 [00:00<00:09, 15.56batch/s]

Training batches on cuda:  12%|█▏        | 19/161 [00:00<00:07, 19.78batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:05, 24.38batch/s]

Training batches on cuda:  19%|█▉        | 31/161 [00:00<00:04, 28.88batch/s]

Training batches on cuda:  23%|██▎       | 37/161 [00:00<00:03, 33.54batch/s]

Training batches on cuda:  27%|██▋       | 43/161 [00:00<00:03, 37.83batch/s]

Training batches on cuda:  30%|███       | 49/161 [00:01<00:02, 41.03batch/s]

Training batches on cuda:  34%|███▍      | 55/161 [00:01<00:02, 43.86batch/s]

Training batches on cuda:  38%|███▊      | 61/161 [00:01<00:02, 46.63batch/s]

Training batches on cuda:  42%|████▏     | 67/161 [00:01<00:01, 47.04batch/s]

Training batches on cuda:  46%|████▌     | 74/161 [00:01<00:01, 51.24batch/s]

Training batches on cuda:  50%|████▉     | 80/161 [00:01<00:01, 53.07batch/s]

Training batches on cuda:  53%|█████▎    | 86/161 [00:01<00:01, 54.51batch/s]

Training batches on cuda:  58%|█████▊    | 93/161 [00:01<00:01, 55.65batch/s]

Training batches on cuda:  62%|██████▏   | 100/161 [00:01<00:01, 57.33batch/s]

Training batches on cuda:  66%|██████▋   | 107/161 [00:02<00:00, 58.59batch/s]

Training batches on cuda:  70%|███████   | 113/161 [00:02<00:00, 52.96batch/s]

Training batches on cuda:  74%|███████▍  | 119/161 [00:02<00:00, 53.90batch/s]

Training batches on cuda:  78%|███████▊  | 125/161 [00:02<00:00, 54.96batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:02<00:00, 56.05batch/s]

Training batches on cuda:  85%|████████▌ | 137/161 [00:02<00:00, 57.14batch/s]

Training batches on cuda:  89%|████████▉ | 143/161 [00:02<00:00, 57.09batch/s]

Training batches on cuda:  93%|█████████▎| 149/161 [00:02<00:00, 53.51batch/s]

Training batches on cuda:  97%|█████████▋| 156/161 [00:02<00:00, 55.10batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:19,  2.85s/epoch, loss=0.000453, prev_loss=0.000471]
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:17,  2.96s/epoch, loss=0.000453, prev_loss=0.000471]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/161 [00:00<00:05, 27.99batch/s]

Training batches on cuda:   6%|▌         | 9/161 [00:00<00:04, 32.47batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:03, 36.93batch/s]

Training batches on cuda:  12%|█▏        | 20/161 [00:00<00:03, 39.24batch/s]

Training batches on cuda:  16%|█▌        | 26/161 [00:00<00:03, 43.21batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:02, 47.54batch/s]

Training batches on cuda:  24%|██▍       | 39/161 [00:00<00:02, 50.38batch/s]

Training batches on cuda:  28%|██▊       | 45/161 [00:00<00:02, 52.89batch/s]

Training batches on cuda:  32%|███▏      | 51/161 [00:00<00:02, 54.12batch/s]

Training batches on cuda:  35%|███▌      | 57/161 [00:01<00:01, 55.42batch/s]

Training batches on cuda:  39%|███▉      | 63/161 [00:01<00:01, 49.82batch/s]

Training batches on cuda:  43%|████▎     | 69/161 [00:01<00:01, 51.72batch/s]

Training batches on cuda:  47%|████▋     | 75/161 [00:01<00:01, 53.33batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:01<00:01, 55.11batch/s]

Training batches on cuda:  54%|█████▍    | 87/161 [00:01<00:01, 56.17batch/s]

Training batches on cuda:  58%|█████▊    | 93/161 [00:01<00:01, 56.48batch/s]

Training batches on cuda:  61%|██████▏   | 99/161 [00:01<00:01, 54.90batch/s]

Training batches on cuda:  65%|██████▌   | 105/161 [00:01<00:01, 53.78batch/s]

Training batches on cuda:  69%|██████▉   | 111/161 [00:02<00:00, 54.33batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:02<00:00, 54.20batch/s]

Training batches on cuda:  76%|███████▋  | 123/161 [00:02<00:00, 54.03batch/s]

Training batches on cuda:  81%|████████  | 130/161 [00:02<00:00, 56.06batch/s]

Training batches on cuda:  85%|████████▌ | 137/161 [00:02<00:00, 57.56batch/s]

Training batches on cuda:  89%|████████▉ | 144/161 [00:02<00:00, 58.54batch/s]

Training batches on cuda:  94%|█████████▍| 151/161 [00:02<00:00, 59.29batch/s]

Training batches on cuda:  98%|█████████▊| 157/161 [00:02<00:00, 58.02batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:17,  2.96s/epoch, loss=0.000436, prev_loss=0.000453]
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:15,  3.03s/epoch, loss=0.000436, prev_loss=0.000453]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/161 [00:00<00:07, 19.98batch/s]

Training batches on cuda:   5%|▍         | 8/161 [00:00<00:06, 24.62batch/s]

Training batches on cuda:   9%|▊         | 14/161 [00:00<00:04, 29.55batch/s]

Training batches on cuda:  12%|█▏        | 20/161 [00:00<00:04, 34.19batch/s]

Training batches on cuda:  16%|█▌        | 26/161 [00:00<00:03, 38.35batch/s]

Training batches on cuda:  20%|█▉        | 32/161 [00:00<00:03, 42.53batch/s]

Training batches on cuda:  24%|██▍       | 39/161 [00:00<00:02, 46.96batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:02, 50.37batch/s]

Training batches on cuda:  33%|███▎      | 53/161 [00:00<00:02, 53.11batch/s]

Training batches on cuda:  37%|███▋      | 60/161 [00:01<00:01, 54.65batch/s]

Training batches on cuda:  41%|████      | 66/161 [00:01<00:01, 53.78batch/s]

Training batches on cuda:  45%|████▍     | 72/161 [00:01<00:01, 51.77batch/s]

Training batches on cuda:  48%|████▊     | 78/161 [00:01<00:01, 53.52batch/s]

Training batches on cuda:  53%|█████▎    | 85/161 [00:01<00:01, 55.79batch/s]

Training batches on cuda:  57%|█████▋    | 92/161 [00:01<00:01, 58.81batch/s]

Training batches on cuda:  61%|██████▏   | 99/161 [00:01<00:01, 60.09batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 59.57batch/s]

Training batches on cuda:  70%|███████   | 113/161 [00:02<00:00, 59.79batch/s]

Training batches on cuda:  75%|███████▍  | 120/161 [00:02<00:00, 57.78batch/s]

Training batches on cuda:  78%|███████▊  | 126/161 [00:02<00:00, 55.10batch/s]

Training batches on cuda:  82%|████████▏ | 132/161 [00:02<00:00, 56.04batch/s]

Training batches on cuda:  86%|████████▌ | 138/161 [00:02<00:00, 53.84batch/s]

Training batches on cuda:  89%|████████▉ | 144/161 [00:02<00:00, 52.83batch/s]

Training batches on cuda:  93%|█████████▎| 150/161 [00:02<00:00, 52.42batch/s]

Training batches on cuda:  98%|█████████▊| 157/161 [00:02<00:00, 54.67batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:15,  3.03s/epoch, loss=0.000372, prev_loss=0.000436]
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:12,  3.08s/epoch, loss=0.000372, prev_loss=0.000436]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/161 [00:00<00:08, 18.90batch/s]

Training batches on cuda:   6%|▌         | 9/161 [00:00<00:06, 23.99batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:04, 29.71batch/s]

Training batches on cuda:  14%|█▍        | 23/161 [00:00<00:03, 35.19batch/s]

Training batches on cuda:  18%|█▊        | 29/161 [00:00<00:03, 39.35batch/s]

Training batches on cuda:  22%|██▏       | 35/161 [00:00<00:02, 43.01batch/s]

Training batches on cuda:  25%|██▌       | 41/161 [00:00<00:02, 45.43batch/s]

Training batches on cuda:  29%|██▉       | 47/161 [00:00<00:02, 48.60batch/s]

Training batches on cuda:  33%|███▎      | 53/161 [00:00<00:02, 51.15batch/s]

Training batches on cuda:  37%|███▋      | 59/161 [00:01<00:01, 53.18batch/s]

Training batches on cuda:  40%|████      | 65/161 [00:01<00:01, 50.91batch/s]

Training batches on cuda:  45%|████▍     | 72/161 [00:01<00:01, 53.53batch/s]

Training batches on cuda:  48%|████▊     | 78/161 [00:01<00:01, 55.32batch/s]

Training batches on cuda:  53%|█████▎    | 85/161 [00:01<00:01, 57.09batch/s]

Training batches on cuda:  57%|█████▋    | 92/161 [00:01<00:01, 57.97batch/s]

Training batches on cuda:  61%|██████    | 98/161 [00:01<00:01, 58.11batch/s]

Training batches on cuda:  65%|██████▌   | 105/161 [00:01<00:00, 58.69batch/s]

Training batches on cuda:  70%|██████▉   | 112/161 [00:01<00:00, 58.77batch/s]

Training batches on cuda:  73%|███████▎  | 118/161 [00:02<00:00, 58.77batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:02<00:00, 58.56batch/s]

Training batches on cuda:  81%|████████  | 130/161 [00:02<00:00, 58.60batch/s]

Training batches on cuda:  84%|████████▍ | 136/161 [00:02<00:00, 58.18batch/s]

Training batches on cuda:  88%|████████▊ | 142/161 [00:02<00:00, 58.34batch/s]

Training batches on cuda:  92%|█████████▏| 148/161 [00:02<00:00, 57.40batch/s]

Training batches on cuda:  96%|█████████▌| 154/161 [00:02<00:00, 54.28batch/s]

Training batches on cuda:  99%|█████████▉| 160/161 [00:02<00:00, 54.62batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:21<00:12,  3.08s/epoch, loss=0.000199, prev_loss=0.000372]
Training epochs on cuda:  70%|███████   | 7/10 [00:21<00:09,  3.08s/epoch, loss=0.000199, prev_loss=0.000372]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 49.29batch/s]

Training batches on cuda:   7%|▋         | 12/161 [00:00<00:02, 53.33batch/s]

Training batches on cuda:  12%|█▏        | 19/161 [00:00<00:02, 56.98batch/s]

Training batches on cuda:  16%|█▌        | 26/161 [00:00<00:02, 59.53batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:02, 60.84batch/s]

Training batches on cuda:  25%|██▍       | 40/161 [00:00<00:01, 61.13batch/s]

Training batches on cuda:  29%|██▉       | 47/161 [00:00<00:01, 62.08batch/s]

Training batches on cuda:  34%|███▎      | 54/161 [00:00<00:01, 62.51batch/s]

Training batches on cuda:  38%|███▊      | 61/161 [00:00<00:01, 61.09batch/s]

Training batches on cuda:  42%|████▏     | 68/161 [00:01<00:01, 62.10batch/s]

Training batches on cuda:  47%|████▋     | 75/161 [00:01<00:01, 63.31batch/s]

Training batches on cuda:  51%|█████     | 82/161 [00:01<00:01, 60.71batch/s]

Training batches on cuda:  55%|█████▌    | 89/161 [00:01<00:01, 58.66batch/s]

Training batches on cuda:  59%|█████▉    | 95/161 [00:01<00:01, 57.45batch/s]

Training batches on cuda:  63%|██████▎   | 101/161 [00:01<00:01, 56.77batch/s]

Training batches on cuda:  66%|██████▋   | 107/161 [00:01<00:00, 55.83batch/s]

Training batches on cuda:  70%|███████   | 113/161 [00:01<00:00, 55.30batch/s]

Training batches on cuda:  74%|███████▍  | 119/161 [00:01<00:00, 55.02batch/s]

Training batches on cuda:  78%|███████▊  | 125/161 [00:02<00:00, 53.83batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:02<00:00, 51.24batch/s]

Training batches on cuda:  85%|████████▌ | 137/161 [00:02<00:00, 51.43batch/s]

Training batches on cuda:  89%|████████▉ | 143/161 [00:02<00:00, 52.55batch/s]

Training batches on cuda:  93%|█████████▎| 149/161 [00:02<00:00, 52.20batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:02<00:00, 51.94batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:24<00:09,  3.08s/epoch, loss=0.000156, prev_loss=0.000199]
Training epochs on cuda:  80%|████████  | 8/10 [00:24<00:06,  3.06s/epoch, loss=0.000156, prev_loss=0.000199]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/161 [00:00<00:05, 28.76batch/s]

Training batches on cuda:   6%|▌         | 9/161 [00:00<00:04, 34.01batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:03, 39.17batch/s]

Training batches on cuda:  14%|█▍        | 23/161 [00:00<00:03, 44.02batch/s]

Training batches on cuda:  19%|█▊        | 30/161 [00:00<00:02, 47.99batch/s]

Training batches on cuda:  23%|██▎       | 37/161 [00:00<00:02, 51.39batch/s]

Training batches on cuda:  27%|██▋       | 43/161 [00:00<00:02, 51.97batch/s]

Training batches on cuda:  31%|███       | 50/161 [00:00<00:02, 54.49batch/s]

Training batches on cuda:  35%|███▌      | 57/161 [00:01<00:01, 56.56batch/s]

Training batches on cuda:  40%|███▉      | 64/161 [00:01<00:01, 58.00batch/s]

Training batches on cuda:  44%|████▍     | 71/161 [00:01<00:01, 59.19batch/s]

Training batches on cuda:  48%|████▊     | 78/161 [00:01<00:01, 59.87batch/s]

Training batches on cuda:  53%|█████▎    | 85/161 [00:01<00:01, 60.14batch/s]

Training batches on cuda:  57%|█████▋    | 92/161 [00:01<00:01, 60.10batch/s]

Training batches on cuda:  61%|██████▏   | 99/161 [00:01<00:01, 59.90batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 59.78batch/s]

Training batches on cuda:  70%|██████▉   | 112/161 [00:01<00:00, 58.93batch/s]

Training batches on cuda:  73%|███████▎  | 118/161 [00:02<00:00, 58.59batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:02<00:00, 58.74batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:02<00:00, 59.07batch/s]

Training batches on cuda:  86%|████████▌ | 138/161 [00:02<00:00, 59.53batch/s]

Training batches on cuda:  89%|████████▉ | 144/161 [00:02<00:00, 59.67batch/s]

Training batches on cuda:  93%|█████████▎| 150/161 [00:02<00:00, 59.20batch/s]

Training batches on cuda:  98%|█████████▊| 157/161 [00:02<00:00, 59.92batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:27<00:06,  3.06s/epoch, loss=0.000144, prev_loss=0.000156]
Training epochs on cuda:  90%|█████████ | 9/10 [00:27<00:03,  3.04s/epoch, loss=0.000144, prev_loss=0.000156]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/161 [00:00<00:19,  8.13batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:14, 10.82batch/s]

Training batches on cuda:   7%|▋         | 12/161 [00:00<00:10, 14.33batch/s]

Training batches on cuda:  11%|█         | 18/161 [00:00<00:07, 18.54batch/s]

Training batches on cuda:  15%|█▍        | 24/161 [00:00<00:05, 23.20batch/s]

Training batches on cuda:  19%|█▊        | 30/161 [00:00<00:04, 28.43batch/s]

Training batches on cuda:  22%|██▏       | 36/161 [00:00<00:03, 32.90batch/s]

Training batches on cuda:  25%|██▌       | 41/161 [00:00<00:03, 36.49batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:02, 39.50batch/s]

Training batches on cuda:  32%|███▏      | 51/161 [00:01<00:02, 42.15batch/s]

Training batches on cuda:  35%|███▌      | 57/161 [00:01<00:02, 44.57batch/s]

Training batches on cuda:  39%|███▉      | 63/161 [00:01<00:02, 47.09batch/s]

Training batches on cuda:  43%|████▎     | 69/161 [00:01<00:01, 49.02batch/s]

Training batches on cuda:  47%|████▋     | 75/161 [00:01<00:01, 47.58batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:01<00:01, 48.59batch/s]

Training batches on cuda:  54%|█████▍    | 87/161 [00:01<00:01, 49.33batch/s]

Training batches on cuda:  58%|█████▊    | 93/161 [00:01<00:01, 48.99batch/s]

Training batches on cuda:  61%|██████▏   | 99/161 [00:01<00:01, 50.84batch/s]

Training batches on cuda:  65%|██████▌   | 105/161 [00:02<00:01, 52.19batch/s]

Training batches on cuda:  69%|██████▉   | 111/161 [00:02<00:00, 53.09batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:02<00:00, 54.44batch/s]

Training batches on cuda:  76%|███████▋  | 123/161 [00:02<00:00, 54.40batch/s]

Training batches on cuda:  80%|████████  | 129/161 [00:02<00:00, 54.79batch/s]

Training batches on cuda:  84%|████████▍ | 135/161 [00:02<00:00, 55.32batch/s]

Training batches on cuda:  88%|████████▊ | 141/161 [00:02<00:00, 56.25batch/s]

Training batches on cuda:  91%|█████████▏| 147/161 [00:02<00:00, 55.04batch/s]

Training batches on cuda:  95%|█████████▌| 153/161 [00:02<00:00, 55.89batch/s]

Training batches on cuda:  99%|█████████▉| 159/161 [00:03<00:00, 56.15batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:30<00:03,  3.04s/epoch, loss=0.000138, prev_loss=0.000144]2021-09-16:19:18:59,326 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:19:00,419 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:19:15,410 INFO     [evaluator.py:657] Evaluation took 14.99s seconds
2021-09-16:19:19:15,587 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00,  8.03s/epoch, loss=0.000138, prev_loss=0.000144]
Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00,  4.70s/epoch, loss=0.000138, prev_loss=0.000144]
2021-09-16:19:19:15,587 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpdrls3fg5'
2021-09-16:19:19:15,649 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpdrls3fg5' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.63ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.63ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.59ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.60ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.62ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.63ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.63ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.63ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:19:30,525 INFO     [evaluator.py:657] Evaluation took 14.86s seconds
[I 2021-09-16 19:19:30,530] Trial 76 finished with value: 0.9770494761570895 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.012139428495095849, 'optimizer.lr': 0.025297941172500296, 'negative_sampler.num_negs_per_pos': 11, 'training.batch_size': 1932}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:19:30,553 WARNING  [api.py:823] No random seed is specified. Setting to 1817417212.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:05, 18.27batch/s]

Training batches on cuda:   5%|▍         | 5/106 [00:00<00:04, 20.43batch/s]

Training batches on cuda:   8%|▊         | 9/106 [00:00<00:04, 23.29batch/s]

Training batches on cuda:  12%|█▏        | 13/106 [00:00<00:03, 25.88batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:00<00:03, 28.07batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:00<00:02, 30.37batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:00<00:02, 32.46batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:00<00:02, 33.72batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:00<00:02, 35.08batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:01<00:01, 36.25batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:01<00:01, 33.98batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:01<00:01, 34.73batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:01<00:01, 35.49batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:01<00:01, 35.85batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:01<00:01, 36.21batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:01<00:01, 36.42batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:01<00:01, 36.74batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:01<00:01, 36.77batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:02<00:00, 37.12batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:02<00:00, 36.98batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:02<00:00, 37.18batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:02<00:00, 37.17batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:02<00:00, 35.59batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:02<00:00, 35.57batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:02<00:00, 34.31batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:02<00:00, 34.54batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:02<00:00, 34.20batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000348, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:29,  3.25s/epoch, loss=0.000348, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:04, 23.38batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:03, 26.03batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:03, 27.92batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:03, 29.40batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:00<00:02, 30.90batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:00<00:02, 31.86batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:00<00:02, 32.72batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:00<00:02, 32.50batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:02, 33.58batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:01<00:01, 34.22batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:01<00:01, 35.12batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:01<00:01, 35.46batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:01<00:01, 35.55batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:01<00:01, 35.87batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:01<00:01, 35.93batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:01<00:01, 36.10batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:01<00:01, 36.24batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:02<00:00, 35.50batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:02<00:00, 35.10batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:02<00:00, 34.75batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:02<00:00, 35.25batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:02<00:00, 34.52batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:02<00:00, 35.21batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:02<00:00, 35.61batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:02<00:00, 35.80batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:03<00:00, 32.05batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:29,  3.25s/epoch, loss=0.000342, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:26,  3.26s/epoch, loss=0.000342, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:05, 19.12batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:04, 22.11batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:03, 25.00batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:03, 27.47batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:00<00:02, 29.60batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:00<00:02, 31.14batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:00<00:02, 32.46batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:00<00:02, 33.32batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:02, 33.97batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:01<00:01, 34.47batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:01<00:01, 34.80batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:01<00:01, 35.13batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:01<00:01, 35.27batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:01<00:01, 35.00batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:01<00:01, 35.23batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:01<00:01, 35.30batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:01<00:01, 35.31batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:02<00:01, 35.28batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:02<00:00, 35.39batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:02<00:00, 35.43batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:02<00:00, 35.53batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:02<00:00, 35.58batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:02<00:00, 35.55batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:02<00:00, 35.64batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:02<00:00, 35.79batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:02<00:00, 35.82batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:03<00:00, 35.62batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:26,  3.26s/epoch, loss=0.000342, prev_loss=0.000342]
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:22,  3.26s/epoch, loss=0.000342, prev_loss=0.000342]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:06, 16.00batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:05, 18.97batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:04, 22.08batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:03, 24.71batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:00<00:03, 26.99batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:00<00:02, 28.80batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:00<00:02, 30.52batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:00<00:02, 31.72batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:02, 32.83batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:01<00:02, 33.37batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:01<00:01, 34.18batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:01<00:01, 34.89batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:01<00:01, 35.76batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:01<00:01, 36.41batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:01<00:01, 36.69batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:01<00:01, 36.24batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:01<00:01, 34.93batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:02<00:01, 34.36batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:02<00:00, 34.95batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:02<00:00, 35.04batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:02<00:00, 34.76batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:02<00:00, 34.66batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:02<00:00, 34.29batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:02<00:00, 34.91batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:02<00:00, 35.15batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:02<00:00, 35.29batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:03<00:00, 34.84batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:22,  3.26s/epoch, loss=0.000342, prev_loss=0.000342]
Training epochs on cuda:  40%|████      | 4/10 [00:13<00:19,  3.26s/epoch, loss=0.000342, prev_loss=0.000342]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:04, 24.84batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:03, 26.71batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:03, 29.08batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:02, 31.09batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:00<00:02, 32.68batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:00<00:02, 33.82batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:00<00:02, 34.61batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:00<00:02, 35.28batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:01, 35.77batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:01<00:01, 36.09batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:01<00:01, 36.52batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:01<00:01, 36.51batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:01<00:01, 36.63batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:01<00:01, 36.63batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:01<00:01, 36.63batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:01<00:01, 36.76batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:01<00:01, 36.79batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:01<00:00, 37.14batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:02<00:00, 37.21batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:02<00:00, 37.14batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:02<00:00, 37.10batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:02<00:00, 37.14batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:02<00:00, 37.05batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:02<00:00, 37.12batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:02<00:00, 37.09batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:02<00:00, 37.06batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:16<00:19,  3.26s/epoch, loss=0.000342, prev_loss=0.000342]
Training epochs on cuda:  50%|█████     | 5/10 [00:16<00:16,  3.22s/epoch, loss=0.000342, prev_loss=0.000342]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:05, 19.93batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:04, 22.47batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:03, 24.95batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:03, 27.33batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:00<00:03, 29.03batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:00<00:02, 30.83batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:00<00:02, 32.37batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:00<00:02, 33.45batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:02, 34.07batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:01<00:01, 34.76batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:01<00:01, 34.96batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:01<00:01, 35.21batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:01<00:01, 34.45batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:01<00:01, 34.35batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:01<00:01, 34.05batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:01<00:01, 33.54batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:01<00:01, 34.50batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:02<00:01, 34.88batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:02<00:00, 35.26batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:02<00:00, 35.29batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:02<00:00, 35.50batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:02<00:00, 35.66batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:02<00:00, 35.75batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:02<00:00, 35.75batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:02<00:00, 35.93batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:02<00:00, 35.97batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:03<00:00, 35.68batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:19<00:16,  3.22s/epoch, loss=0.000343, prev_loss=0.000342]
Training epochs on cuda:  60%|██████    | 6/10 [00:19<00:12,  3.23s/epoch, loss=0.000343, prev_loss=0.000342]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:05, 17.75batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:04, 20.55batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:04, 23.56batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:03, 26.18batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:00<00:03, 28.70batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:00<00:02, 30.55batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:00<00:02, 32.03batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:00<00:02, 32.96batch/s]

Training batches on cuda:  32%|███▏      | 34/106 [00:01<00:02, 33.91batch/s]

Training batches on cuda:  36%|███▌      | 38/106 [00:01<00:01, 34.49batch/s]

Training batches on cuda:  40%|███▉      | 42/106 [00:01<00:01, 35.15batch/s]

Training batches on cuda:  43%|████▎     | 46/106 [00:01<00:01, 34.76batch/s]

Training batches on cuda:  47%|████▋     | 50/106 [00:01<00:01, 34.87batch/s]

Training batches on cuda:  51%|█████     | 54/106 [00:01<00:01, 34.67batch/s]

Training batches on cuda:  55%|█████▍    | 58/106 [00:01<00:01, 34.82batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:01<00:01, 34.92batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:01<00:01, 34.90batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:02<00:01, 34.86batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:02<00:00, 34.46batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:02<00:00, 34.55batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:02<00:00, 34.60batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:02<00:00, 35.17batch/s]

Training batches on cuda:  85%|████████▍ | 90/106 [00:02<00:00, 35.36batch/s]

Training batches on cuda:  89%|████████▊ | 94/106 [00:02<00:00, 35.44batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:02<00:00, 35.35batch/s]

Training batches on cuda:  96%|█████████▌| 102/106 [00:02<00:00, 35.41batch/s]

Training batches on cuda: 100%|██████████| 106/106 [00:03<00:00, 33.48batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:22<00:12,  3.23s/epoch, loss=0.000342, prev_loss=0.000343]
Training epochs on cuda:  70%|███████   | 7/10 [00:22<00:09,  3.25s/epoch, loss=0.000342, prev_loss=0.000343]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:05, 18.34batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:04, 21.33batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:03, 24.04batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:03, 26.84batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:00<00:03, 29.03batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:00<00:02, 29.15batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:00<00:02, 30.38batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:00<00:02, 31.83batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:02, 32.67batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:01<00:02, 33.57batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:01<00:01, 34.45batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:01<00:01, 34.84batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:01<00:01, 35.17batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:01<00:01, 35.46batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:01<00:01, 35.22batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:01<00:01, 34.51batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:01<00:01, 35.03batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:02<00:01, 35.25batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:02<00:00, 35.72batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:02<00:00, 36.07batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:02<00:00, 36.28batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:02<00:00, 36.33batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:02<00:00, 36.38batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:02<00:00, 35.63batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:02<00:00, 35.19batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:02<00:00, 34.71batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:03<00:00, 34.73batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:26<00:09,  3.25s/epoch, loss=0.000342, prev_loss=0.000342]
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:06,  3.26s/epoch, loss=0.000342, prev_loss=0.000342]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:04, 24.88batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:03, 27.33batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:03, 28.54batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:03, 29.78batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:00<00:02, 31.14batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:00<00:02, 31.15batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:00<00:02, 31.90batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:00<00:02, 32.49batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:02, 32.94batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:01<00:01, 33.59batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:01<00:01, 33.58batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:01<00:01, 33.98batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:01<00:01, 34.51batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:01<00:01, 35.13batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:01<00:01, 34.86batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:01<00:01, 34.58batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:01<00:01, 35.11batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:02<00:00, 35.25batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:02<00:00, 35.32batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:02<00:00, 35.55batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:02<00:00, 35.46batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:02<00:00, 35.27batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:02<00:00, 35.20batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:02<00:00, 35.00batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:02<00:00, 32.88batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:03<00:00, 33.31batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:29<00:06,  3.26s/epoch, loss=0.00034, prev_loss=0.000342] 
Training epochs on cuda:  90%|█████████ | 9/10 [00:29<00:03,  3.28s/epoch, loss=0.00034, prev_loss=0.000342]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:04, 23.42batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:03, 26.18batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:03, 28.36batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:03, 29.94batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:00<00:02, 30.93batch/s]

Training batches on cuda:  22%|██▏       | 23/106 [00:00<00:02, 32.19batch/s]

Training batches on cuda:  25%|██▌       | 27/106 [00:00<00:02, 33.18batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:00<00:02, 33.18batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:02, 33.21batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:01<00:01, 34.11batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:01<00:01, 34.69batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:01<00:01, 34.88batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:01<00:01, 35.04batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:01<00:01, 34.87batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:01<00:01, 34.92batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:01<00:01, 35.06batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:01<00:01, 33.89batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:02<00:01, 34.74batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:02<00:00, 35.73batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:02<00:00, 36.05batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:02<00:00, 36.18batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:02<00:00, 36.70batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:02<00:00, 36.69batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:02<00:00, 37.00batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:02<00:00, 36.87batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:02<00:00, 37.03batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:32<00:03,  3.28s/epoch, loss=0.000338, prev_loss=0.00034]2021-09-16:19:20:03,689 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:20:04,715 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:20:19,718 INFO     [evaluator.py:657] Evaluation took 15.00s seconds
2021-09-16:19:20:19,895 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00,  8.14s/epoch, loss=0.000338, prev_loss=0.00034]
Training epochs on cuda: 100%|██████████| 10/10 [00:48<00:00,  4.88s/epoch, loss=0.000338, prev_loss=0.00034]
2021-09-16:19:20:19,895 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpv3xoesyb'
2021-09-16:19:20:19,939 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpv3xoesyb' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.51ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:07, 2.55ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.58ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.59ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.61ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.62ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.62ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.63ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.63ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.63ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:20:34,822 INFO     [evaluator.py:657] Evaluation took 14.86s seconds
[I 2021-09-16 19:20:34,827] Trial 77 finished with value: 0.1814896633474582 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.13221501555890516, 'optimizer.lr': 0.022996433008569923, 'negative_sampler.num_negs_per_pos': 11, 'training.batch_size': 2932}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:20:34,851 WARNING  [api.py:823] No random seed is specified. Setting to 2622518167.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:08, 17.18batch/s]

Training batches on cuda:   5%|▌         | 8/146 [00:00<00:06, 21.72batch/s]

Training batches on cuda:  10%|▉         | 14/146 [00:00<00:04, 26.85batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:04, 30.91batch/s]

Training batches on cuda:  18%|█▊        | 26/146 [00:00<00:03, 36.85batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:00<00:02, 42.54batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:00<00:02, 47.14batch/s]

Training batches on cuda:  32%|███▏      | 47/146 [00:00<00:01, 51.55batch/s]

Training batches on cuda:  37%|███▋      | 54/146 [00:00<00:01, 54.92batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:01<00:01, 57.93batch/s]

Training batches on cuda:  47%|████▋     | 68/146 [00:01<00:01, 60.22batch/s]

Training batches on cuda:  51%|█████▏    | 75/146 [00:01<00:01, 61.95batch/s]

Training batches on cuda:  56%|█████▌    | 82/146 [00:01<00:01, 62.93batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:01<00:00, 63.05batch/s]

Training batches on cuda:  66%|██████▌   | 96/146 [00:01<00:00, 63.87batch/s]

Training batches on cuda:  71%|███████   | 103/146 [00:01<00:00, 64.51batch/s]

Training batches on cuda:  75%|███████▌  | 110/146 [00:01<00:00, 64.94batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:01<00:00, 65.17batch/s]

Training batches on cuda:  85%|████████▍ | 124/146 [00:02<00:00, 65.40batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:02<00:00, 60.91batch/s]

Training batches on cuda:  95%|█████████▍| 138/146 [00:02<00:00, 57.42batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:02<00:00, 55.65batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000483, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:23,  2.65s/epoch, loss=0.000483, prev_loss=nan]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:04, 35.13batch/s]

Training batches on cuda:   7%|▋         | 10/146 [00:00<00:03, 38.78batch/s]

Training batches on cuda:  11%|█         | 16/146 [00:00<00:03, 42.00batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:02, 44.59batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:02, 46.53batch/s]

Training batches on cuda:  23%|██▎       | 34/146 [00:00<00:02, 47.97batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:00<00:02, 49.32batch/s]

Training batches on cuda:  32%|███▏      | 46/146 [00:00<00:01, 50.33batch/s]

Training batches on cuda:  36%|███▌      | 52/146 [00:01<00:01, 50.87batch/s]

Training batches on cuda:  40%|███▉      | 58/146 [00:01<00:01, 52.21batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:01<00:01, 53.18batch/s]

Training batches on cuda:  48%|████▊     | 70/146 [00:01<00:01, 50.37batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:01<00:01, 51.56batch/s]

Training batches on cuda:  56%|█████▌    | 82/146 [00:01<00:01, 53.21batch/s]

Training batches on cuda:  60%|██████    | 88/146 [00:01<00:01, 54.64batch/s]

Training batches on cuda:  64%|██████▍   | 94/146 [00:01<00:00, 54.79batch/s]

Training batches on cuda:  68%|██████▊   | 100/146 [00:01<00:00, 55.80batch/s]

Training batches on cuda:  73%|███████▎  | 106/146 [00:02<00:00, 55.50batch/s]

Training batches on cuda:  77%|███████▋  | 112/146 [00:02<00:00, 53.23batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:02<00:00, 54.55batch/s]

Training batches on cuda:  85%|████████▍ | 124/146 [00:02<00:00, 55.61batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:02<00:00, 56.22batch/s]

Training batches on cuda:  93%|█████████▎| 136/146 [00:02<00:00, 56.94batch/s]

Training batches on cuda:  97%|█████████▋| 142/146 [00:02<00:00, 57.33batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:23,  2.65s/epoch, loss=0.000471, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:21,  2.73s/epoch, loss=0.000471, prev_loss=nan]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/146 [00:00<00:17,  8.27batch/s]

Training batches on cuda:   5%|▍         | 7/146 [00:00<00:12, 11.11batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:09, 14.67batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:06, 18.79batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:05, 23.33batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:00<00:04, 27.69batch/s]

Training batches on cuda:  25%|██▍       | 36/146 [00:00<00:03, 32.13batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:00<00:02, 35.90batch/s]

Training batches on cuda:  32%|███▏      | 46/146 [00:00<00:02, 39.08batch/s]

Training batches on cuda:  35%|███▍      | 51/146 [00:01<00:02, 41.62batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:01<00:02, 43.41batch/s]

Training batches on cuda:  42%|████▏     | 62/146 [00:01<00:01, 45.50batch/s]

Training batches on cuda:  46%|████▌     | 67/146 [00:01<00:01, 43.11batch/s]

Training batches on cuda:  50%|█████     | 73/146 [00:01<00:01, 44.85batch/s]

Training batches on cuda:  53%|█████▎    | 78/146 [00:01<00:01, 45.61batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:01<00:01, 47.39batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:01<00:01, 47.57batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:01<00:01, 48.75batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:02<00:00, 49.62batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:02<00:00, 44.68batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:02<00:00, 45.54batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:02<00:00, 46.42batch/s]

Training batches on cuda:  84%|████████▍ | 123/146 [00:02<00:00, 44.88batch/s]

Training batches on cuda:  88%|████████▊ | 128/146 [00:02<00:00, 43.55batch/s]

Training batches on cuda:  92%|█████████▏| 134/146 [00:02<00:00, 45.95batch/s]

Training batches on cuda:  96%|█████████▌| 140/146 [00:02<00:00, 47.27batch/s]

Training batches on cuda: 100%|██████████| 146/146 [00:03<00:00, 48.96batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:21,  2.73s/epoch, loss=0.000471, prev_loss=0.000471]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:20,  2.90s/epoch, loss=0.000471, prev_loss=0.000471]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/146 [00:00<00:05, 25.92batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:04, 30.54batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:03, 34.94batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:03, 39.03batch/s]

Training batches on cuda:  18%|█▊        | 27/146 [00:00<00:02, 42.38batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:00<00:02, 45.31batch/s]

Training batches on cuda:  27%|██▋       | 39/146 [00:00<00:02, 48.90batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:00<00:01, 51.02batch/s]

Training batches on cuda:  35%|███▍      | 51/146 [00:00<00:01, 52.05batch/s]

Training batches on cuda:  39%|███▉      | 57/146 [00:01<00:01, 52.66batch/s]

Training batches on cuda:  43%|████▎     | 63/146 [00:01<00:01, 52.51batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:01<00:01, 52.94batch/s]

Training batches on cuda:  51%|█████▏    | 75/146 [00:01<00:01, 53.57batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:01<00:01, 54.08batch/s]

Training batches on cuda:  60%|█████▉    | 87/146 [00:01<00:01, 54.35batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:01<00:00, 54.43batch/s]

Training batches on cuda:  68%|██████▊   | 99/146 [00:01<00:00, 54.07batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:01<00:00, 54.42batch/s]

Training batches on cuda:  77%|███████▋  | 112/146 [00:02<00:00, 56.34batch/s]

Training batches on cuda:  82%|████████▏ | 119/146 [00:02<00:00, 57.78batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:02<00:00, 58.12batch/s]

Training batches on cuda:  90%|█████████ | 132/146 [00:02<00:00, 60.10batch/s]

Training batches on cuda:  95%|█████████▌| 139/146 [00:02<00:00, 61.78batch/s]

Training batches on cuda: 100%|██████████| 146/146 [00:02<00:00, 62.95batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:20,  2.90s/epoch, loss=0.000472, prev_loss=0.000471]
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:17,  2.88s/epoch, loss=0.000472, prev_loss=0.000471]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:03, 45.18batch/s]

Training batches on cuda:   8%|▊         | 12/146 [00:00<00:02, 49.52batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:02, 53.28batch/s]

Training batches on cuda:  18%|█▊        | 26/146 [00:00<00:02, 56.35batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:00<00:01, 58.81batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:00<00:01, 60.82batch/s]

Training batches on cuda:  32%|███▏      | 47/146 [00:00<00:01, 60.13batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:00<00:01, 55.62batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:01<00:01, 56.86batch/s]

Training batches on cuda:  45%|████▌     | 66/146 [00:01<00:01, 51.61batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:01<00:01, 53.69batch/s]

Training batches on cuda:  53%|█████▎    | 78/146 [00:01<00:01, 55.35batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:01<00:01, 56.07batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:01<00:00, 56.91batch/s]

Training batches on cuda:  66%|██████▌   | 96/146 [00:01<00:00, 56.42batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:01<00:00, 56.37batch/s]

Training batches on cuda:  74%|███████▍  | 108/146 [00:01<00:00, 56.26batch/s]

Training batches on cuda:  78%|███████▊  | 114/146 [00:01<00:00, 54.82batch/s]

Training batches on cuda:  82%|████████▏ | 120/146 [00:02<00:00, 55.39batch/s]

Training batches on cuda:  86%|████████▋ | 126/146 [00:02<00:00, 56.31batch/s]

Training batches on cuda:  90%|█████████ | 132/146 [00:02<00:00, 55.83batch/s]

Training batches on cuda:  95%|█████████▍| 138/146 [00:02<00:00, 56.44batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:02<00:00, 56.56batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:17,  2.88s/epoch, loss=0.000472, prev_loss=0.000472]
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:14,  2.85s/epoch, loss=0.000472, prev_loss=0.000472]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:08, 16.83batch/s]

Training batches on cuda:   5%|▌         | 8/146 [00:00<00:06, 21.36batch/s]

Training batches on cuda:  10%|▉         | 14/146 [00:00<00:05, 26.05batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:03, 31.62batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:03, 37.48batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:00<00:02, 43.13batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:00<00:02, 48.35batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:00<00:01, 52.32batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:00<00:01, 55.26batch/s]

Training batches on cuda:  43%|████▎     | 63/146 [00:01<00:01, 57.74batch/s]

Training batches on cuda:  48%|████▊     | 70/146 [00:01<00:01, 59.95batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:01<00:01, 61.08batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:01<00:00, 62.40batch/s]

Training batches on cuda:  62%|██████▏   | 91/146 [00:01<00:00, 62.06batch/s]

Training batches on cuda:  67%|██████▋   | 98/146 [00:01<00:00, 60.82batch/s]

Training batches on cuda:  72%|███████▏  | 105/146 [00:01<00:00, 62.08batch/s]

Training batches on cuda:  77%|███████▋  | 112/146 [00:01<00:00, 63.06batch/s]

Training batches on cuda:  82%|████████▏ | 119/146 [00:01<00:00, 64.30batch/s]

Training batches on cuda:  86%|████████▋ | 126/146 [00:02<00:00, 60.85batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:02<00:00, 62.19batch/s]

Training batches on cuda:  96%|█████████▌| 140/146 [00:02<00:00, 59.00batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:14,  2.85s/epoch, loss=0.000472, prev_loss=0.000472]
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:11,  2.77s/epoch, loss=0.000472, prev_loss=0.000472]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/146 [00:00<00:05, 25.50batch/s]

Training batches on cuda:   6%|▌         | 9/146 [00:00<00:04, 30.04batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:03, 34.42batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:03, 38.37batch/s]

Training batches on cuda:  18%|█▊        | 27/146 [00:00<00:02, 41.71batch/s]

Training batches on cuda:  23%|██▎       | 33/146 [00:00<00:02, 44.28batch/s]

Training batches on cuda:  26%|██▌       | 38/146 [00:00<00:02, 44.74batch/s]

Training batches on cuda:  30%|███       | 44/146 [00:00<00:02, 46.71batch/s]

Training batches on cuda:  34%|███▍      | 50/146 [00:01<00:02, 47.78batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:01<00:01, 49.02batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:01<00:01, 47.88batch/s]

Training batches on cuda:  45%|████▌     | 66/146 [00:01<00:01, 47.45batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:01<00:01, 48.00batch/s]

Training batches on cuda:  53%|█████▎    | 78/146 [00:01<00:01, 48.65batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:01<00:01, 51.11batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:01<00:01, 53.36batch/s]

Training batches on cuda:  66%|██████▌   | 96/146 [00:01<00:00, 55.14batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:02<00:00, 56.46batch/s]

Training batches on cuda:  74%|███████▍  | 108/146 [00:02<00:00, 57.46batch/s]

Training batches on cuda:  78%|███████▊  | 114/146 [00:02<00:00, 58.15batch/s]

Training batches on cuda:  82%|████████▏ | 120/146 [00:02<00:00, 58.46batch/s]

Training batches on cuda:  86%|████████▋ | 126/146 [00:02<00:00, 57.97batch/s]

Training batches on cuda:  90%|█████████ | 132/146 [00:02<00:00, 58.28batch/s]

Training batches on cuda:  95%|█████████▌| 139/146 [00:02<00:00, 58.96batch/s]

Training batches on cuda: 100%|██████████| 146/146 [00:02<00:00, 59.57batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:20<00:11,  2.77s/epoch, loss=0.000472, prev_loss=0.000472]
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:08,  2.84s/epoch, loss=0.000472, prev_loss=0.000472]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:04, 33.43batch/s]

Training batches on cuda:   7%|▋         | 10/146 [00:00<00:03, 37.43batch/s]

Training batches on cuda:  11%|█         | 16/146 [00:00<00:03, 42.10batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:02, 46.12batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:02, 49.21batch/s]

Training batches on cuda:  23%|██▎       | 34/146 [00:00<00:02, 51.96batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:00<00:01, 54.58batch/s]

Training batches on cuda:  33%|███▎      | 48/146 [00:00<00:01, 57.42batch/s]

Training batches on cuda:  38%|███▊      | 55/146 [00:00<00:01, 59.85batch/s]

Training batches on cuda:  42%|████▏     | 62/146 [00:01<00:01, 61.59batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:01<00:01, 62.39batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:01<00:01, 62.86batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:01<00:01, 62.70batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:01<00:00, 61.31batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:01<00:00, 60.45batch/s]

Training batches on cuda:  71%|███████   | 104/146 [00:01<00:00, 60.20batch/s]

Training batches on cuda:  76%|███████▌  | 111/146 [00:01<00:00, 60.98batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:01<00:00, 61.40batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:02<00:00, 61.25batch/s]

Training batches on cuda:  90%|█████████ | 132/146 [00:02<00:00, 62.05batch/s]

Training batches on cuda:  95%|█████████▌| 139/146 [00:02<00:00, 64.24batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:22<00:08,  2.84s/epoch, loss=0.000471, prev_loss=0.000472]
Training epochs on cuda:  80%|████████  | 8/10 [00:22<00:05,  2.77s/epoch, loss=0.000471, prev_loss=0.000472]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:04, 32.08batch/s]

Training batches on cuda:   5%|▌         | 8/146 [00:00<00:04, 33.65batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:03, 39.26batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:02, 43.78batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:02, 47.14batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:00<00:02, 50.95batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:00<00:02, 52.29batch/s]

Training batches on cuda:  32%|███▏      | 47/146 [00:00<00:01, 53.65batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:01<00:01, 52.90batch/s]

Training batches on cuda:  40%|████      | 59/146 [00:01<00:01, 53.71batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:01<00:01, 55.39batch/s]

Training batches on cuda:  49%|████▊     | 71/146 [00:01<00:01, 56.43batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:01<00:01, 57.25batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:01<00:01, 55.15batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:01<00:01, 53.42batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:01<00:00, 55.23batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:01<00:00, 54.73batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:01<00:00, 55.93batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:02<00:00, 55.32batch/s]

Training batches on cuda:  82%|████████▏ | 119/146 [00:02<00:00, 55.71batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:02<00:00, 56.05batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:02<00:00, 56.58batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:02<00:00, 57.11batch/s]

Training batches on cuda:  98%|█████████▊| 143/146 [00:02<00:00, 57.00batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:25<00:05,  2.77s/epoch, loss=0.000471, prev_loss=0.000471]
Training epochs on cuda:  90%|█████████ | 9/10 [00:25<00:02,  2.79s/epoch, loss=0.000471, prev_loss=0.000471]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/146 [00:00<00:14,  9.90batch/s]

Training batches on cuda:   5%|▍         | 7/146 [00:00<00:10, 13.10batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:07, 17.08batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:05, 21.35batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:04, 26.16batch/s]

Training batches on cuda:  21%|██        | 30/146 [00:00<00:03, 30.52batch/s]

Training batches on cuda:  25%|██▍       | 36/146 [00:00<00:03, 34.98batch/s]

Training batches on cuda:  28%|██▊       | 41/146 [00:00<00:02, 37.98batch/s]

Training batches on cuda:  32%|███▏      | 47/146 [00:00<00:02, 40.75batch/s]

Training batches on cuda:  36%|███▌      | 52/146 [00:01<00:02, 42.89batch/s]

Training batches on cuda:  40%|███▉      | 58/146 [00:01<00:01, 45.55batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:01<00:01, 48.78batch/s]

Training batches on cuda:  48%|████▊     | 70/146 [00:01<00:01, 50.70batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:01<00:01, 52.42batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:01<00:01, 54.86batch/s]

Training batches on cuda:  61%|██████    | 89/146 [00:01<00:01, 55.47batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:01<00:00, 54.17batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:01<00:00, 53.76batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:02<00:00, 54.86batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:02<00:00, 54.15batch/s]

Training batches on cuda:  82%|████████▏ | 119/146 [00:02<00:00, 54.98batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:02<00:00, 50.06batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:02<00:00, 51.33batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:02<00:00, 53.22batch/s]

Training batches on cuda:  98%|█████████▊| 143/146 [00:02<00:00, 54.14batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:28<00:02,  2.79s/epoch, loss=0.000471, prev_loss=0.000471]2021-09-16:19:21:03,876 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:21:04,886 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:21:19,877 INFO     [evaluator.py:657] Evaluation took 14.99s seconds
2021-09-16:19:21:20,35 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00,  7.71s/epoch, loss=0.000471, prev_loss=0.000471]
Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00,  4.47s/epoch, loss=0.000471, prev_loss=0.000471]
2021-09-16:19:21:20,36 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpycxe2ljd'
2021-09-16:19:21:20,118 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpycxe2ljd' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.37ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.44ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.48ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.53ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.56ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.58ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.62ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:12, 2.50ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.54ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.57ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:11, 2.59ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.60ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.61ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.61ktriple/s]
2021-09-16:19:21:35,5 INFO     [evaluator.py:657] Evaluation took 14.86s seconds
[I 2021-09-16 19:21:35,009] Trial 78 finished with value: 0.02025523048213207 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.20752631081652118, 'optimizer.lr': 0.015614748211049892, 'negative_sampler.num_negs_per_pos': 9, 'training.batch_size': 2132}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:21:35,32 WARNING  [api.py:823] No random seed is specified. Setting to 3061414326.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/128 [00:00<00:04, 27.34batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:04, 29.94batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:03, 32.21batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:03, 33.79batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:03, 34.98batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:02, 35.96batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:00<00:02, 36.58batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:00<00:02, 37.04batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:00<00:02, 37.49batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 37.67batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 37.79batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 38.08batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:02, 37.74batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:01, 36.54batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:01, 35.81batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:01<00:01, 34.82batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:01<00:01, 34.30batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:01<00:01, 34.72batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:02<00:01, 34.99batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 35.74batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 36.08batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 36.45batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:01, 36.85batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:02<00:00, 36.54batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:02<00:00, 36.17batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:02<00:00, 36.28batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:02<00:00, 34.94batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:03<00:00, 33.59batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 32.93batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 33.63batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 34.86batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 35.74batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000827, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:34,  3.80s/epoch, loss=0.000827, prev_loss=nan]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/128 [00:00<00:04, 25.83batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:04, 28.34batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:03, 30.55batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:03, 32.02batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:03, 33.34batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:03, 34.30batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:00<00:02, 35.05batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:00<00:02, 35.25batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:02, 33.78batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 35.16batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 34.32batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 35.44batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:02, 36.60batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:01, 36.70batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:01, 35.16batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:01<00:01, 34.93batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:01<00:01, 35.50batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:02<00:01, 35.72batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:02<00:01, 36.16batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 36.86batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 37.24batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 37.34batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:00, 37.64batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:02<00:00, 37.70batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:02<00:00, 37.66batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:02<00:00, 38.07batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:02<00:00, 38.13batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:03<00:00, 38.15batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 38.27batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 37.97batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 38.06batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 37.44batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:07<00:34,  3.80s/epoch, loss=0.000824, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:30,  3.77s/epoch, loss=0.000824, prev_loss=nan]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/128 [00:00<00:04, 25.40batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:04, 27.68batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:03, 29.75batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:03, 30.72batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:03, 32.07batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:03, 33.51batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:00<00:02, 34.02batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:00<00:02, 34.55batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:02, 35.71batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 36.55batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 37.48batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 37.21batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:02, 37.94batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:01, 38.16batch/s]

Training batches on cuda:  47%|████▋     | 60/128 [00:01<00:01, 37.92batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:01<00:01, 38.69batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:01<00:01, 39.27batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 39.41batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 39.85batch/s]

Training batches on cuda:  66%|██████▌   | 84/128 [00:02<00:01, 39.88batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:02<00:00, 40.06batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 36.74batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 37.48batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:02<00:00, 38.14batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:02<00:00, 38.55batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:02<00:00, 38.74batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 38.91batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 38.34batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 38.10batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 37.48batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:30,  3.77s/epoch, loss=0.000824, prev_loss=0.000824]
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:26,  3.75s/epoch, loss=0.000824, prev_loss=0.000824]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 16.72batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:06, 20.03batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 23.10batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 26.23batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 28.79batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 30.23batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 31.82batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 32.37batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 33.09batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 33.77batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 33.73batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 34.12batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 34.28batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 34.04batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 34.47batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 34.30batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 34.57batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 35.03batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 34.89batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 34.92batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 34.92batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 34.76batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 34.86batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 34.89batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 34.42batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:02<00:00, 34.32batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 35.83batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 36.93batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 36.17batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 36.91batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 37.64batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 37.55batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:26,  3.75s/epoch, loss=0.000822, prev_loss=0.000824]
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:22,  3.81s/epoch, loss=0.000822, prev_loss=0.000824]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:06, 18.60batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 22.02batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 25.42batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:03, 28.51batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 30.98batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 33.10batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:02, 34.80batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:02, 35.95batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:00<00:02, 36.54batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 37.67batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 38.05batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 38.58batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:01, 38.66batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:01, 38.69batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:01, 38.98batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:01<00:01, 38.83batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:01<00:01, 38.73batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:01<00:01, 38.82batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:01<00:01, 39.08batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 39.23batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 38.84batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 38.86batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:00, 38.71batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:02<00:00, 38.93batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:02<00:00, 39.05batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:02<00:00, 38.94batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:02<00:00, 38.59batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:02<00:00, 38.83batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:02<00:00, 39.00batch/s]

Training batches on cuda:  94%|█████████▍| 120/128 [00:03<00:00, 39.30batch/s]

Training batches on cuda:  97%|█████████▋| 124/128 [00:03<00:00, 39.17batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:18<00:22,  3.81s/epoch, loss=0.000819, prev_loss=0.000822]
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:18,  3.73s/epoch, loss=0.000819, prev_loss=0.000822]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:06, 18.16batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 21.50batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 24.83batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 27.73batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 30.47batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 32.52batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:02, 34.34batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:02, 35.51batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:00<00:02, 36.39batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 36.76batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 37.61batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 37.75batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 37.99batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:01, 38.27batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:01, 38.58batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 38.57batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 38.67batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:01<00:01, 38.71batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:01<00:01, 38.85batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 38.81batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 38.81batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 38.41batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:00, 38.65batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 38.98batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 38.82batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:02<00:00, 38.63batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:02<00:00, 38.50batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:02<00:00, 38.49batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 38.93batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 39.03batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 38.87batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 38.85batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:18,  3.73s/epoch, loss=0.000811, prev_loss=0.000819]
Training epochs on cuda:  60%|██████    | 6/10 [00:22<00:14,  3.70s/epoch, loss=0.000811, prev_loss=0.000819]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/128 [00:00<00:04, 26.26batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:04, 28.92batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:03, 31.33batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:03, 33.16batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:03, 34.83batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:02, 36.09batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:00<00:02, 34.82batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:00<00:02, 35.96batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:00<00:02, 36.86batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 37.28batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 37.75batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 38.15batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:01, 38.59batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:01, 38.96batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:01, 38.95batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:01<00:01, 38.59batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:01<00:01, 38.56batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:01<00:01, 38.72batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:01<00:01, 38.84batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 38.81batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 38.84batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 38.83batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:00, 38.86batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:02<00:00, 38.76batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:02<00:00, 38.79batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:02<00:00, 38.71batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:02<00:00, 38.78batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:02<00:00, 39.05batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 38.87batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 38.92batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 38.76batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 38.68batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:25<00:14,  3.70s/epoch, loss=0.000778, prev_loss=0.000811]
Training epochs on cuda:  70%|███████   | 7/10 [00:25<00:11,  3.68s/epoch, loss=0.000778, prev_loss=0.000811]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:06, 19.31batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 22.60batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 25.56batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 27.99batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 30.35batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 31.74batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 33.15batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:02, 34.74batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:00<00:02, 35.67batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 36.38batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 36.92batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 36.45batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 36.83batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 36.98batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:01, 36.51batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 36.96batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 36.82batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:01<00:01, 36.90batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 37.03batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 34.01batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 34.89batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 35.28batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 35.63batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 36.11batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 36.22batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:02<00:00, 36.60batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:02<00:00, 35.70batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 36.46batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 37.51batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 38.08batch/s]

Training batches on cuda:  97%|█████████▋| 124/128 [00:03<00:00, 38.75batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:29<00:11,  3.68s/epoch, loss=0.000762, prev_loss=0.000778]
Training epochs on cuda:  80%|████████  | 8/10 [00:29<00:07,  3.69s/epoch, loss=0.000762, prev_loss=0.000778]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/128 [00:00<00:05, 24.05batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:04, 26.65batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:04, 29.03batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:03, 31.08batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:03, 32.85batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:03, 34.05batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:00<00:02, 34.88batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:00<00:02, 35.61batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:00<00:02, 36.24batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 36.85batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 36.81batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 36.87batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:02, 36.90batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:01, 37.37batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:01, 37.53batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:01<00:01, 37.22batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:01<00:01, 36.87batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:01<00:01, 36.36batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:02<00:01, 36.47batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 36.20batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 36.20batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 36.36batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:01, 36.47batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:02<00:00, 36.91batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:02<00:00, 36.83batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:02<00:00, 37.05batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:02<00:00, 37.05batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:03<00:00, 36.93batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 36.90batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 37.10batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 37.14batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 36.17batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:33<00:07,  3.69s/epoch, loss=0.000754, prev_loss=0.000762]
Training epochs on cuda:  90%|█████████ | 9/10 [00:33<00:03,  3.70s/epoch, loss=0.000754, prev_loss=0.000762]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 16.24batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:06, 19.59batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 22.64batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 25.20batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 27.89batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 29.86batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 31.74batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:02, 33.08batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 34.28batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 35.18batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 35.49batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 35.63batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 36.19batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 35.60batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:01, 35.91batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 36.17batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 36.00batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:01<00:01, 36.52batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 36.89batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 37.21batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 37.51batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 37.44batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 37.57batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 37.56batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 37.37batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:02<00:00, 38.09batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:02<00:00, 38.88batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:03<00:00, 38.69batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 38.88batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 35.99batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 36.48batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 36.61batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:37<00:03,  3.70s/epoch, loss=0.000732, prev_loss=0.000754]2021-09-16:19:22:12,701 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:22:13,780 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:22:28,721 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:19:22:28,850 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00,  8.57s/epoch, loss=0.000732, prev_loss=0.000754]
Training epochs on cuda: 100%|██████████| 10/10 [00:53<00:00,  5.33s/epoch, loss=0.000732, prev_loss=0.000754]
2021-09-16:19:22:28,851 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpex7zmk75'
2021-09-16:19:22:28,906 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpex7zmk75' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.30ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.39ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.59ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.56ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.59ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.60ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.61ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.62ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.62ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.63ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:19:22:43,780 INFO     [evaluator.py:657] Evaluation took 14.86s seconds
[I 2021-09-16 19:22:43,785] Trial 79 finished with value: 0.38228725627814275 and parameters: {'loss.margin': 2, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.041047177306202845, 'optimizer.lr': 0.018364585854486495, 'negative_sampler.num_negs_per_pos': 16, 'training.batch_size': 2432}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:22:43,807 WARNING  [api.py:823] No random seed is specified. Setting to 2725816775.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 4/100 [00:00<00:02, 32.36batch/s]

Training batches on cuda:   8%|▊         | 8/100 [00:00<00:02, 33.56batch/s]

Training batches on cuda:  12%|█▏        | 12/100 [00:00<00:02, 34.73batch/s]

Training batches on cuda:  16%|█▌        | 16/100 [00:00<00:02, 34.98batch/s]

Training batches on cuda:  20%|██        | 20/100 [00:00<00:02, 35.27batch/s]

Training batches on cuda:  24%|██▍       | 24/100 [00:00<00:02, 34.97batch/s]

Training batches on cuda:  28%|██▊       | 28/100 [00:00<00:02, 35.29batch/s]

Training batches on cuda:  32%|███▏      | 32/100 [00:00<00:01, 35.31batch/s]

Training batches on cuda:  36%|███▌      | 36/100 [00:01<00:01, 35.09batch/s]

Training batches on cuda:  40%|████      | 40/100 [00:01<00:01, 35.07batch/s]

Training batches on cuda:  44%|████▍     | 44/100 [00:01<00:01, 34.94batch/s]

Training batches on cuda:  48%|████▊     | 48/100 [00:01<00:01, 35.07batch/s]

Training batches on cuda:  52%|█████▏    | 52/100 [00:01<00:01, 35.34batch/s]

Training batches on cuda:  56%|█████▌    | 56/100 [00:01<00:01, 35.20batch/s]

Training batches on cuda:  60%|██████    | 60/100 [00:01<00:01, 35.01batch/s]

Training batches on cuda:  64%|██████▍   | 64/100 [00:01<00:01, 35.12batch/s]

Training batches on cuda:  68%|██████▊   | 68/100 [00:01<00:00, 35.01batch/s]

Training batches on cuda:  72%|███████▏  | 72/100 [00:02<00:00, 35.06batch/s]

Training batches on cuda:  76%|███████▌  | 76/100 [00:02<00:00, 34.92batch/s]

Training batches on cuda:  80%|████████  | 80/100 [00:02<00:00, 34.42batch/s]

Training batches on cuda:  84%|████████▍ | 84/100 [00:02<00:00, 34.44batch/s]

Training batches on cuda:  88%|████████▊ | 88/100 [00:02<00:00, 34.89batch/s]

Training batches on cuda:  92%|█████████▏| 92/100 [00:02<00:00, 34.63batch/s]

Training batches on cuda:  96%|█████████▌| 96/100 [00:02<00:00, 34.78batch/s]

                                                                             
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=3.85e-6, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:27,  3.02s/epoch, loss=3.85e-6, prev_loss=nan]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:03, 26.78batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:03, 29.12batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:00<00:02, 31.12batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:00<00:02, 33.09batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:00<00:02, 34.34batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:00<00:02, 35.43batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:00<00:02, 35.24batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:00<00:01, 35.47batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:00<00:01, 35.58batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:01<00:01, 35.68batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:01<00:01, 35.50batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:01<00:01, 35.29batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:01<00:01, 35.46batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:01<00:01, 35.57batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:01<00:01, 35.44batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:01<00:01, 35.01batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:01<00:00, 34.78batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:02<00:00, 34.98batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:02<00:00, 34.38batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:02<00:00, 34.43batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:02<00:00, 34.50batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:02<00:00, 34.47batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:02<00:00, 34.54batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:02<00:00, 34.09batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:02<00:00, 32.27batch/s]

                                                                             
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:27,  3.02s/epoch, loss=5.29e-7, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:24,  3.03s/epoch, loss=5.29e-7, prev_loss=nan]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:03, 25.72batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:03, 27.93batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:00<00:03, 29.58batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:00<00:02, 31.00batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:00<00:02, 31.87batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:00<00:02, 32.49batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:00<00:02, 33.51batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:00<00:02, 34.31batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:01<00:01, 35.03batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:01<00:01, 35.55batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:01<00:01, 35.43batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:01<00:01, 35.26batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:01<00:01, 35.38batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:01<00:01, 35.40batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:01<00:01, 33.37batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:01<00:01, 31.59batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:01<00:01, 32.52batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:02<00:00, 33.19batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:02<00:00, 32.95batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:02<00:00, 34.12batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:02<00:00, 34.76batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:02<00:00, 35.19batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:02<00:00, 35.38batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:02<00:00, 36.03batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:02<00:00, 36.25batch/s]

                                                                             
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:24,  3.03s/epoch, loss=5.22e-7, prev_loss=5.29e-7]
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:21,  3.04s/epoch, loss=5.22e-7, prev_loss=5.29e-7]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:03, 28.76batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:03, 30.93batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:00<00:02, 32.62batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:00<00:02, 34.27batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:00<00:02, 35.45batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:00<00:02, 35.84batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:00<00:02, 36.32batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:00<00:01, 36.98batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:00<00:01, 37.08batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:01<00:01, 36.52batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:01<00:01, 35.95batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:01<00:01, 35.50batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:01<00:01, 35.33batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:01<00:01, 35.46batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:01<00:01, 35.53batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:01<00:01, 35.18batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:01<00:00, 34.73batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:01<00:00, 34.81batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:02<00:00, 34.64batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:02<00:00, 34.68batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:02<00:00, 34.79batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:02<00:00, 34.65batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:02<00:00, 33.24batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:02<00:00, 33.79batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:02<00:00, 33.87batch/s]

                                                                             
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:21,  3.04s/epoch, loss=5.11e-7, prev_loss=5.22e-7]
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:18,  3.04s/epoch, loss=5.11e-7, prev_loss=5.22e-7]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:03, 25.89batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:03, 27.87batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:00<00:03, 29.54batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:00<00:02, 30.74batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:00<00:02, 31.89batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:00<00:02, 32.76batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:00<00:02, 33.16batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:00<00:02, 33.73batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:01<00:01, 34.30batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:01<00:01, 34.17batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:01<00:01, 34.40batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:01<00:01, 34.52batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:01<00:01, 34.54batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:01<00:01, 34.33batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:01<00:01, 34.57batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:01<00:01, 34.42batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:01<00:00, 34.39batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:02<00:00, 34.49batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:02<00:00, 34.20batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:02<00:00, 33.16batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:02<00:00, 33.26batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:02<00:00, 33.74batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:02<00:00, 33.69batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:02<00:00, 31.21batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:02<00:00, 32.14batch/s]

                                                                             
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:18,  3.04s/epoch, loss=5.32e-7, prev_loss=5.11e-7]
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:15,  3.08s/epoch, loss=5.32e-7, prev_loss=5.11e-7]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:04, 22.84batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:03, 25.46batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:00<00:03, 27.47batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:00<00:02, 29.23batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:00<00:02, 30.92batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:00<00:02, 32.27batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:00<00:02, 33.00batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:00<00:02, 33.55batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:01<00:01, 34.10batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:01<00:01, 34.40batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:01<00:01, 34.74batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:01<00:01, 35.04batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:01<00:01, 35.38batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:01<00:01, 35.41batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:01<00:01, 35.49batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:01<00:01, 34.70batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:01<00:00, 34.92batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:02<00:00, 34.98batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:02<00:00, 35.29batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:02<00:00, 35.79batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:02<00:00, 35.98batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:02<00:00, 36.37batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:02<00:00, 36.53batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:02<00:00, 36.55batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:02<00:00, 36.08batch/s]

                                                                             
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:15,  3.08s/epoch, loss=5.27e-7, prev_loss=5.32e-7]
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:12,  3.08s/epoch, loss=5.27e-7, prev_loss=5.32e-7]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:03, 26.03batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:03, 27.91batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:00<00:03, 29.57batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:00<00:02, 30.98batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:00<00:02, 32.18batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:00<00:02, 32.84batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:00<00:02, 33.42batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:00<00:02, 33.56batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:01<00:01, 34.06batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:01<00:01, 34.15batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:01<00:01, 34.60batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:01<00:01, 34.68batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:01<00:01, 34.92batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:01<00:01, 31.94batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:01<00:01, 32.87batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:01<00:01, 33.37batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:01<00:00, 33.76batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:02<00:00, 34.23batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:02<00:00, 34.28batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:02<00:00, 34.64batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:02<00:00, 34.72batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:02<00:00, 34.55batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:02<00:00, 34.87batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:02<00:00, 34.94batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:02<00:00, 35.10batch/s]

                                                                             
Training epochs on cuda:  60%|██████    | 6/10 [00:21<00:12,  3.08s/epoch, loss=4.97e-7, prev_loss=5.27e-7]
Training epochs on cuda:  70%|███████   | 7/10 [00:21<00:09,  3.10s/epoch, loss=4.97e-7, prev_loss=5.27e-7]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/100 [00:00<00:06, 16.23batch/s]

Training batches on cuda:   6%|▌         | 6/100 [00:00<00:04, 19.23batch/s]

Training batches on cuda:  10%|█         | 10/100 [00:00<00:04, 22.18batch/s]

Training batches on cuda:  14%|█▍        | 14/100 [00:00<00:03, 24.99batch/s]

Training batches on cuda:  18%|█▊        | 18/100 [00:00<00:03, 27.23batch/s]

Training batches on cuda:  22%|██▏       | 22/100 [00:00<00:02, 29.37batch/s]

Training batches on cuda:  26%|██▌       | 26/100 [00:00<00:02, 30.98batch/s]

Training batches on cuda:  30%|███       | 30/100 [00:00<00:02, 32.35batch/s]

Training batches on cuda:  34%|███▍      | 34/100 [00:01<00:01, 33.34batch/s]

Training batches on cuda:  38%|███▊      | 38/100 [00:01<00:01, 34.05batch/s]

Training batches on cuda:  42%|████▏     | 42/100 [00:01<00:01, 34.44batch/s]

Training batches on cuda:  46%|████▌     | 46/100 [00:01<00:01, 34.87batch/s]

Training batches on cuda:  50%|█████     | 50/100 [00:01<00:01, 35.10batch/s]

Training batches on cuda:  54%|█████▍    | 54/100 [00:01<00:01, 35.35batch/s]

Training batches on cuda:  58%|█████▊    | 58/100 [00:01<00:01, 35.38batch/s]

Training batches on cuda:  62%|██████▏   | 62/100 [00:01<00:01, 35.78batch/s]

Training batches on cuda:  66%|██████▌   | 66/100 [00:01<00:00, 35.80batch/s]

Training batches on cuda:  70%|███████   | 70/100 [00:02<00:00, 35.84batch/s]

Training batches on cuda:  74%|███████▍  | 74/100 [00:02<00:00, 35.65batch/s]

Training batches on cuda:  78%|███████▊  | 78/100 [00:02<00:00, 35.78batch/s]

Training batches on cuda:  82%|████████▏ | 82/100 [00:02<00:00, 35.82batch/s]

Training batches on cuda:  86%|████████▌ | 86/100 [00:02<00:00, 35.64batch/s]

Training batches on cuda:  90%|█████████ | 90/100 [00:02<00:00, 35.17batch/s]

Training batches on cuda:  94%|█████████▍| 94/100 [00:02<00:00, 35.05batch/s]

Training batches on cuda:  98%|█████████▊| 98/100 [00:02<00:00, 35.08batch/s]

                                                                             
Training epochs on cuda:  70%|███████   | 7/10 [00:24<00:09,  3.10s/epoch, loss=5.65e-7, prev_loss=4.97e-7]
Training epochs on cuda:  80%|████████  | 8/10 [00:24<00:06,  3.09s/epoch, loss=5.65e-7, prev_loss=4.97e-7]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:04, 22.91batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:03, 25.60batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:00<00:03, 27.80batch/s]

Training batches on cuda:  14%|█▍        | 14/100 [00:00<00:03, 28.11batch/s]

Training batches on cuda:  18%|█▊        | 18/100 [00:00<00:02, 30.10batch/s]

Training batches on cuda:  22%|██▏       | 22/100 [00:00<00:02, 31.55batch/s]

Training batches on cuda:  26%|██▌       | 26/100 [00:00<00:02, 32.71batch/s]

Training batches on cuda:  30%|███       | 30/100 [00:00<00:02, 33.53batch/s]

Training batches on cuda:  34%|███▍      | 34/100 [00:01<00:01, 34.16batch/s]

Training batches on cuda:  38%|███▊      | 38/100 [00:01<00:01, 34.54batch/s]

Training batches on cuda:  42%|████▏     | 42/100 [00:01<00:01, 32.65batch/s]

Training batches on cuda:  46%|████▌     | 46/100 [00:01<00:01, 33.71batch/s]

Training batches on cuda:  50%|█████     | 50/100 [00:01<00:01, 34.20batch/s]

Training batches on cuda:  54%|█████▍    | 54/100 [00:01<00:01, 34.45batch/s]

Training batches on cuda:  58%|█████▊    | 58/100 [00:01<00:01, 35.06batch/s]

Training batches on cuda:  62%|██████▏   | 62/100 [00:01<00:01, 36.21batch/s]

Training batches on cuda:  66%|██████▌   | 66/100 [00:01<00:00, 35.17batch/s]

Training batches on cuda:  70%|███████   | 70/100 [00:02<00:00, 35.55batch/s]

Training batches on cuda:  74%|███████▍  | 74/100 [00:02<00:00, 33.47batch/s]

Training batches on cuda:  78%|███████▊  | 78/100 [00:02<00:00, 30.71batch/s]

Training batches on cuda:  82%|████████▏ | 82/100 [00:02<00:00, 31.59batch/s]

Training batches on cuda:  86%|████████▌ | 86/100 [00:02<00:00, 32.88batch/s]

Training batches on cuda:  90%|█████████ | 90/100 [00:02<00:00, 32.37batch/s]

Training batches on cuda:  94%|█████████▍| 94/100 [00:02<00:00, 33.63batch/s]

Training batches on cuda:  98%|█████████▊| 98/100 [00:02<00:00, 33.78batch/s]

                                                                             
Training epochs on cuda:  80%|████████  | 8/10 [00:27<00:06,  3.09s/epoch, loss=5.6e-7, prev_loss=5.65e-7] 
Training epochs on cuda:  90%|█████████ | 9/10 [00:27<00:03,  3.11s/epoch, loss=5.6e-7, prev_loss=5.65e-7]

Training batches on cuda:   0%|          | 0/100 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/100 [00:00<00:03, 28.09batch/s]

Training batches on cuda:   7%|▋         | 7/100 [00:00<00:03, 30.56batch/s]

Training batches on cuda:  11%|█         | 11/100 [00:00<00:02, 32.39batch/s]

Training batches on cuda:  15%|█▌        | 15/100 [00:00<00:02, 33.78batch/s]

Training batches on cuda:  19%|█▉        | 19/100 [00:00<00:02, 34.45batch/s]

Training batches on cuda:  23%|██▎       | 23/100 [00:00<00:02, 34.79batch/s]

Training batches on cuda:  27%|██▋       | 27/100 [00:00<00:02, 34.99batch/s]

Training batches on cuda:  31%|███       | 31/100 [00:00<00:01, 35.40batch/s]

Training batches on cuda:  35%|███▌      | 35/100 [00:00<00:01, 35.45batch/s]

Training batches on cuda:  39%|███▉      | 39/100 [00:01<00:01, 35.37batch/s]

Training batches on cuda:  43%|████▎     | 43/100 [00:01<00:01, 35.02batch/s]

Training batches on cuda:  47%|████▋     | 47/100 [00:01<00:01, 34.90batch/s]

Training batches on cuda:  51%|█████     | 51/100 [00:01<00:01, 34.62batch/s]

Training batches on cuda:  55%|█████▌    | 55/100 [00:01<00:01, 34.12batch/s]

Training batches on cuda:  59%|█████▉    | 59/100 [00:01<00:01, 34.66batch/s]

Training batches on cuda:  63%|██████▎   | 63/100 [00:01<00:01, 34.95batch/s]

Training batches on cuda:  67%|██████▋   | 67/100 [00:01<00:00, 34.81batch/s]

Training batches on cuda:  71%|███████   | 71/100 [00:02<00:00, 34.88batch/s]

Training batches on cuda:  75%|███████▌  | 75/100 [00:02<00:00, 34.90batch/s]

Training batches on cuda:  79%|███████▉  | 79/100 [00:02<00:00, 34.90batch/s]

Training batches on cuda:  83%|████████▎ | 83/100 [00:02<00:00, 32.58batch/s]

Training batches on cuda:  87%|████████▋ | 87/100 [00:02<00:00, 33.13batch/s]

Training batches on cuda:  91%|█████████ | 91/100 [00:02<00:00, 32.92batch/s]

Training batches on cuda:  95%|█████████▌| 95/100 [00:02<00:00, 33.15batch/s]

Training batches on cuda:  99%|█████████▉| 99/100 [00:02<00:00, 32.94batch/s]

                                                                             
Training epochs on cuda:  90%|█████████ | 9/10 [00:30<00:03,  3.11s/epoch, loss=5.76e-7, prev_loss=5.6e-7]2021-09-16:19:23:15,146 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:23:16,215 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:23:31,171 INFO     [evaluator.py:657] Evaluation took 14.96s seconds
2021-09-16:19:23:31,300 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00,  7.95s/epoch, loss=5.76e-7, prev_loss=5.6e-7]
Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00,  4.70s/epoch, loss=5.76e-7, prev_loss=5.6e-7]
2021-09-16:19:23:31,301 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpw9y9gdp1'
2021-09-16:19:23:31,356 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpw9y9gdp1' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:18, 2.03ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:17, 2.18ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:16, 2.30ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.39ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.46ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.51ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.55ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.63ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:19:23:46,190 INFO     [evaluator.py:657] Evaluation took 14.82s seconds
[I 2021-09-16 19:23:46,195] Trial 80 finished with value: 0.00468643770789523 and parameters: {'loss.margin': 0, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.06891572455317907, 'optimizer.lr': 0.02491108446327289, 'negative_sampler.num_negs_per_pos': 9, 'training.batch_size': 3132}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:23:46,217 WARNING  [api.py:823] No random seed is specified. Setting to 1511108613.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:03, 48.22batch/s]

Training batches on cuda:   6%|▋         | 11/170 [00:00<00:03, 49.92batch/s]

Training batches on cuda:  11%|█         | 19/170 [00:00<00:02, 55.64batch/s]

Training batches on cuda:  15%|█▌        | 26/170 [00:00<00:02, 59.29batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:00<00:02, 62.34batch/s]

Training batches on cuda:  25%|██▍       | 42/170 [00:00<00:01, 64.60batch/s]

Training batches on cuda:  29%|██▉       | 50/170 [00:00<00:01, 66.51batch/s]

Training batches on cuda:  34%|███▍      | 58/170 [00:00<00:01, 69.27batch/s]

Training batches on cuda:  39%|███▉      | 66/170 [00:00<00:01, 71.79batch/s]

Training batches on cuda:  44%|████▍     | 75/170 [00:01<00:01, 74.76batch/s]

Training batches on cuda:  49%|████▉     | 84/170 [00:01<00:01, 76.54batch/s]

Training batches on cuda:  55%|█████▍    | 93/170 [00:01<00:00, 78.09batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:01<00:00, 78.27batch/s]

Training batches on cuda:  64%|██████▍   | 109/170 [00:01<00:00, 76.88batch/s]

Training batches on cuda:  69%|██████▉   | 118/170 [00:01<00:00, 78.03batch/s]

Training batches on cuda:  74%|███████▍  | 126/170 [00:01<00:00, 78.28batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:01<00:00, 78.78batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:01<00:00, 79.85batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:02<00:00, 77.63batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:02<00:00, 76.44batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:02<00:00, 75.28batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000547, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:22,  2.47s/epoch, loss=0.000547, prev_loss=nan]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:02, 59.73batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:02, 60.02batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:02, 60.88batch/s]

Training batches on cuda:  16%|█▌        | 27/170 [00:00<00:02, 63.29batch/s]

Training batches on cuda:  21%|██        | 35/170 [00:00<00:02, 65.60batch/s]

Training batches on cuda:  25%|██▍       | 42/170 [00:00<00:01, 66.44batch/s]

Training batches on cuda:  29%|██▉       | 49/170 [00:00<00:01, 67.42batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:00<00:01, 63.90batch/s]

Training batches on cuda:  38%|███▊      | 65/170 [00:00<00:01, 68.51batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:01<00:01, 72.13batch/s]

Training batches on cuda:  49%|████▉     | 83/170 [00:01<00:01, 74.77batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:01<00:01, 77.44batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:01<00:00, 79.44batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:01<00:00, 78.38batch/s]

Training batches on cuda:  69%|██████▉   | 118/170 [00:01<00:00, 71.89batch/s]

Training batches on cuda:  74%|███████▍  | 126/170 [00:01<00:00, 69.24batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:01<00:00, 68.80batch/s]

Training batches on cuda:  84%|████████▎ | 142/170 [00:02<00:00, 70.21batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:02<00:00, 73.32batch/s]

Training batches on cuda:  94%|█████████▍| 160/170 [00:02<00:00, 76.27batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:02<00:00, 78.17batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:22,  2.47s/epoch, loss=0.000535, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:19,  2.49s/epoch, loss=0.000535, prev_loss=nan]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:03, 53.98batch/s]

Training batches on cuda:   9%|▉         | 15/170 [00:00<00:02, 59.93batch/s]

Training batches on cuda:  14%|█▍        | 24/170 [00:00<00:02, 65.18batch/s]

Training batches on cuda:  19%|█▉        | 33/170 [00:00<00:01, 69.51batch/s]

Training batches on cuda:  25%|██▍       | 42/170 [00:00<00:01, 72.98batch/s]

Training batches on cuda:  30%|███       | 51/170 [00:00<00:01, 75.28batch/s]

Training batches on cuda:  35%|███▌      | 60/170 [00:00<00:01, 77.83batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:00<00:01, 78.15batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:01<00:01, 73.87batch/s]

Training batches on cuda:  51%|█████     | 86/170 [00:01<00:01, 75.07batch/s]

Training batches on cuda:  55%|█████▌    | 94/170 [00:01<00:01, 73.48batch/s]

Training batches on cuda:  60%|██████    | 102/170 [00:01<00:00, 71.44batch/s]

Training batches on cuda:  65%|██████▍   | 110/170 [00:01<00:00, 66.79batch/s]

Training batches on cuda:  69%|██████▉   | 118/170 [00:01<00:00, 69.69batch/s]

Training batches on cuda:  74%|███████▍  | 126/170 [00:01<00:00, 69.60batch/s]

Training batches on cuda:  79%|███████▉  | 134/170 [00:01<00:00, 69.92batch/s]

Training batches on cuda:  84%|████████▎ | 142/170 [00:01<00:00, 72.22batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:02<00:00, 74.87batch/s]

Training batches on cuda:  94%|█████████▍| 160/170 [00:02<00:00, 77.30batch/s]

Training batches on cuda:  99%|█████████▉| 169/170 [00:02<00:00, 79.06batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:19,  2.49s/epoch, loss=0.00051, prev_loss=0.000535]
Training epochs on cuda:  30%|███       | 3/10 [00:07<00:17,  2.47s/epoch, loss=0.00051, prev_loss=0.000535]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:02, 56.20batch/s]

Training batches on cuda:   8%|▊         | 14/170 [00:00<00:02, 60.76batch/s]

Training batches on cuda:  14%|█▎        | 23/170 [00:00<00:02, 65.91batch/s]

Training batches on cuda:  18%|█▊        | 30/170 [00:00<00:02, 66.58batch/s]

Training batches on cuda:  22%|██▏       | 37/170 [00:00<00:01, 67.28batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:00<00:01, 67.04batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:00<00:01, 68.31batch/s]

Training batches on cuda:  35%|███▌      | 60/170 [00:00<00:01, 68.06batch/s]

Training batches on cuda:  39%|███▉      | 67/170 [00:00<00:01, 63.99batch/s]

Training batches on cuda:  44%|████▍     | 75/170 [00:01<00:01, 67.92batch/s]

Training batches on cuda:  49%|████▉     | 84/170 [00:01<00:01, 71.07batch/s]

Training batches on cuda:  54%|█████▍    | 92/170 [00:01<00:01, 72.20batch/s]

Training batches on cuda:  59%|█████▉    | 101/170 [00:01<00:00, 74.45batch/s]

Training batches on cuda:  64%|██████▍   | 109/170 [00:01<00:00, 73.94batch/s]

Training batches on cuda:  69%|██████▉   | 118/170 [00:01<00:00, 76.65batch/s]

Training batches on cuda:  75%|███████▍  | 127/170 [00:01<00:00, 78.58batch/s]

Training batches on cuda:  80%|████████  | 136/170 [00:01<00:00, 79.65batch/s]

Training batches on cuda:  85%|████████▌ | 145/170 [00:01<00:00, 80.08batch/s]

Training batches on cuda:  91%|█████████ | 154/170 [00:02<00:00, 80.05batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:02<00:00, 79.82batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:17,  2.47s/epoch, loss=0.000435, prev_loss=0.00051]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:14,  2.47s/epoch, loss=0.000435, prev_loss=0.00051]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:03, 47.92batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:02, 54.47batch/s]

Training batches on cuda:  13%|█▎        | 22/170 [00:00<00:02, 61.17batch/s]

Training batches on cuda:  18%|█▊        | 31/170 [00:00<00:02, 66.55batch/s]

Training batches on cuda:  23%|██▎       | 39/170 [00:00<00:01, 68.98batch/s]

Training batches on cuda:  28%|██▊       | 47/170 [00:00<00:01, 70.64batch/s]

Training batches on cuda:  32%|███▏      | 55/170 [00:00<00:01, 72.65batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:00<00:01, 75.43batch/s]

Training batches on cuda:  42%|████▏     | 72/170 [00:00<00:01, 76.25batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:01<00:01, 77.08batch/s]

Training batches on cuda:  52%|█████▏    | 89/170 [00:01<00:01, 78.57batch/s]

Training batches on cuda:  57%|█████▋    | 97/170 [00:01<00:00, 74.30batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:01<00:00, 71.53batch/s]

Training batches on cuda:  66%|██████▋   | 113/170 [00:01<00:00, 71.46batch/s]

Training batches on cuda:  71%|███████   | 121/170 [00:01<00:00, 71.49batch/s]

Training batches on cuda:  76%|███████▋  | 130/170 [00:01<00:00, 74.96batch/s]

Training batches on cuda:  82%|████████▏ | 139/170 [00:01<00:00, 76.84batch/s]

Training batches on cuda:  86%|████████▋ | 147/170 [00:01<00:00, 77.56batch/s]

Training batches on cuda:  92%|█████████▏| 156/170 [00:02<00:00, 79.51batch/s]

Training batches on cuda:  97%|█████████▋| 165/170 [00:02<00:00, 80.62batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:14,  2.47s/epoch, loss=0.000309, prev_loss=0.000435]
Training epochs on cuda:  50%|█████     | 5/10 [00:12<00:12,  2.45s/epoch, loss=0.000309, prev_loss=0.000435]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:03, 47.03batch/s]

Training batches on cuda:   8%|▊         | 13/170 [00:00<00:02, 53.02batch/s]

Training batches on cuda:  12%|█▏        | 21/170 [00:00<00:02, 58.86batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:00<00:02, 63.92batch/s]

Training batches on cuda:  22%|██▏       | 37/170 [00:00<00:01, 67.55batch/s]

Training batches on cuda:  26%|██▌       | 44/170 [00:00<00:01, 65.73batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:00<00:01, 67.19batch/s]

Training batches on cuda:  35%|███▍      | 59/170 [00:00<00:01, 61.07batch/s]

Training batches on cuda:  39%|███▉      | 67/170 [00:00<00:01, 64.26batch/s]

Training batches on cuda:  44%|████▍     | 75/170 [00:01<00:01, 67.45batch/s]

Training batches on cuda:  48%|████▊     | 82/170 [00:01<00:01, 67.84batch/s]

Training batches on cuda:  53%|█████▎    | 90/170 [00:01<00:01, 69.28batch/s]

Training batches on cuda:  58%|█████▊    | 98/170 [00:01<00:01, 70.43batch/s]

Training batches on cuda:  62%|██████▏   | 106/170 [00:01<00:00, 71.35batch/s]

Training batches on cuda:  67%|██████▋   | 114/170 [00:01<00:00, 72.54batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:01<00:00, 75.60batch/s]

Training batches on cuda:  78%|███████▊  | 132/170 [00:01<00:00, 77.35batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:01<00:00, 79.19batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:02<00:00, 80.19batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:02<00:00, 80.63batch/s]

Training batches on cuda:  99%|█████████▉| 168/170 [00:02<00:00, 81.20batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:12,  2.45s/epoch, loss=0.000235, prev_loss=0.000309]
Training epochs on cuda:  60%|██████    | 6/10 [00:14<00:09,  2.47s/epoch, loss=0.000235, prev_loss=0.000309]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:03, 54.43batch/s]

Training batches on cuda:   9%|▉         | 15/170 [00:00<00:02, 60.46batch/s]

Training batches on cuda:  14%|█▍        | 24/170 [00:00<00:02, 65.21batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:00<00:02, 68.85batch/s]

Training batches on cuda:  24%|██▎       | 40/170 [00:00<00:01, 70.53batch/s]

Training batches on cuda:  28%|██▊       | 48/170 [00:00<00:01, 71.24batch/s]

Training batches on cuda:  33%|███▎      | 56/170 [00:00<00:01, 71.94batch/s]

Training batches on cuda:  38%|███▊      | 64/170 [00:00<00:01, 72.30batch/s]

Training batches on cuda:  42%|████▏     | 72/170 [00:00<00:01, 72.65batch/s]

Training batches on cuda:  47%|████▋     | 80/170 [00:01<00:01, 72.68batch/s]

Training batches on cuda:  52%|█████▏    | 88/170 [00:01<00:01, 72.82batch/s]

Training batches on cuda:  56%|█████▋    | 96/170 [00:01<00:01, 72.98batch/s]

Training batches on cuda:  62%|██████▏   | 105/170 [00:01<00:00, 75.48batch/s]

Training batches on cuda:  67%|██████▋   | 114/170 [00:01<00:00, 77.10batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:01<00:00, 78.97batch/s]

Training batches on cuda:  78%|███████▊  | 132/170 [00:01<00:00, 80.13batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:01<00:00, 80.48batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:01<00:00, 81.14batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:02<00:00, 81.56batch/s]

Training batches on cuda:  99%|█████████▉| 168/170 [00:02<00:00, 81.62batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:09,  2.47s/epoch, loss=0.000193, prev_loss=0.000235]
Training epochs on cuda:  70%|███████   | 7/10 [00:17<00:07,  2.44s/epoch, loss=0.000193, prev_loss=0.000235]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 7/170 [00:00<00:02, 63.69batch/s]

Training batches on cuda:   9%|▉         | 16/170 [00:00<00:02, 68.49batch/s]

Training batches on cuda:  15%|█▍        | 25/170 [00:00<00:02, 71.95batch/s]

Training batches on cuda:  20%|██        | 34/170 [00:00<00:01, 75.38batch/s]

Training batches on cuda:  25%|██▌       | 43/170 [00:00<00:01, 77.71batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:00<00:01, 79.40batch/s]

Training batches on cuda:  36%|███▌      | 61/170 [00:00<00:01, 79.68batch/s]

Training batches on cuda:  41%|████      | 69/170 [00:00<00:01, 76.59batch/s]

Training batches on cuda:  45%|████▌     | 77/170 [00:01<00:01, 70.25batch/s]

Training batches on cuda:  50%|█████     | 85/170 [00:01<00:01, 71.19batch/s]

Training batches on cuda:  55%|█████▌    | 94/170 [00:01<00:01, 74.38batch/s]

Training batches on cuda:  61%|██████    | 103/170 [00:01<00:00, 76.69batch/s]

Training batches on cuda:  66%|██████▌   | 112/170 [00:01<00:00, 78.31batch/s]

Training batches on cuda:  71%|███████   | 121/170 [00:01<00:00, 79.09batch/s]

Training batches on cuda:  76%|███████▌  | 129/170 [00:01<00:00, 78.57batch/s]

Training batches on cuda:  81%|████████  | 137/170 [00:01<00:00, 78.95batch/s]

Training batches on cuda:  85%|████████▌ | 145/170 [00:01<00:00, 78.35batch/s]

Training batches on cuda:  91%|█████████ | 154/170 [00:01<00:00, 79.11batch/s]

Training batches on cuda:  96%|█████████▌| 163/170 [00:02<00:00, 80.40batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:19<00:07,  2.44s/epoch, loss=0.000177, prev_loss=0.000193]
Training epochs on cuda:  80%|████████  | 8/10 [00:19<00:04,  2.42s/epoch, loss=0.000177, prev_loss=0.000193]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/170 [00:00<00:03, 54.28batch/s]

Training batches on cuda:   9%|▉         | 15/170 [00:00<00:02, 60.14batch/s]

Training batches on cuda:  14%|█▎        | 23/170 [00:00<00:02, 64.34batch/s]

Training batches on cuda:  19%|█▉        | 32/170 [00:00<00:02, 68.96batch/s]

Training batches on cuda:  24%|██▎       | 40/170 [00:00<00:01, 71.68batch/s]

Training batches on cuda:  28%|██▊       | 48/170 [00:00<00:01, 72.73batch/s]

Training batches on cuda:  34%|███▎      | 57/170 [00:00<00:01, 75.30batch/s]

Training batches on cuda:  39%|███▉      | 66/170 [00:00<00:01, 77.16batch/s]

Training batches on cuda:  44%|████▎     | 74/170 [00:00<00:01, 76.88batch/s]

Training batches on cuda:  48%|████▊     | 82/170 [00:01<00:01, 77.40batch/s]

Training batches on cuda:  54%|█████▎    | 91/170 [00:01<00:01, 78.56batch/s]

Training batches on cuda:  58%|█████▊    | 99/170 [00:01<00:00, 71.94batch/s]

Training batches on cuda:  63%|██████▎   | 107/170 [00:01<00:00, 69.87batch/s]

Training batches on cuda:  68%|██████▊   | 115/170 [00:01<00:00, 71.05batch/s]

Training batches on cuda:  72%|███████▏  | 123/170 [00:01<00:00, 71.67batch/s]

Training batches on cuda:  78%|███████▊  | 132/170 [00:01<00:00, 74.92batch/s]

Training batches on cuda:  83%|████████▎ | 141/170 [00:01<00:00, 76.99batch/s]

Training batches on cuda:  88%|████████▊ | 150/170 [00:01<00:00, 78.74batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:02<00:00, 79.92batch/s]

Training batches on cuda:  99%|█████████▉| 168/170 [00:02<00:00, 79.63batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:21<00:04,  2.42s/epoch, loss=0.00017, prev_loss=0.000177] 
Training epochs on cuda:  90%|█████████ | 9/10 [00:21<00:02,  2.41s/epoch, loss=0.00017, prev_loss=0.000177]

Training batches on cuda:   0%|          | 0/170 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/170 [00:00<00:03, 44.59batch/s]

Training batches on cuda:   7%|▋         | 12/170 [00:00<00:03, 49.07batch/s]

Training batches on cuda:  12%|█▏        | 20/170 [00:00<00:02, 55.29batch/s]

Training batches on cuda:  17%|█▋        | 29/170 [00:00<00:02, 61.90batch/s]

Training batches on cuda:  22%|██▏       | 37/170 [00:00<00:02, 65.53batch/s]

Training batches on cuda:  26%|██▋       | 45/170 [00:00<00:01, 66.42batch/s]

Training batches on cuda:  31%|███       | 52/170 [00:00<00:01, 65.88batch/s]

Training batches on cuda:  35%|███▍      | 59/170 [00:00<00:01, 65.42batch/s]

Training batches on cuda:  39%|███▉      | 66/170 [00:00<00:01, 65.01batch/s]

Training batches on cuda:  43%|████▎     | 73/170 [00:01<00:01, 65.97batch/s]

Training batches on cuda:  48%|████▊     | 81/170 [00:01<00:01, 67.77batch/s]

Training batches on cuda:  52%|█████▏    | 88/170 [00:01<00:01, 63.78batch/s]

Training batches on cuda:  56%|█████▋    | 96/170 [00:01<00:01, 66.18batch/s]

Training batches on cuda:  61%|██████    | 104/170 [00:01<00:00, 68.40batch/s]

Training batches on cuda:  66%|██████▌   | 112/170 [00:01<00:00, 68.77batch/s]

Training batches on cuda:  70%|███████   | 119/170 [00:01<00:00, 67.80batch/s]

Training batches on cuda:  75%|███████▍  | 127/170 [00:01<00:00, 69.56batch/s]

Training batches on cuda:  79%|███████▉  | 135/170 [00:01<00:00, 70.52batch/s]

Training batches on cuda:  84%|████████▍ | 143/170 [00:02<00:00, 71.02batch/s]

Training batches on cuda:  89%|████████▉ | 151/170 [00:02<00:00, 71.56batch/s]

Training batches on cuda:  94%|█████████▎| 159/170 [00:02<00:00, 70.52batch/s]

Training batches on cuda:  98%|█████████▊| 167/170 [00:02<00:00, 71.88batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:24<00:02,  2.41s/epoch, loss=0.000167, prev_loss=0.00017]2021-09-16:19:24:11,195 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:24:12,189 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:24:27,90 INFO     [evaluator.py:657] Evaluation took 14.90s seconds
2021-09-16:19:24:27,237 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00,  7.29s/epoch, loss=0.000167, prev_loss=0.00017]
Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00,  4.06s/epoch, loss=0.000167, prev_loss=0.00017]
2021-09-16:19:24:27,237 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp4736tqos'
2021-09-16:19:24:27,297 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp4736tqos' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.33ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.42ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:24:42,75 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 19:24:42,081] Trial 81 finished with value: 0.9585260118271107 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.01413271375250177, 'optimizer.lr': 0.04432192654355775, 'negative_sampler.num_negs_per_pos': 7, 'training.batch_size': 1832}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:24:42,106 WARNING  [api.py:823] No random seed is specified. Setting to 367984747.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:04, 32.12batch/s]

Training batches on cuda:   7%|▋         | 10/146 [00:00<00:03, 36.87batch/s]

Training batches on cuda:  11%|█         | 16/146 [00:00<00:03, 41.23batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:02, 45.41batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:02, 48.55batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:00<00:02, 52.43batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:00<00:01, 55.44batch/s]

Training batches on cuda:  34%|███▎      | 49/146 [00:00<00:01, 58.23batch/s]

Training batches on cuda:  38%|███▊      | 56/146 [00:00<00:01, 60.96batch/s]

Training batches on cuda:  44%|████▍     | 64/146 [00:01<00:01, 63.46batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:01<00:01, 65.85batch/s]

Training batches on cuda:  54%|█████▍    | 79/146 [00:01<00:00, 67.03batch/s]

Training batches on cuda:  60%|█████▉    | 87/146 [00:01<00:00, 68.34batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:01<00:00, 68.91batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:01<00:00, 68.96batch/s]

Training batches on cuda:  75%|███████▌  | 110/146 [00:01<00:00, 69.74batch/s]

Training batches on cuda:  81%|████████  | 118/146 [00:01<00:00, 70.76batch/s]

Training batches on cuda:  86%|████████▋ | 126/146 [00:01<00:00, 71.07batch/s]

Training batches on cuda:  92%|█████████▏| 134/146 [00:02<00:00, 70.95batch/s]

Training batches on cuda:  97%|█████████▋| 142/146 [00:02<00:00, 70.59batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000472, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:21,  2.43s/epoch, loss=0.000472, prev_loss=nan]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:02, 49.32batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:02, 54.41batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:02, 58.11batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:01, 61.48batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:00<00:01, 64.00batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:00<00:01, 65.90batch/s]

Training batches on cuda:  36%|███▌      | 52/146 [00:00<00:01, 66.48batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:00<00:01, 67.95batch/s]

Training batches on cuda:  47%|████▋     | 68/146 [00:00<00:01, 69.12batch/s]

Training batches on cuda:  51%|█████▏    | 75/146 [00:01<00:01, 68.82batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:01<00:00, 69.04batch/s]

Training batches on cuda:  62%|██████▏   | 91/146 [00:01<00:00, 69.46batch/s]

Training batches on cuda:  68%|██████▊   | 99/146 [00:01<00:00, 69.90batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:01<00:00, 70.20batch/s]

Training batches on cuda:  79%|███████▉  | 115/146 [00:01<00:00, 70.67batch/s]

Training batches on cuda:  84%|████████▍ | 123/146 [00:01<00:00, 69.68batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:01<00:00, 70.24batch/s]

Training batches on cuda:  95%|█████████▌| 139/146 [00:01<00:00, 71.40batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:21,  2.43s/epoch, loss=0.00047, prev_loss=nan] 
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:19,  2.40s/epoch, loss=0.00047, prev_loss=nan]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:02, 49.38batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:02, 54.65batch/s]

Training batches on cuda:  14%|█▎        | 20/146 [00:00<00:02, 58.29batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:01, 61.38batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:00<00:01, 63.56batch/s]

Training batches on cuda:  29%|██▉       | 43/146 [00:00<00:01, 66.31batch/s]

Training batches on cuda:  35%|███▍      | 51/146 [00:00<00:01, 67.98batch/s]

Training batches on cuda:  40%|████      | 59/146 [00:00<00:01, 69.33batch/s]

Training batches on cuda:  46%|████▌     | 67/146 [00:00<00:01, 70.06batch/s]

Training batches on cuda:  51%|█████▏    | 75/146 [00:01<00:01, 70.96batch/s]

Training batches on cuda:  56%|█████▌    | 82/146 [00:01<00:00, 70.66batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:01<00:00, 70.91batch/s]

Training batches on cuda:  67%|██████▋   | 98/146 [00:01<00:00, 71.10batch/s]

Training batches on cuda:  73%|███████▎  | 106/146 [00:01<00:00, 71.27batch/s]

Training batches on cuda:  78%|███████▊  | 114/146 [00:01<00:00, 71.98batch/s]

Training batches on cuda:  84%|████████▎ | 122/146 [00:01<00:00, 72.20batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:01<00:00, 72.66batch/s]

Training batches on cuda:  95%|█████████▍| 138/146 [00:01<00:00, 72.43batch/s]

Training batches on cuda: 100%|██████████| 146/146 [00:02<00:00, 72.42batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:19,  2.40s/epoch, loss=0.000469, prev_loss=0.00047]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:16,  2.35s/epoch, loss=0.000469, prev_loss=0.00047]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:03, 45.99batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:02, 51.49batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:02, 56.03batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:01, 60.09batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:00<00:01, 59.76batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:00<00:01, 60.03batch/s]

Training batches on cuda:  33%|███▎      | 48/146 [00:00<00:01, 53.77batch/s]

Training batches on cuda:  38%|███▊      | 55/146 [00:00<00:01, 56.37batch/s]

Training batches on cuda:  42%|████▏     | 62/146 [00:01<00:01, 58.36batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:01<00:01, 60.10batch/s]

Training batches on cuda:  52%|█████▏    | 76/146 [00:01<00:01, 61.99batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:01<00:00, 63.39batch/s]

Training batches on cuda:  62%|██████▏   | 90/146 [00:01<00:00, 64.27batch/s]

Training batches on cuda:  66%|██████▋   | 97/146 [00:01<00:00, 64.99batch/s]

Training batches on cuda:  71%|███████   | 104/146 [00:01<00:00, 65.56batch/s]

Training batches on cuda:  77%|███████▋  | 112/146 [00:01<00:00, 68.54batch/s]

Training batches on cuda:  82%|████████▏ | 120/146 [00:01<00:00, 70.92batch/s]

Training batches on cuda:  88%|████████▊ | 128/146 [00:01<00:00, 72.60batch/s]

Training batches on cuda:  93%|█████████▎| 136/146 [00:02<00:00, 73.72batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:02<00:00, 74.40batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:16,  2.35s/epoch, loss=0.000465, prev_loss=0.000469]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:14,  2.37s/epoch, loss=0.000465, prev_loss=0.000469]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:04, 35.37batch/s]

Training batches on cuda:   8%|▊         | 12/146 [00:00<00:03, 41.66batch/s]

Training batches on cuda:  14%|█▎        | 20/146 [00:00<00:02, 47.79batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:02, 53.16batch/s]

Training batches on cuda:  25%|██▍       | 36/146 [00:00<00:01, 57.57batch/s]

Training batches on cuda:  30%|███       | 44/146 [00:00<00:01, 61.65batch/s]

Training batches on cuda:  36%|███▌      | 52/146 [00:00<00:01, 64.23batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:00<00:01, 65.96batch/s]

Training batches on cuda:  46%|████▌     | 67/146 [00:00<00:01, 66.37batch/s]

Training batches on cuda:  51%|█████     | 74/146 [00:01<00:01, 66.91batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:01<00:00, 67.25batch/s]

Training batches on cuda:  60%|██████    | 88/146 [00:01<00:00, 66.82batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:01<00:00, 67.29batch/s]

Training batches on cuda:  71%|███████   | 103/146 [00:01<00:00, 68.17batch/s]

Training batches on cuda:  75%|███████▌  | 110/146 [00:01<00:00, 67.34batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:01<00:00, 67.50batch/s]

Training batches on cuda:  85%|████████▍ | 124/146 [00:01<00:00, 66.90batch/s]

Training batches on cuda:  90%|████████▉ | 131/146 [00:01<00:00, 67.54batch/s]

Training batches on cuda:  95%|█████████▍| 138/146 [00:02<00:00, 67.75batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:02<00:00, 66.91batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:14,  2.37s/epoch, loss=0.000462, prev_loss=0.000465]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.37s/epoch, loss=0.000462, prev_loss=0.000465]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:03, 37.63batch/s]

Training batches on cuda:   8%|▊         | 12/146 [00:00<00:03, 43.89batch/s]

Training batches on cuda:  13%|█▎        | 19/146 [00:00<00:02, 49.36batch/s]

Training batches on cuda:  18%|█▊        | 27/146 [00:00<00:02, 54.68batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:00<00:01, 58.74batch/s]

Training batches on cuda:  29%|██▉       | 43/146 [00:00<00:01, 61.88batch/s]

Training batches on cuda:  35%|███▍      | 51/146 [00:00<00:01, 64.66batch/s]

Training batches on cuda:  40%|███▉      | 58/146 [00:00<00:01, 65.53batch/s]

Training batches on cuda:  45%|████▍     | 65/146 [00:01<00:01, 59.59batch/s]

Training batches on cuda:  49%|████▉     | 72/146 [00:01<00:01, 62.09batch/s]

Training batches on cuda:  54%|█████▍    | 79/146 [00:01<00:01, 62.26batch/s]

Training batches on cuda:  59%|█████▉    | 86/146 [00:01<00:00, 62.63batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:01<00:00, 62.08batch/s]

Training batches on cuda:  68%|██████▊   | 100/146 [00:01<00:00, 62.61batch/s]

Training batches on cuda:  73%|███████▎  | 107/146 [00:01<00:00, 62.86batch/s]

Training batches on cuda:  78%|███████▊  | 114/146 [00:01<00:00, 63.19batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:01<00:00, 62.46batch/s]

Training batches on cuda:  88%|████████▊ | 128/146 [00:02<00:00, 62.60batch/s]

Training batches on cuda:  93%|█████████▎| 136/146 [00:02<00:00, 65.19batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:02<00:00, 66.64batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:11,  2.37s/epoch, loss=0.000455, prev_loss=0.000462]
Training epochs on cuda:  60%|██████    | 6/10 [00:14<00:09,  2.41s/epoch, loss=0.000455, prev_loss=0.000462]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:04, 35.22batch/s]

Training batches on cuda:   7%|▋         | 10/146 [00:00<00:03, 39.76batch/s]

Training batches on cuda:  11%|█         | 16/146 [00:00<00:02, 43.56batch/s]

Training batches on cuda:  15%|█▌        | 22/146 [00:00<00:02, 46.65batch/s]

Training batches on cuda:  19%|█▉        | 28/146 [00:00<00:02, 49.15batch/s]

Training batches on cuda:  23%|██▎       | 34/146 [00:00<00:02, 51.03batch/s]

Training batches on cuda:  27%|██▋       | 40/146 [00:00<00:02, 52.31batch/s]

Training batches on cuda:  32%|███▏      | 46/146 [00:00<00:01, 53.21batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:00<00:01, 55.52batch/s]

Training batches on cuda:  41%|████      | 60/146 [00:01<00:01, 57.57batch/s]

Training batches on cuda:  46%|████▌     | 67/146 [00:01<00:01, 59.27batch/s]

Training batches on cuda:  51%|█████     | 74/146 [00:01<00:01, 60.07batch/s]

Training batches on cuda:  55%|█████▌    | 81/146 [00:01<00:01, 60.77batch/s]

Training batches on cuda:  60%|██████    | 88/146 [00:01<00:00, 61.45batch/s]

Training batches on cuda:  65%|██████▌   | 95/146 [00:01<00:00, 62.97batch/s]

Training batches on cuda:  70%|██████▉   | 102/146 [00:01<00:00, 63.86batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:01<00:00, 64.88batch/s]

Training batches on cuda:  79%|███████▉  | 116/146 [00:01<00:00, 64.75batch/s]

Training batches on cuda:  84%|████████▍ | 123/146 [00:02<00:00, 64.48batch/s]

Training batches on cuda:  89%|████████▉ | 130/146 [00:02<00:00, 63.70batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:02<00:00, 61.64batch/s]

Training batches on cuda:  99%|█████████▊| 144/146 [00:02<00:00, 60.68batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:16<00:09,  2.41s/epoch, loss=0.000409, prev_loss=0.000455]
Training epochs on cuda:  70%|███████   | 7/10 [00:16<00:07,  2.48s/epoch, loss=0.000409, prev_loss=0.000455]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/146 [00:00<00:08, 17.69batch/s]

Training batches on cuda:   5%|▍         | 7/146 [00:00<00:06, 21.93batch/s]

Training batches on cuda:  10%|█         | 15/146 [00:00<00:04, 27.75batch/s]

Training batches on cuda:  16%|█▌        | 23/146 [00:00<00:03, 34.10batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:03, 38.33batch/s]

Training batches on cuda:  24%|██▍       | 35/146 [00:00<00:02, 42.09batch/s]

Training batches on cuda:  29%|██▉       | 42/146 [00:00<00:02, 47.17batch/s]

Training batches on cuda:  33%|███▎      | 48/146 [00:00<00:02, 45.07batch/s]

Training batches on cuda:  38%|███▊      | 55/146 [00:01<00:01, 49.72batch/s]

Training batches on cuda:  42%|████▏     | 62/146 [00:01<00:01, 54.18batch/s]

Training batches on cuda:  48%|████▊     | 70/146 [00:01<00:01, 58.32batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:01<00:01, 61.23batch/s]

Training batches on cuda:  58%|█████▊    | 84/146 [00:01<00:00, 63.50batch/s]

Training batches on cuda:  63%|██████▎   | 92/146 [00:01<00:00, 65.42batch/s]

Training batches on cuda:  68%|██████▊   | 99/146 [00:01<00:00, 65.71batch/s]

Training batches on cuda:  73%|███████▎  | 106/146 [00:01<00:00, 66.41batch/s]

Training batches on cuda:  77%|███████▋  | 113/146 [00:01<00:00, 67.42batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:01<00:00, 68.74batch/s]

Training batches on cuda:  88%|████████▊ | 128/146 [00:02<00:00, 68.53batch/s]

Training batches on cuda:  92%|█████████▏| 135/146 [00:02<00:00, 66.79batch/s]

Training batches on cuda:  97%|█████████▋| 142/146 [00:02<00:00, 66.94batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:19<00:07,  2.48s/epoch, loss=0.00032, prev_loss=0.000409] 
Training epochs on cuda:  80%|████████  | 8/10 [00:19<00:04,  2.50s/epoch, loss=0.00032, prev_loss=0.000409]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/146 [00:00<00:02, 47.72batch/s]

Training batches on cuda:   9%|▉         | 13/146 [00:00<00:02, 53.31batch/s]

Training batches on cuda:  14%|█▍        | 21/146 [00:00<00:02, 58.01batch/s]

Training batches on cuda:  20%|█▉        | 29/146 [00:00<00:01, 61.90batch/s]

Training batches on cuda:  25%|██▌       | 37/146 [00:00<00:01, 64.52batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:00<00:01, 67.12batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:00<00:01, 69.07batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:00<00:01, 70.45batch/s]

Training batches on cuda:  47%|████▋     | 69/146 [00:00<00:01, 71.51batch/s]

Training batches on cuda:  53%|█████▎    | 77/146 [00:01<00:00, 71.69batch/s]

Training batches on cuda:  58%|█████▊    | 85/146 [00:01<00:00, 72.45batch/s]

Training batches on cuda:  64%|██████▎   | 93/146 [00:01<00:00, 72.79batch/s]

Training batches on cuda:  69%|██████▉   | 101/146 [00:01<00:00, 72.74batch/s]

Training batches on cuda:  75%|███████▍  | 109/146 [00:01<00:00, 72.38batch/s]

Training batches on cuda:  80%|████████  | 117/146 [00:01<00:00, 72.25batch/s]

Training batches on cuda:  86%|████████▌ | 125/146 [00:01<00:00, 71.29batch/s]

Training batches on cuda:  91%|█████████ | 133/146 [00:01<00:00, 66.26batch/s]

Training batches on cuda:  96%|█████████▌| 140/146 [00:02<00:00, 65.34batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:21<00:04,  2.50s/epoch, loss=0.000269, prev_loss=0.00032]
Training epochs on cuda:  90%|█████████ | 9/10 [00:21<00:02,  2.44s/epoch, loss=0.000269, prev_loss=0.00032]

Training batches on cuda:   0%|          | 0/146 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 4/146 [00:00<00:03, 35.65batch/s]

Training batches on cuda:   8%|▊         | 11/146 [00:00<00:03, 41.18batch/s]

Training batches on cuda:  12%|█▏        | 18/146 [00:00<00:02, 46.18batch/s]

Training batches on cuda:  17%|█▋        | 25/146 [00:00<00:02, 49.92batch/s]

Training batches on cuda:  22%|██▏       | 32/146 [00:00<00:02, 53.53batch/s]

Training batches on cuda:  26%|██▌       | 38/146 [00:00<00:01, 55.03batch/s]

Training batches on cuda:  31%|███       | 45/146 [00:00<00:01, 56.86batch/s]

Training batches on cuda:  36%|███▋      | 53/146 [00:00<00:01, 60.47batch/s]

Training batches on cuda:  42%|████▏     | 61/146 [00:01<00:01, 63.62batch/s]

Training batches on cuda:  47%|████▋     | 68/146 [00:01<00:01, 63.13batch/s]

Training batches on cuda:  51%|█████▏    | 75/146 [00:01<00:01, 65.02batch/s]

Training batches on cuda:  57%|█████▋    | 83/146 [00:01<00:00, 66.98batch/s]

Training batches on cuda:  62%|██████▏   | 91/146 [00:01<00:00, 68.09batch/s]

Training batches on cuda:  68%|██████▊   | 99/146 [00:01<00:00, 69.00batch/s]

Training batches on cuda:  73%|███████▎  | 106/146 [00:01<00:00, 69.17batch/s]

Training batches on cuda:  78%|███████▊  | 114/146 [00:01<00:00, 69.80batch/s]

Training batches on cuda:  83%|████████▎ | 121/146 [00:01<00:00, 68.80batch/s]

Training batches on cuda:  88%|████████▊ | 129/146 [00:01<00:00, 70.30batch/s]

Training batches on cuda:  94%|█████████▍| 137/146 [00:02<00:00, 71.18batch/s]

Training batches on cuda:  99%|█████████▉| 145/146 [00:02<00:00, 71.00batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:24<00:02,  2.44s/epoch, loss=0.000232, prev_loss=0.000269]2021-09-16:19:25:06,659 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:25:07,724 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:25:22,598 INFO     [evaluator.py:657] Evaluation took 14.87s seconds
2021-09-16:19:25:22,726 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00,  7.25s/epoch, loss=0.000232, prev_loss=0.000269]
Training epochs on cuda: 100%|██████████| 10/10 [00:40<00:00,  4.02s/epoch, loss=0.000232, prev_loss=0.000269]
2021-09-16:19:25:22,727 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpi3kugdct'
2021-09-16:19:25:22,789 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpi3kugdct' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:18, 2.05ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:17, 2.20ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:16, 2.32ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.40ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.47ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.52ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.55ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.54ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.60ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.63ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.62ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.63ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.63ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.63ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.63ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.63ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:19:25:37,638 INFO     [evaluator.py:657] Evaluation took 14.84s seconds
[I 2021-09-16 19:25:37,643] Trial 82 finished with value: 0.9165437325913777 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.0363563159781825, 'optimizer.lr': 0.04453608671842387, 'negative_sampler.num_negs_per_pos': 7, 'training.batch_size': 2132}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:25:37,665 WARNING  [api.py:823] No random seed is specified. Setting to 3230777585.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 49.81batch/s]

Training batches on cuda:   6%|▌         | 9/161 [00:00<00:03, 45.74batch/s]

Training batches on cuda:  11%|█         | 17/161 [00:00<00:02, 51.86batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:02, 57.24batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:02, 61.72batch/s]

Training batches on cuda:  25%|██▌       | 41/161 [00:00<00:01, 65.07batch/s]

Training batches on cuda:  30%|███       | 49/161 [00:00<00:01, 67.74batch/s]

Training batches on cuda:  36%|███▌      | 58/161 [00:00<00:01, 71.95batch/s]

Training batches on cuda:  42%|████▏     | 67/161 [00:00<00:01, 74.38batch/s]

Training batches on cuda:  47%|████▋     | 76/161 [00:01<00:01, 76.85batch/s]

Training batches on cuda:  52%|█████▏    | 84/161 [00:01<00:00, 77.68batch/s]

Training batches on cuda:  57%|█████▋    | 92/161 [00:01<00:00, 75.21batch/s]

Training batches on cuda:  62%|██████▏   | 100/161 [00:01<00:00, 76.55batch/s]

Training batches on cuda:  68%|██████▊   | 109/161 [00:01<00:00, 78.19batch/s]

Training batches on cuda:  73%|███████▎  | 118/161 [00:01<00:00, 79.38batch/s]

Training batches on cuda:  79%|███████▉  | 127/161 [00:01<00:00, 80.68batch/s]

Training batches on cuda:  84%|████████▍ | 136/161 [00:01<00:00, 81.31batch/s]

Training batches on cuda:  90%|█████████ | 145/161 [00:01<00:00, 81.37batch/s]

Training batches on cuda:  96%|█████████▌| 154/161 [00:02<00:00, 81.09batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000524, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:21,  2.37s/epoch, loss=0.000524, prev_loss=nan]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 57.35batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 63.78batch/s]

Training batches on cuda:  14%|█▍        | 23/161 [00:00<00:02, 66.60batch/s]

Training batches on cuda:  19%|█▉        | 31/161 [00:00<00:01, 68.03batch/s]

Training batches on cuda:  24%|██▍       | 39/161 [00:00<00:01, 69.00batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:01, 66.15batch/s]

Training batches on cuda:  34%|███▎      | 54/161 [00:00<00:01, 68.44batch/s]

Training batches on cuda:  39%|███▊      | 62/161 [00:00<00:01, 70.44batch/s]

Training batches on cuda:  43%|████▎     | 70/161 [00:00<00:01, 71.24batch/s]

Training batches on cuda:  48%|████▊     | 77/161 [00:01<00:01, 68.21batch/s]

Training batches on cuda:  53%|█████▎    | 86/161 [00:01<00:01, 72.55batch/s]

Training batches on cuda:  59%|█████▉    | 95/161 [00:01<00:00, 75.76batch/s]

Training batches on cuda:  65%|██████▍   | 104/161 [00:01<00:00, 78.20batch/s]

Training batches on cuda:  70%|███████   | 113/161 [00:01<00:00, 80.51batch/s]

Training batches on cuda:  76%|███████▌  | 122/161 [00:01<00:00, 80.51batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:01<00:00, 77.20batch/s]

Training batches on cuda:  86%|████████▋ | 139/161 [00:01<00:00, 76.05batch/s]

Training batches on cuda:  91%|█████████▏| 147/161 [00:01<00:00, 75.77batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:02<00:00, 75.19batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:21,  2.37s/epoch, loss=0.000521, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:04<00:18,  2.37s/epoch, loss=0.000521, prev_loss=nan]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 58.79batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 64.74batch/s]

Training batches on cuda:  15%|█▍        | 24/161 [00:00<00:01, 69.61batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:01, 72.75batch/s]

Training batches on cuda:  25%|██▍       | 40/161 [00:00<00:01, 66.24batch/s]

Training batches on cuda:  30%|███       | 49/161 [00:00<00:01, 70.55batch/s]

Training batches on cuda:  36%|███▌      | 58/161 [00:00<00:01, 73.44batch/s]

Training batches on cuda:  42%|████▏     | 67/161 [00:00<00:01, 75.50batch/s]

Training batches on cuda:  47%|████▋     | 76/161 [00:00<00:01, 77.62batch/s]

Training batches on cuda:  52%|█████▏    | 84/161 [00:01<00:00, 78.30batch/s]

Training batches on cuda:  58%|█████▊    | 93/161 [00:01<00:00, 79.11batch/s]

Training batches on cuda:  63%|██████▎   | 101/161 [00:01<00:00, 79.12batch/s]

Training batches on cuda:  68%|██████▊   | 110/161 [00:01<00:00, 79.61batch/s]

Training batches on cuda:  74%|███████▍  | 119/161 [00:01<00:00, 80.02batch/s]

Training batches on cuda:  80%|███████▉  | 128/161 [00:01<00:00, 80.93batch/s]

Training batches on cuda:  85%|████████▌ | 137/161 [00:01<00:00, 80.53batch/s]

Training batches on cuda:  91%|█████████ | 146/161 [00:01<00:00, 80.72batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:01<00:00, 81.56batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:18,  2.37s/epoch, loss=0.000521, prev_loss=0.000521]
Training epochs on cuda:  30%|███       | 3/10 [00:06<00:16,  2.33s/epoch, loss=0.000521, prev_loss=0.000521]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/161 [00:00<00:05, 26.53batch/s]

Training batches on cuda:   7%|▋         | 11/161 [00:00<00:04, 33.03batch/s]

Training batches on cuda:  11%|█         | 18/161 [00:00<00:03, 38.69batch/s]

Training batches on cuda:  15%|█▍        | 24/161 [00:00<00:03, 42.96batch/s]

Training batches on cuda:  19%|█▊        | 30/161 [00:00<00:02, 45.75batch/s]

Training batches on cuda:  22%|██▏       | 36/161 [00:00<00:02, 49.12batch/s]

Training batches on cuda:  26%|██▌       | 42/161 [00:00<00:02, 51.78batch/s]

Training batches on cuda:  31%|███       | 50/161 [00:00<00:01, 56.60batch/s]

Training batches on cuda:  36%|███▌      | 58/161 [00:00<00:01, 61.64batch/s]

Training batches on cuda:  42%|████▏     | 67/161 [00:01<00:01, 67.04batch/s]

Training batches on cuda:  47%|████▋     | 76/161 [00:01<00:01, 70.74batch/s]

Training batches on cuda:  53%|█████▎    | 85/161 [00:01<00:01, 73.62batch/s]

Training batches on cuda:  58%|█████▊    | 94/161 [00:01<00:00, 76.23batch/s]

Training batches on cuda:  64%|██████▍   | 103/161 [00:01<00:00, 78.83batch/s]

Training batches on cuda:  70%|██████▉   | 112/161 [00:01<00:00, 80.63batch/s]

Training batches on cuda:  75%|███████▌  | 121/161 [00:01<00:00, 81.02batch/s]

Training batches on cuda:  81%|████████  | 130/161 [00:01<00:00, 81.62batch/s]

Training batches on cuda:  86%|████████▋ | 139/161 [00:01<00:00, 80.49batch/s]

Training batches on cuda:  92%|█████████▏| 148/161 [00:02<00:00, 77.62batch/s]

Training batches on cuda:  97%|█████████▋| 156/161 [00:02<00:00, 78.02batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:16,  2.33s/epoch, loss=0.000521, prev_loss=0.000521]
Training epochs on cuda:  40%|████      | 4/10 [00:09<00:14,  2.35s/epoch, loss=0.000521, prev_loss=0.000521]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 54.82batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 60.86batch/s]

Training batches on cuda:  14%|█▍        | 23/161 [00:00<00:02, 65.36batch/s]

Training batches on cuda:  19%|█▉        | 31/161 [00:00<00:01, 66.93batch/s]

Training batches on cuda:  24%|██▎       | 38/161 [00:00<00:01, 67.13batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:01, 69.02batch/s]

Training batches on cuda:  33%|███▎      | 53/161 [00:00<00:01, 68.70batch/s]

Training batches on cuda:  39%|███▊      | 62/161 [00:00<00:01, 72.12batch/s]

Training batches on cuda:  44%|████▍     | 71/161 [00:00<00:01, 75.45batch/s]

Training batches on cuda:  50%|████▉     | 80/161 [00:01<00:01, 77.70batch/s]

Training batches on cuda:  55%|█████▍    | 88/161 [00:01<00:00, 76.50batch/s]

Training batches on cuda:  60%|█████▉    | 96/161 [00:01<00:00, 73.50batch/s]

Training batches on cuda:  65%|██████▍   | 104/161 [00:01<00:00, 72.16batch/s]

Training batches on cuda:  70%|██████▉   | 112/161 [00:01<00:00, 68.18batch/s]

Training batches on cuda:  74%|███████▍  | 119/161 [00:01<00:00, 68.71batch/s]

Training batches on cuda:  79%|███████▉  | 127/161 [00:01<00:00, 69.58batch/s]

Training batches on cuda:  83%|████████▎ | 134/161 [00:01<00:00, 69.62batch/s]

Training batches on cuda:  88%|████████▊ | 142/161 [00:01<00:00, 70.94batch/s]

Training batches on cuda:  94%|█████████▍| 151/161 [00:02<00:00, 73.94batch/s]

Training batches on cuda:  99%|█████████▉| 159/161 [00:02<00:00, 73.98batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:14,  2.35s/epoch, loss=0.000521, prev_loss=0.000521]
Training epochs on cuda:  50%|█████     | 5/10 [00:11<00:11,  2.37s/epoch, loss=0.000521, prev_loss=0.000521]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 55.93batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 61.89batch/s]

Training batches on cuda:  14%|█▍        | 23/161 [00:00<00:02, 66.19batch/s]

Training batches on cuda:  19%|█▉        | 31/161 [00:00<00:01, 68.58batch/s]

Training batches on cuda:  24%|██▍       | 39/161 [00:00<00:01, 71.05batch/s]

Training batches on cuda:  30%|██▉       | 48/161 [00:00<00:01, 74.24batch/s]

Training batches on cuda:  35%|███▌      | 57/161 [00:00<00:01, 76.43batch/s]

Training batches on cuda:  41%|████      | 66/161 [00:00<00:01, 78.13batch/s]

Training batches on cuda:  47%|████▋     | 75/161 [00:00<00:01, 79.89batch/s]

Training batches on cuda:  52%|█████▏    | 84/161 [00:01<00:00, 80.18batch/s]

Training batches on cuda:  58%|█████▊    | 93/161 [00:01<00:00, 81.40batch/s]

Training batches on cuda:  63%|██████▎   | 102/161 [00:01<00:00, 82.62batch/s]

Training batches on cuda:  69%|██████▉   | 111/161 [00:01<00:00, 81.93batch/s]

Training batches on cuda:  75%|███████▍  | 120/161 [00:01<00:00, 78.93batch/s]

Training batches on cuda:  80%|███████▉  | 128/161 [00:01<00:00, 77.31batch/s]

Training batches on cuda:  84%|████████▍ | 136/161 [00:01<00:00, 76.29batch/s]

Training batches on cuda:  89%|████████▉ | 144/161 [00:01<00:00, 75.28batch/s]

Training batches on cuda:  94%|█████████▍| 152/161 [00:01<00:00, 74.76batch/s]

Training batches on cuda: 100%|██████████| 161/161 [00:02<00:00, 76.67batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:11,  2.37s/epoch, loss=0.000521, prev_loss=0.000521]
Training epochs on cuda:  60%|██████    | 6/10 [00:14<00:09,  2.33s/epoch, loss=0.000521, prev_loss=0.000521]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/161 [00:00<00:04, 37.24batch/s]

Training batches on cuda:   7%|▋         | 12/161 [00:00<00:03, 44.32batch/s]

Training batches on cuda:  13%|█▎        | 21/161 [00:00<00:02, 51.60batch/s]

Training batches on cuda:  19%|█▊        | 30/161 [00:00<00:02, 57.68batch/s]

Training batches on cuda:  24%|██▎       | 38/161 [00:00<00:01, 62.21batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:01, 66.46batch/s]

Training batches on cuda:  34%|███▍      | 55/161 [00:00<00:01, 70.86batch/s]

Training batches on cuda:  40%|███▉      | 64/161 [00:00<00:01, 74.03batch/s]

Training batches on cuda:  45%|████▍     | 72/161 [00:00<00:01, 74.65batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:01<00:01, 76.70batch/s]

Training batches on cuda:  56%|█████▌    | 90/161 [00:01<00:00, 78.22batch/s]

Training batches on cuda:  61%|██████    | 98/161 [00:01<00:00, 77.27batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 77.96batch/s]

Training batches on cuda:  71%|███████▏  | 115/161 [00:01<00:00, 79.40batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:01<00:00, 80.62batch/s]

Training batches on cuda:  83%|████████▎ | 133/161 [00:01<00:00, 81.51batch/s]

Training batches on cuda:  88%|████████▊ | 142/161 [00:01<00:00, 80.64batch/s]

Training batches on cuda:  94%|█████████▍| 151/161 [00:01<00:00, 81.71batch/s]

Training batches on cuda:  99%|█████████▉| 160/161 [00:02<00:00, 80.89batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:16<00:09,  2.33s/epoch, loss=0.000521, prev_loss=0.000521]
Training epochs on cuda:  70%|███████   | 7/10 [00:16<00:06,  2.30s/epoch, loss=0.000521, prev_loss=0.000521]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▍         | 7/161 [00:00<00:02, 63.50batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:02, 68.15batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:01, 72.28batch/s]

Training batches on cuda:  21%|██        | 34/161 [00:00<00:01, 75.23batch/s]

Training batches on cuda:  27%|██▋       | 43/161 [00:00<00:01, 77.33batch/s]

Training batches on cuda:  32%|███▏      | 52/161 [00:00<00:01, 79.02batch/s]

Training batches on cuda:  38%|███▊      | 61/161 [00:00<00:01, 80.26batch/s]

Training batches on cuda:  43%|████▎     | 70/161 [00:00<00:01, 80.82batch/s]

Training batches on cuda:  49%|████▉     | 79/161 [00:00<00:01, 81.65batch/s]

Training batches on cuda:  55%|█████▍    | 88/161 [00:01<00:00, 82.37batch/s]

Training batches on cuda:  60%|██████    | 97/161 [00:01<00:00, 82.59batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 83.05batch/s]

Training batches on cuda:  71%|███████▏  | 115/161 [00:01<00:00, 83.13batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:01<00:00, 83.04batch/s]

Training batches on cuda:  83%|████████▎ | 133/161 [00:01<00:00, 84.44batch/s]

Training batches on cuda:  88%|████████▊ | 142/161 [00:01<00:00, 85.75batch/s]

Training batches on cuda:  94%|█████████▍| 151/161 [00:01<00:00, 86.20batch/s]

Training batches on cuda:  99%|█████████▉| 160/161 [00:01<00:00, 86.29batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:18<00:06,  2.30s/epoch, loss=0.000522, prev_loss=0.000521]
Training epochs on cuda:  80%|████████  | 8/10 [00:18<00:04,  2.25s/epoch, loss=0.000522, prev_loss=0.000521]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/161 [00:00<00:03, 39.33batch/s]

Training batches on cuda:   8%|▊         | 13/161 [00:00<00:03, 46.92batch/s]

Training batches on cuda:  14%|█▍        | 23/161 [00:00<00:02, 54.85batch/s]

Training batches on cuda:  20%|█▉        | 32/161 [00:00<00:02, 61.09batch/s]

Training batches on cuda:  25%|██▌       | 41/161 [00:00<00:01, 66.18batch/s]

Training batches on cuda:  31%|███       | 50/161 [00:00<00:01, 70.44batch/s]

Training batches on cuda:  37%|███▋      | 59/161 [00:00<00:01, 74.11batch/s]

Training batches on cuda:  42%|████▏     | 68/161 [00:00<00:01, 77.92batch/s]

Training batches on cuda:  48%|████▊     | 77/161 [00:00<00:01, 79.77batch/s]

Training batches on cuda:  53%|█████▎    | 86/161 [00:01<00:00, 81.09batch/s]

Training batches on cuda:  59%|█████▉    | 95/161 [00:01<00:00, 81.38batch/s]

Training batches on cuda:  65%|██████▍   | 104/161 [00:01<00:00, 79.33batch/s]

Training batches on cuda:  70%|███████   | 113/161 [00:01<00:00, 80.72batch/s]

Training batches on cuda:  76%|███████▌  | 122/161 [00:01<00:00, 81.47batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:01<00:00, 81.19batch/s]

Training batches on cuda:  87%|████████▋ | 140/161 [00:01<00:00, 81.83batch/s]

Training batches on cuda:  93%|█████████▎| 149/161 [00:01<00:00, 83.32batch/s]

Training batches on cuda:  98%|█████████▊| 158/161 [00:01<00:00, 74.53batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:20<00:04,  2.25s/epoch, loss=0.000522, prev_loss=0.000522]
Training epochs on cuda:  90%|█████████ | 9/10 [00:20<00:02,  2.24s/epoch, loss=0.000522, prev_loss=0.000522]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 53.75batch/s]

Training batches on cuda:   9%|▉         | 15/161 [00:00<00:02, 60.69batch/s]

Training batches on cuda:  15%|█▍        | 24/161 [00:00<00:02, 66.60batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:01, 71.13batch/s]

Training batches on cuda:  26%|██▌       | 42/161 [00:00<00:01, 75.13batch/s]

Training batches on cuda:  32%|███▏      | 51/161 [00:00<00:01, 78.38batch/s]

Training batches on cuda:  37%|███▋      | 60/161 [00:00<00:01, 80.10batch/s]

Training batches on cuda:  43%|████▎     | 69/161 [00:00<00:01, 81.86batch/s]

Training batches on cuda:  48%|████▊     | 78/161 [00:00<00:00, 83.28batch/s]

Training batches on cuda:  54%|█████▍    | 87/161 [00:01<00:00, 83.94batch/s]

Training batches on cuda:  60%|█████▉    | 96/161 [00:01<00:00, 84.16batch/s]

Training batches on cuda:  65%|██████▌   | 105/161 [00:01<00:00, 85.25batch/s]

Training batches on cuda:  71%|███████   | 114/161 [00:01<00:00, 85.68batch/s]

Training batches on cuda:  76%|███████▋  | 123/161 [00:01<00:00, 85.18batch/s]

Training batches on cuda:  82%|████████▏ | 132/161 [00:01<00:00, 85.49batch/s]

Training batches on cuda:  88%|████████▊ | 141/161 [00:01<00:00, 85.07batch/s]

Training batches on cuda:  93%|█████████▎| 150/161 [00:01<00:00, 85.65batch/s]

Training batches on cuda:  99%|█████████▉| 159/161 [00:01<00:00, 86.38batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:22<00:02,  2.24s/epoch, loss=0.000522, prev_loss=0.000522]2021-09-16:19:26:00,851 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:26:01,977 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:26:16,889 INFO     [evaluator.py:657] Evaluation took 14.91s seconds
2021-09-16:19:26:17,97 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  7.09s/epoch, loss=0.000522, prev_loss=0.000522]
Training epochs on cuda: 100%|██████████| 10/10 [00:39<00:00,  3.90s/epoch, loss=0.000522, prev_loss=0.000522]
2021-09-16:19:26:17,98 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmplzc2gheg'
2021-09-16:19:26:17,155 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmplzc2gheg' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.43ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.49ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:26:31,957 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 19:26:31,963] Trial 83 finished with value: 0.008983548186342838 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.09028060214472658, 'optimizer.lr': 0.052102280184560416, 'negative_sampler.num_negs_per_pos': 5, 'training.batch_size': 1932}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:26:31,985 WARNING  [api.py:823] No random seed is specified. Setting to 825250765.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/217 [00:00<00:07, 28.35batch/s]

Training batches on cuda:   5%|▍         | 10/217 [00:00<00:06, 34.38batch/s]

Training batches on cuda:   8%|▊         | 18/217 [00:00<00:04, 41.20batch/s]

Training batches on cuda:  12%|█▏        | 26/217 [00:00<00:04, 47.68batch/s]

Training batches on cuda:  16%|█▌        | 34/217 [00:00<00:03, 53.84batch/s]

Training batches on cuda:  19%|█▉        | 42/217 [00:00<00:02, 58.97batch/s]

Training batches on cuda:  23%|██▎       | 50/217 [00:00<00:02, 62.79batch/s]

Training batches on cuda:  27%|██▋       | 59/217 [00:00<00:02, 67.68batch/s]

Training batches on cuda:  31%|███▏      | 68/217 [00:00<00:02, 72.21batch/s]

Training batches on cuda:  35%|███▌      | 77/217 [00:01<00:01, 74.43batch/s]

Training batches on cuda:  39%|███▉      | 85/217 [00:01<00:01, 75.00batch/s]

Training batches on cuda:  43%|████▎     | 93/217 [00:01<00:01, 74.76batch/s]

Training batches on cuda:  47%|████▋     | 101/217 [00:01<00:01, 74.70batch/s]

Training batches on cuda:  50%|█████     | 109/217 [00:01<00:01, 74.59batch/s]

Training batches on cuda:  54%|█████▍    | 117/217 [00:01<00:01, 67.91batch/s]

Training batches on cuda:  58%|█████▊    | 125/217 [00:01<00:01, 70.34batch/s]

Training batches on cuda:  61%|██████▏   | 133/217 [00:01<00:01, 69.50batch/s]

Training batches on cuda:  65%|██████▍   | 141/217 [00:01<00:01, 68.28batch/s]

Training batches on cuda:  69%|██████▊   | 149/217 [00:02<00:00, 70.83batch/s]

Training batches on cuda:  72%|███████▏  | 157/217 [00:02<00:00, 72.71batch/s]

Training batches on cuda:  76%|███████▌  | 165/217 [00:02<00:00, 73.83batch/s]

Training batches on cuda:  80%|███████▉  | 173/217 [00:02<00:00, 73.45batch/s]

Training batches on cuda:  84%|████████▍ | 182/217 [00:02<00:00, 76.56batch/s]

Training batches on cuda:  88%|████████▊ | 191/217 [00:02<00:00, 78.37batch/s]

Training batches on cuda:  92%|█████████▏| 200/217 [00:02<00:00, 80.24batch/s]

Training batches on cuda:  96%|█████████▋| 209/217 [00:02<00:00, 78.25batch/s]

Training batches on cuda: 100%|██████████| 217/217 [00:02<00:00, 77.33batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000698, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:28,  3.15s/epoch, loss=0.000698, prev_loss=nan]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/217 [00:00<00:04, 45.53batch/s]

Training batches on cuda:   6%|▋         | 14/217 [00:00<00:03, 52.39batch/s]

Training batches on cuda:  11%|█         | 23/217 [00:00<00:03, 58.63batch/s]

Training batches on cuda:  14%|█▍        | 31/217 [00:00<00:02, 62.58batch/s]

Training batches on cuda:  18%|█▊        | 38/217 [00:00<00:02, 64.11batch/s]

Training batches on cuda:  21%|██        | 46/217 [00:00<00:02, 66.37batch/s]

Training batches on cuda:  25%|██▌       | 55/217 [00:00<00:02, 71.09batch/s]

Training batches on cuda:  29%|██▉       | 64/217 [00:00<00:02, 73.71batch/s]

Training batches on cuda:  34%|███▎      | 73/217 [00:00<00:01, 75.55batch/s]

Training batches on cuda:  38%|███▊      | 82/217 [00:01<00:01, 77.05batch/s]

Training batches on cuda:  42%|████▏     | 91/217 [00:01<00:01, 78.82batch/s]

Training batches on cuda:  46%|████▌     | 100/217 [00:01<00:01, 80.38batch/s]

Training batches on cuda:  50%|█████     | 109/217 [00:01<00:01, 81.17batch/s]

Training batches on cuda:  54%|█████▍    | 118/217 [00:01<00:01, 81.70batch/s]

Training batches on cuda:  59%|█████▊    | 127/217 [00:01<00:01, 80.91batch/s]

Training batches on cuda:  63%|██████▎   | 136/217 [00:01<00:00, 81.94batch/s]

Training batches on cuda:  67%|██████▋   | 145/217 [00:01<00:00, 82.11batch/s]

Training batches on cuda:  71%|███████   | 154/217 [00:01<00:00, 82.31batch/s]

Training batches on cuda:  75%|███████▌  | 163/217 [00:02<00:00, 81.10batch/s]

Training batches on cuda:  79%|███████▉  | 172/217 [00:02<00:00, 80.03batch/s]

Training batches on cuda:  83%|████████▎ | 181/217 [00:02<00:00, 78.94batch/s]

Training batches on cuda:  87%|████████▋ | 189/217 [00:02<00:00, 78.40batch/s]

Training batches on cuda:  91%|█████████ | 197/217 [00:02<00:00, 77.91batch/s]

Training batches on cuda:  94%|█████████▍| 205/217 [00:02<00:00, 77.70batch/s]

Training batches on cuda:  98%|█████████▊| 213/217 [00:02<00:00, 77.59batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:28,  3.15s/epoch, loss=0.000685, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:24,  3.09s/epoch, loss=0.000685, prev_loss=nan]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/217 [00:00<00:04, 47.92batch/s]

Training batches on cuda:   6%|▋         | 14/217 [00:00<00:03, 55.01batch/s]

Training batches on cuda:  11%|█         | 23/217 [00:00<00:03, 61.67batch/s]

Training batches on cuda:  15%|█▍        | 32/217 [00:00<00:02, 67.05batch/s]

Training batches on cuda:  18%|█▊        | 39/217 [00:00<00:02, 62.38batch/s]

Training batches on cuda:  22%|██▏       | 48/217 [00:00<00:02, 67.32batch/s]

Training batches on cuda:  26%|██▋       | 57/217 [00:00<00:02, 71.57batch/s]

Training batches on cuda:  30%|███       | 66/217 [00:00<00:02, 75.35batch/s]

Training batches on cuda:  35%|███▍      | 75/217 [00:00<00:01, 77.90batch/s]

Training batches on cuda:  39%|███▊      | 84/217 [00:01<00:01, 79.40batch/s]

Training batches on cuda:  43%|████▎     | 93/217 [00:01<00:01, 81.04batch/s]

Training batches on cuda:  47%|████▋     | 102/217 [00:01<00:01, 82.36batch/s]

Training batches on cuda:  51%|█████     | 111/217 [00:01<00:01, 81.51batch/s]

Training batches on cuda:  55%|█████▌    | 120/217 [00:01<00:01, 78.85batch/s]

Training batches on cuda:  59%|█████▉    | 128/217 [00:01<00:01, 78.40batch/s]

Training batches on cuda:  63%|██████▎   | 136/217 [00:01<00:01, 77.77batch/s]

Training batches on cuda:  66%|██████▋   | 144/217 [00:01<00:00, 77.86batch/s]

Training batches on cuda:  71%|███████   | 153/217 [00:01<00:00, 80.19batch/s]

Training batches on cuda:  75%|███████▍  | 162/217 [00:02<00:00, 81.56batch/s]

Training batches on cuda:  79%|███████▉  | 171/217 [00:02<00:00, 78.75batch/s]

Training batches on cuda:  82%|████████▏ | 179/217 [00:02<00:00, 77.40batch/s]

Training batches on cuda:  86%|████████▌ | 187/217 [00:02<00:00, 77.93batch/s]

Training batches on cuda:  90%|█████████ | 196/217 [00:02<00:00, 78.82batch/s]

Training batches on cuda:  94%|█████████▍| 205/217 [00:02<00:00, 80.88batch/s]

Training batches on cuda:  99%|█████████▊| 214/217 [00:02<00:00, 80.45batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:24,  3.09s/epoch, loss=0.000643, prev_loss=0.000685]
Training epochs on cuda:  30%|███       | 3/10 [00:09<00:21,  3.07s/epoch, loss=0.000643, prev_loss=0.000685]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/217 [00:00<00:04, 49.41batch/s]

Training batches on cuda:   6%|▌         | 12/217 [00:00<00:03, 52.79batch/s]

Training batches on cuda:   8%|▊         | 18/217 [00:00<00:03, 51.66batch/s]

Training batches on cuda:  11%|█         | 24/217 [00:00<00:03, 53.68batch/s]

Training batches on cuda:  15%|█▍        | 32/217 [00:00<00:03, 58.92batch/s]

Training batches on cuda:  18%|█▊        | 40/217 [00:00<00:02, 63.82batch/s]

Training batches on cuda:  23%|██▎       | 49/217 [00:00<00:02, 68.44batch/s]

Training batches on cuda:  26%|██▋       | 57/217 [00:00<00:02, 71.27batch/s]

Training batches on cuda:  30%|███       | 66/217 [00:00<00:02, 74.16batch/s]

Training batches on cuda:  35%|███▍      | 75/217 [00:01<00:01, 77.46batch/s]

Training batches on cuda:  39%|███▊      | 84/217 [00:01<00:01, 79.43batch/s]

Training batches on cuda:  43%|████▎     | 93/217 [00:01<00:01, 80.82batch/s]

Training batches on cuda:  47%|████▋     | 102/217 [00:01<00:01, 81.99batch/s]

Training batches on cuda:  51%|█████     | 111/217 [00:01<00:01, 81.45batch/s]

Training batches on cuda:  55%|█████▌    | 120/217 [00:01<00:01, 78.35batch/s]

Training batches on cuda:  59%|█████▉    | 129/217 [00:01<00:01, 79.24batch/s]

Training batches on cuda:  64%|██████▎   | 138/217 [00:01<00:00, 79.79batch/s]

Training batches on cuda:  68%|██████▊   | 147/217 [00:01<00:00, 78.55batch/s]

Training batches on cuda:  71%|███████▏  | 155/217 [00:02<00:00, 78.17batch/s]

Training batches on cuda:  75%|███████▌  | 163/217 [00:02<00:00, 77.99batch/s]

Training batches on cuda:  79%|███████▉  | 171/217 [00:02<00:00, 77.48batch/s]

Training batches on cuda:  82%|████████▏ | 179/217 [00:02<00:00, 77.96batch/s]

Training batches on cuda:  87%|████████▋ | 188/217 [00:02<00:00, 80.60batch/s]

Training batches on cuda:  91%|█████████ | 197/217 [00:02<00:00, 82.09batch/s]

Training batches on cuda:  95%|█████████▍| 206/217 [00:02<00:00, 83.10batch/s]

Training batches on cuda:  99%|█████████▉| 215/217 [00:02<00:00, 84.24batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:21,  3.07s/epoch, loss=0.000468, prev_loss=0.000643]
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:18,  3.04s/epoch, loss=0.000468, prev_loss=0.000643]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/217 [00:00<00:03, 56.04batch/s]

Training batches on cuda:   7%|▋         | 15/217 [00:00<00:03, 62.92batch/s]

Training batches on cuda:  11%|█         | 24/217 [00:00<00:02, 68.59batch/s]

Training batches on cuda:  15%|█▌        | 33/217 [00:00<00:02, 73.21batch/s]

Training batches on cuda:  19%|█▉        | 42/217 [00:00<00:02, 76.56batch/s]

Training batches on cuda:  24%|██▎       | 51/217 [00:00<00:02, 79.06batch/s]

Training batches on cuda:  28%|██▊       | 60/217 [00:00<00:01, 80.83batch/s]

Training batches on cuda:  32%|███▏      | 69/217 [00:00<00:01, 82.70batch/s]

Training batches on cuda:  36%|███▌      | 78/217 [00:00<00:01, 83.46batch/s]

Training batches on cuda:  40%|████      | 87/217 [00:01<00:01, 83.53batch/s]

Training batches on cuda:  44%|████▍     | 96/217 [00:01<00:01, 84.12batch/s]

Training batches on cuda:  48%|████▊     | 105/217 [00:01<00:01, 84.49batch/s]

Training batches on cuda:  53%|█████▎    | 114/217 [00:01<00:01, 84.20batch/s]

Training batches on cuda:  57%|█████▋    | 123/217 [00:01<00:01, 85.13batch/s]

Training batches on cuda:  61%|██████    | 132/217 [00:01<00:00, 85.25batch/s]

Training batches on cuda:  65%|██████▍   | 141/217 [00:01<00:00, 85.34batch/s]

Training batches on cuda:  69%|██████▉   | 150/217 [00:01<00:00, 85.30batch/s]

Training batches on cuda:  73%|███████▎  | 159/217 [00:01<00:00, 83.28batch/s]

Training batches on cuda:  77%|███████▋  | 168/217 [00:02<00:00, 83.10batch/s]

Training batches on cuda:  82%|████████▏ | 177/217 [00:02<00:00, 80.86batch/s]

Training batches on cuda:  86%|████████▌ | 186/217 [00:02<00:00, 78.17batch/s]

Training batches on cuda:  89%|████████▉ | 194/217 [00:02<00:00, 77.48batch/s]

Training batches on cuda:  93%|█████████▎| 202/217 [00:02<00:00, 74.34batch/s]

Training batches on cuda:  97%|█████████▋| 210/217 [00:02<00:00, 73.51batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:18,  3.04s/epoch, loss=0.000323, prev_loss=0.000468]
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:14,  3.00s/epoch, loss=0.000323, prev_loss=0.000468]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/217 [00:00<00:03, 53.34batch/s]

Training batches on cuda:   6%|▋         | 14/217 [00:00<00:03, 58.05batch/s]

Training batches on cuda:  11%|█         | 23/217 [00:00<00:03, 63.49batch/s]

Training batches on cuda:  15%|█▍        | 32/217 [00:00<00:02, 68.84batch/s]

Training batches on cuda:  19%|█▉        | 41/217 [00:00<00:02, 73.30batch/s]

Training batches on cuda:  23%|██▎       | 50/217 [00:00<00:02, 76.06batch/s]

Training batches on cuda:  27%|██▋       | 58/217 [00:00<00:02, 76.69batch/s]

Training batches on cuda:  30%|███       | 66/217 [00:00<00:02, 71.31batch/s]

Training batches on cuda:  34%|███▍      | 74/217 [00:01<00:02, 70.16batch/s]

Training batches on cuda:  38%|███▊      | 82/217 [00:01<00:01, 71.42batch/s]

Training batches on cuda:  41%|████▏     | 90/217 [00:01<00:01, 71.95batch/s]

Training batches on cuda:  45%|████▌     | 98/217 [00:01<00:01, 72.77batch/s]

Training batches on cuda:  49%|████▉     | 106/217 [00:01<00:01, 72.48batch/s]

Training batches on cuda:  53%|█████▎    | 114/217 [00:01<00:01, 72.54batch/s]

Training batches on cuda:  56%|█████▌    | 122/217 [00:01<00:01, 64.27batch/s]

Training batches on cuda:  60%|██████    | 131/217 [00:01<00:01, 69.02batch/s]

Training batches on cuda:  65%|██████▍   | 140/217 [00:01<00:01, 72.75batch/s]

Training batches on cuda:  69%|██████▊   | 149/217 [00:02<00:00, 74.83batch/s]

Training batches on cuda:  72%|███████▏  | 157/217 [00:02<00:00, 74.69batch/s]

Training batches on cuda:  76%|███████▌  | 165/217 [00:02<00:00, 74.80batch/s]

Training batches on cuda:  80%|███████▉  | 173/217 [00:02<00:00, 75.27batch/s]

Training batches on cuda:  83%|████████▎ | 181/217 [00:02<00:00, 76.12batch/s]

Training batches on cuda:  88%|████████▊ | 190/217 [00:02<00:00, 78.84batch/s]

Training batches on cuda:  92%|█████████▏| 199/217 [00:02<00:00, 80.69batch/s]

Training batches on cuda:  96%|█████████▌| 208/217 [00:02<00:00, 79.48batch/s]

Training batches on cuda: 100%|██████████| 217/217 [00:02<00:00, 80.18batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:14,  3.00s/epoch, loss=0.000243, prev_loss=0.000323]
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:12,  3.02s/epoch, loss=0.000243, prev_loss=0.000323]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 3/217 [00:00<00:07, 27.63batch/s]

Training batches on cuda:   5%|▍         | 10/217 [00:00<00:06, 33.46batch/s]

Training batches on cuda:   8%|▊         | 17/217 [00:00<00:05, 39.51batch/s]

Training batches on cuda:  11%|█         | 24/217 [00:00<00:04, 44.89batch/s]

Training batches on cuda:  15%|█▍        | 32/217 [00:00<00:03, 50.74batch/s]

Training batches on cuda:  18%|█▊        | 40/217 [00:00<00:03, 55.75batch/s]

Training batches on cuda:  22%|██▏       | 48/217 [00:00<00:02, 59.52batch/s]

Training batches on cuda:  25%|██▌       | 55/217 [00:00<00:02, 62.25batch/s]

Training batches on cuda:  29%|██▉       | 63/217 [00:00<00:02, 64.85batch/s]

Training batches on cuda:  33%|███▎      | 71/217 [00:01<00:02, 67.25batch/s]

Training batches on cuda:  36%|███▋      | 79/217 [00:01<00:01, 70.23batch/s]

Training batches on cuda:  40%|████      | 87/217 [00:01<00:01, 72.27batch/s]

Training batches on cuda:  44%|████▍     | 96/217 [00:01<00:01, 74.63batch/s]

Training batches on cuda:  48%|████▊     | 105/217 [00:01<00:01, 77.12batch/s]

Training batches on cuda:  52%|█████▏    | 113/217 [00:01<00:01, 76.75batch/s]

Training batches on cuda:  56%|█████▌    | 121/217 [00:01<00:01, 77.31batch/s]

Training batches on cuda:  60%|█████▉    | 130/217 [00:01<00:01, 79.19batch/s]

Training batches on cuda:  64%|██████▍   | 139/217 [00:01<00:00, 80.16batch/s]

Training batches on cuda:  68%|██████▊   | 148/217 [00:02<00:00, 81.44batch/s]

Training batches on cuda:  72%|███████▏  | 157/217 [00:02<00:00, 82.78batch/s]

Training batches on cuda:  76%|███████▋  | 166/217 [00:02<00:00, 82.71batch/s]

Training batches on cuda:  81%|████████  | 175/217 [00:02<00:00, 82.65batch/s]

Training batches on cuda:  85%|████████▍ | 184/217 [00:02<00:00, 82.96batch/s]

Training batches on cuda:  89%|████████▉ | 193/217 [00:02<00:00, 82.51batch/s]

Training batches on cuda:  93%|█████████▎| 202/217 [00:02<00:00, 81.74batch/s]

Training batches on cuda:  97%|█████████▋| 211/217 [00:02<00:00, 81.85batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:21<00:12,  3.02s/epoch, loss=0.000218, prev_loss=0.000243]
Training epochs on cuda:  70%|███████   | 7/10 [00:21<00:09,  3.03s/epoch, loss=0.000218, prev_loss=0.000243]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/217 [00:00<00:04, 45.60batch/s]

Training batches on cuda:   6%|▌         | 12/217 [00:00<00:04, 50.41batch/s]

Training batches on cuda:   9%|▉         | 20/217 [00:00<00:03, 55.75batch/s]

Training batches on cuda:  13%|█▎        | 29/217 [00:00<00:03, 62.15batch/s]

Training batches on cuda:  16%|█▌        | 35/217 [00:00<00:03, 58.69batch/s]

Training batches on cuda:  20%|██        | 44/217 [00:00<00:02, 64.02batch/s]

Training batches on cuda:  24%|██▍       | 52/217 [00:00<00:02, 66.86batch/s]

Training batches on cuda:  28%|██▊       | 60/217 [00:00<00:02, 69.29batch/s]

Training batches on cuda:  31%|███▏      | 68/217 [00:00<00:02, 71.03batch/s]

Training batches on cuda:  35%|███▌      | 77/217 [00:01<00:01, 74.34batch/s]

Training batches on cuda:  40%|███▉      | 86/217 [00:01<00:01, 77.13batch/s]

Training batches on cuda:  44%|████▍     | 95/217 [00:01<00:01, 79.60batch/s]

Training batches on cuda:  48%|████▊     | 104/217 [00:01<00:01, 80.93batch/s]

Training batches on cuda:  52%|█████▏    | 113/217 [00:01<00:01, 82.44batch/s]

Training batches on cuda:  56%|█████▌    | 122/217 [00:01<00:01, 83.38batch/s]

Training batches on cuda:  60%|██████    | 131/217 [00:01<00:01, 83.75batch/s]

Training batches on cuda:  65%|██████▍   | 140/217 [00:01<00:00, 84.10batch/s]

Training batches on cuda:  69%|██████▊   | 149/217 [00:01<00:00, 84.44batch/s]

Training batches on cuda:  73%|███████▎  | 158/217 [00:02<00:00, 84.75batch/s]

Training batches on cuda:  77%|███████▋  | 167/217 [00:02<00:00, 85.21batch/s]

Training batches on cuda:  81%|████████  | 176/217 [00:02<00:00, 85.32batch/s]

Training batches on cuda:  85%|████████▌ | 185/217 [00:02<00:00, 85.74batch/s]

Training batches on cuda:  89%|████████▉ | 194/217 [00:02<00:00, 85.40batch/s]

Training batches on cuda:  94%|█████████▎| 203/217 [00:02<00:00, 85.33batch/s]

Training batches on cuda:  98%|█████████▊| 212/217 [00:02<00:00, 85.60batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:24<00:09,  3.03s/epoch, loss=0.00021, prev_loss=0.000218] 
Training epochs on cuda:  80%|████████  | 8/10 [00:24<00:05,  3.00s/epoch, loss=0.00021, prev_loss=0.000218]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/217 [00:00<00:03, 53.04batch/s]

Training batches on cuda:   6%|▋         | 14/217 [00:00<00:03, 57.34batch/s]

Training batches on cuda:  10%|█         | 22/217 [00:00<00:03, 62.15batch/s]

Training batches on cuda:  14%|█▍        | 30/217 [00:00<00:02, 66.05batch/s]

Training batches on cuda:  18%|█▊        | 39/217 [00:00<00:02, 71.04batch/s]

Training batches on cuda:  22%|██▏       | 48/217 [00:00<00:02, 75.09batch/s]

Training batches on cuda:  26%|██▋       | 57/217 [00:00<00:02, 78.21batch/s]

Training batches on cuda:  30%|███       | 66/217 [00:00<00:01, 80.62batch/s]

Training batches on cuda:  35%|███▍      | 75/217 [00:00<00:01, 82.43batch/s]

Training batches on cuda:  39%|███▊      | 84/217 [00:01<00:01, 83.46batch/s]

Training batches on cuda:  43%|████▎     | 93/217 [00:01<00:01, 84.01batch/s]

Training batches on cuda:  47%|████▋     | 102/217 [00:01<00:01, 84.22batch/s]

Training batches on cuda:  51%|█████     | 111/217 [00:01<00:01, 85.16batch/s]

Training batches on cuda:  55%|█████▌    | 120/217 [00:01<00:01, 85.41batch/s]

Training batches on cuda:  59%|█████▉    | 129/217 [00:01<00:01, 82.89batch/s]

Training batches on cuda:  64%|██████▎   | 138/217 [00:01<00:00, 80.82batch/s]

Training batches on cuda:  68%|██████▊   | 147/217 [00:01<00:00, 79.29batch/s]

Training batches on cuda:  71%|███████▏  | 155/217 [00:01<00:00, 78.84batch/s]

Training batches on cuda:  75%|███████▌  | 163/217 [00:02<00:00, 78.33batch/s]

Training batches on cuda:  79%|███████▉  | 171/217 [00:02<00:00, 76.15batch/s]

Training batches on cuda:  82%|████████▏ | 179/217 [00:02<00:00, 75.55batch/s]

Training batches on cuda:  86%|████████▌ | 187/217 [00:02<00:00, 74.38batch/s]

Training batches on cuda:  90%|████████▉ | 195/217 [00:02<00:00, 74.56batch/s]

Training batches on cuda:  94%|█████████▎| 203/217 [00:02<00:00, 73.85batch/s]

Training batches on cuda:  97%|█████████▋| 211/217 [00:02<00:00, 68.91batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:27<00:05,  3.00s/epoch, loss=0.000207, prev_loss=0.00021]
Training epochs on cuda:  90%|█████████ | 9/10 [00:27<00:03,  3.01s/epoch, loss=0.000207, prev_loss=0.00021]

Training batches on cuda:   0%|          | 0/217 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/217 [00:00<00:04, 49.58batch/s]

Training batches on cuda:   6%|▌         | 13/217 [00:00<00:03, 55.76batch/s]

Training batches on cuda:  10%|▉         | 21/217 [00:00<00:03, 61.05batch/s]

Training batches on cuda:  14%|█▍        | 30/217 [00:00<00:02, 65.34batch/s]

Training batches on cuda:  18%|█▊        | 38/217 [00:00<00:02, 67.08batch/s]

Training batches on cuda:  21%|██        | 46/217 [00:00<00:02, 69.47batch/s]

Training batches on cuda:  25%|██▍       | 54/217 [00:00<00:02, 71.33batch/s]

Training batches on cuda:  28%|██▊       | 61/217 [00:00<00:02, 70.01batch/s]

Training batches on cuda:  32%|███▏      | 69/217 [00:00<00:02, 72.58batch/s]

Training batches on cuda:  36%|███▌      | 78/217 [00:01<00:01, 75.69batch/s]

Training batches on cuda:  40%|████      | 87/217 [00:01<00:01, 78.08batch/s]

Training batches on cuda:  44%|████▍     | 95/217 [00:01<00:01, 74.41batch/s]

Training batches on cuda:  47%|████▋     | 103/217 [00:01<00:01, 74.03batch/s]

Training batches on cuda:  52%|█████▏    | 112/217 [00:01<00:01, 77.84batch/s]

Training batches on cuda:  55%|█████▌    | 120/217 [00:01<00:01, 76.28batch/s]

Training batches on cuda:  59%|█████▉    | 128/217 [00:01<00:01, 73.13batch/s]

Training batches on cuda:  63%|██████▎   | 136/217 [00:01<00:01, 71.37batch/s]

Training batches on cuda:  66%|██████▋   | 144/217 [00:01<00:01, 70.03batch/s]

Training batches on cuda:  70%|███████   | 152/217 [00:02<00:00, 69.23batch/s]

Training batches on cuda:  74%|███████▎  | 160/217 [00:02<00:00, 70.41batch/s]

Training batches on cuda:  77%|███████▋  | 168/217 [00:02<00:00, 71.63batch/s]

Training batches on cuda:  81%|████████  | 176/217 [00:02<00:00, 72.70batch/s]

Training batches on cuda:  85%|████████▍ | 184/217 [00:02<00:00, 73.62batch/s]

Training batches on cuda:  88%|████████▊ | 192/217 [00:02<00:00, 73.95batch/s]

Training batches on cuda:  92%|█████████▏| 200/217 [00:02<00:00, 74.59batch/s]

Training batches on cuda:  96%|█████████▌| 208/217 [00:02<00:00, 75.22batch/s]

Training batches on cuda: 100%|█████████▉| 216/217 [00:02<00:00, 75.76batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:30<00:03,  3.01s/epoch, loss=0.000206, prev_loss=0.000207]2021-09-16:19:27:02,667 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:27:03,717 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:27:18,633 INFO     [evaluator.py:657] Evaluation took 14.92s seconds
2021-09-16:19:27:18,755 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:46<00:00,  7.87s/epoch, loss=0.000206, prev_loss=0.000207]
Training epochs on cuda: 100%|██████████| 10/10 [00:46<00:00,  4.63s/epoch, loss=0.000206, prev_loss=0.000207]
2021-09-16:19:27:18,755 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp_8vk5k79'
2021-09-16:19:27:18,831 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp_8vk5k79' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.42ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.53ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.63ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.64ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.66ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.66ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.66ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.66ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.66ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.66ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.66ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.66ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.66ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.66ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.66ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.66ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:11<00:02, 2.66ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.66ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.66ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.66ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:27:33,617 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 19:27:33,622] Trial 84 finished with value: 0.9590269141520377 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.012920570518507172, 'optimizer.lr': 0.048483301858701755, 'negative_sampler.num_negs_per_pos': 14, 'training.batch_size': 1432}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:27:33,644 WARNING  [api.py:823] No random seed is specified. Setting to 175723287.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/203 [00:00<00:04, 39.86batch/s]

Training batches on cuda:   5%|▌         | 11/203 [00:00<00:04, 44.97batch/s]

Training batches on cuda:   8%|▊         | 16/203 [00:00<00:04, 44.96batch/s]

Training batches on cuda:  11%|█▏        | 23/203 [00:00<00:03, 50.20batch/s]

Training batches on cuda:  15%|█▌        | 31/203 [00:00<00:03, 55.01batch/s]

Training batches on cuda:  19%|█▉        | 39/203 [00:00<00:02, 60.56batch/s]

Training batches on cuda:  23%|██▎       | 47/203 [00:00<00:02, 65.26batch/s]

Training batches on cuda:  27%|██▋       | 55/203 [00:00<00:02, 68.91batch/s]

Training batches on cuda:  31%|███       | 63/203 [00:00<00:01, 71.52batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:01<00:01, 73.65batch/s]

Training batches on cuda:  39%|███▉      | 80/203 [00:01<00:01, 75.74batch/s]

Training batches on cuda:  44%|████▍     | 89/203 [00:01<00:01, 77.23batch/s]

Training batches on cuda:  48%|████▊     | 97/203 [00:01<00:01, 77.95batch/s]

Training batches on cuda:  52%|█████▏    | 105/203 [00:01<00:01, 74.63batch/s]

Training batches on cuda:  56%|█████▌    | 113/203 [00:01<00:01, 73.49batch/s]

Training batches on cuda:  60%|██████    | 122/203 [00:01<00:01, 75.71batch/s]

Training batches on cuda:  65%|██████▍   | 131/203 [00:01<00:00, 77.05batch/s]

Training batches on cuda:  69%|██████▉   | 140/203 [00:01<00:00, 77.92batch/s]

Training batches on cuda:  73%|███████▎  | 148/203 [00:02<00:00, 73.33batch/s]

Training batches on cuda:  77%|███████▋  | 156/203 [00:02<00:00, 71.31batch/s]

Training batches on cuda:  81%|████████  | 164/203 [00:02<00:00, 71.39batch/s]

Training batches on cuda:  85%|████████▌ | 173/203 [00:02<00:00, 74.19batch/s]

Training batches on cuda:  89%|████████▉ | 181/203 [00:02<00:00, 71.90batch/s]

Training batches on cuda:  94%|█████████▎| 190/203 [00:02<00:00, 74.11batch/s]

Training batches on cuda:  98%|█████████▊| 198/203 [00:02<00:00, 75.51batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000658, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:27,  3.00s/epoch, loss=0.000658, prev_loss=nan]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/203 [00:00<00:04, 48.12batch/s]

Training batches on cuda:   6%|▌         | 12/203 [00:00<00:03, 52.21batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:03, 56.26batch/s]

Training batches on cuda:  13%|█▎        | 26/203 [00:00<00:02, 59.66batch/s]

Training batches on cuda:  17%|█▋        | 34/203 [00:00<00:02, 62.61batch/s]

Training batches on cuda:  20%|██        | 41/203 [00:00<00:02, 64.33batch/s]

Training batches on cuda:  24%|██▍       | 49/203 [00:00<00:02, 65.93batch/s]

Training batches on cuda:  28%|██▊       | 56/203 [00:00<00:02, 66.58batch/s]

Training batches on cuda:  32%|███▏      | 64/203 [00:00<00:02, 68.20batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:01<00:02, 65.78batch/s]

Training batches on cuda:  38%|███▊      | 78/203 [00:01<00:01, 66.67batch/s]

Training batches on cuda:  42%|████▏     | 85/203 [00:01<00:01, 67.09batch/s]

Training batches on cuda:  46%|████▌     | 93/203 [00:01<00:01, 68.75batch/s]

Training batches on cuda:  50%|████▉     | 101/203 [00:01<00:01, 70.06batch/s]

Training batches on cuda:  54%|█████▎    | 109/203 [00:01<00:01, 70.87batch/s]

Training batches on cuda:  58%|█████▊    | 117/203 [00:01<00:01, 71.44batch/s]

Training batches on cuda:  62%|██████▏   | 125/203 [00:01<00:01, 72.69batch/s]

Training batches on cuda:  66%|██████▌   | 134/203 [00:01<00:00, 75.37batch/s]

Training batches on cuda:  70%|███████   | 143/203 [00:02<00:00, 76.78batch/s]

Training batches on cuda:  75%|███████▍  | 152/203 [00:02<00:00, 78.19batch/s]

Training batches on cuda:  79%|███████▉  | 161/203 [00:02<00:00, 79.32batch/s]

Training batches on cuda:  83%|████████▎ | 169/203 [00:02<00:00, 79.39batch/s]

Training batches on cuda:  87%|████████▋ | 177/203 [00:02<00:00, 79.25batch/s]

Training batches on cuda:  91%|█████████ | 185/203 [00:02<00:00, 78.90batch/s]

Training batches on cuda:  95%|█████████▌| 193/203 [00:02<00:00, 79.12batch/s]

Training batches on cuda:  99%|█████████▉| 201/203 [00:02<00:00, 71.98batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:06<00:27,  3.00s/epoch, loss=0.000656, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:06<00:24,  3.00s/epoch, loss=0.000656, prev_loss=nan]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/203 [00:00<00:03, 49.51batch/s]

Training batches on cuda:   6%|▋         | 13/203 [00:00<00:03, 55.33batch/s]

Training batches on cuda:  10%|█         | 21/203 [00:00<00:02, 60.79batch/s]

Training batches on cuda:  15%|█▍        | 30/203 [00:00<00:02, 65.44batch/s]

Training batches on cuda:  19%|█▊        | 38/203 [00:00<00:02, 68.99batch/s]

Training batches on cuda:  23%|██▎       | 46/203 [00:00<00:02, 71.95batch/s]

Training batches on cuda:  27%|██▋       | 54/203 [00:00<00:02, 72.68batch/s]

Training batches on cuda:  31%|███       | 62/203 [00:00<00:01, 72.95batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:00<00:01, 74.27batch/s]

Training batches on cuda:  39%|███▉      | 79/203 [00:01<00:01, 74.88batch/s]

Training batches on cuda:  43%|████▎     | 87/203 [00:01<00:01, 75.36batch/s]

Training batches on cuda:  47%|████▋     | 96/203 [00:01<00:01, 76.96batch/s]

Training batches on cuda:  52%|█████▏    | 105/203 [00:01<00:01, 78.49batch/s]

Training batches on cuda:  56%|█████▌    | 113/203 [00:01<00:01, 78.24batch/s]

Training batches on cuda:  60%|██████    | 122/203 [00:01<00:01, 79.35batch/s]

Training batches on cuda:  64%|██████▍   | 130/203 [00:01<00:00, 76.56batch/s]

Training batches on cuda:  68%|██████▊   | 138/203 [00:01<00:00, 74.18batch/s]

Training batches on cuda:  72%|███████▏  | 146/203 [00:01<00:00, 72.76batch/s]

Training batches on cuda:  76%|███████▌  | 154/203 [00:02<00:00, 72.06batch/s]

Training batches on cuda:  80%|███████▉  | 162/203 [00:02<00:00, 69.40batch/s]

Training batches on cuda:  84%|████████▎ | 170/203 [00:02<00:00, 69.70batch/s]

Training batches on cuda:  88%|████████▊ | 178/203 [00:02<00:00, 71.38batch/s]

Training batches on cuda:  92%|█████████▏| 186/203 [00:02<00:00, 73.13batch/s]

Training batches on cuda:  96%|█████████▌| 194/203 [00:02<00:00, 74.16batch/s]

Training batches on cuda: 100%|█████████▉| 202/203 [00:02<00:00, 75.23batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:24,  3.00s/epoch, loss=0.000656, prev_loss=0.000656]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:20,  2.98s/epoch, loss=0.000656, prev_loss=0.000656]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/203 [00:00<00:03, 52.31batch/s]

Training batches on cuda:   7%|▋         | 14/203 [00:00<00:03, 56.58batch/s]

Training batches on cuda:  10%|█         | 21/203 [00:00<00:03, 59.86batch/s]

Training batches on cuda:  14%|█▍        | 29/203 [00:00<00:02, 63.10batch/s]

Training batches on cuda:  18%|█▊        | 37/203 [00:00<00:02, 65.46batch/s]

Training batches on cuda:  22%|██▏       | 45/203 [00:00<00:02, 67.72batch/s]

Training batches on cuda:  26%|██▌       | 53/203 [00:00<00:02, 69.36batch/s]

Training batches on cuda:  30%|███       | 61/203 [00:00<00:02, 70.64batch/s]

Training batches on cuda:  34%|███▍      | 69/203 [00:00<00:01, 71.51batch/s]

Training batches on cuda:  38%|███▊      | 77/203 [00:01<00:01, 71.85batch/s]

Training batches on cuda:  42%|████▏     | 85/203 [00:01<00:01, 71.95batch/s]

Training batches on cuda:  46%|████▌     | 93/203 [00:01<00:01, 69.53batch/s]

Training batches on cuda:  49%|████▉     | 100/203 [00:01<00:01, 68.76batch/s]

Training batches on cuda:  54%|█████▎    | 109/203 [00:01<00:01, 71.91batch/s]

Training batches on cuda:  58%|█████▊    | 118/203 [00:01<00:01, 74.15batch/s]

Training batches on cuda:  62%|██████▏   | 126/203 [00:01<00:01, 75.60batch/s]

Training batches on cuda:  66%|██████▌   | 134/203 [00:01<00:01, 66.87batch/s]

Training batches on cuda:  70%|██████▉   | 142/203 [00:02<00:00, 68.08batch/s]

Training batches on cuda:  73%|███████▎  | 149/203 [00:02<00:00, 67.24batch/s]

Training batches on cuda:  77%|███████▋  | 156/203 [00:02<00:00, 65.75batch/s]

Training batches on cuda:  81%|████████  | 164/203 [00:02<00:00, 67.26batch/s]

Training batches on cuda:  85%|████████▍ | 172/203 [00:02<00:00, 68.61batch/s]

Training batches on cuda:  88%|████████▊ | 179/203 [00:02<00:00, 67.64batch/s]

Training batches on cuda:  92%|█████████▏| 187/203 [00:02<00:00, 67.84batch/s]

Training batches on cuda:  96%|█████████▌| 194/203 [00:02<00:00, 66.91batch/s]

Training batches on cuda:  99%|█████████▉| 201/203 [00:02<00:00, 67.42batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:20,  2.98s/epoch, loss=0.000656, prev_loss=0.000656]
Training epochs on cuda:  40%|████      | 4/10 [00:12<00:18,  3.05s/epoch, loss=0.000656, prev_loss=0.000656]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/203 [00:00<00:04, 46.99batch/s]

Training batches on cuda:   6%|▌         | 12/203 [00:00<00:03, 51.56batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:03, 55.98batch/s]

Training batches on cuda:  13%|█▎        | 27/203 [00:00<00:02, 58.76batch/s]

Training batches on cuda:  17%|█▋        | 34/203 [00:00<00:02, 59.35batch/s]

Training batches on cuda:  20%|██        | 41/203 [00:00<00:02, 60.67batch/s]

Training batches on cuda:  23%|██▎       | 47/203 [00:00<00:02, 59.07batch/s]

Training batches on cuda:  26%|██▌       | 53/203 [00:00<00:02, 58.96batch/s]

Training batches on cuda:  30%|███       | 61/203 [00:00<00:02, 62.19batch/s]

Training batches on cuda:  34%|███▍      | 69/203 [00:01<00:02, 65.14batch/s]

Training batches on cuda:  38%|███▊      | 77/203 [00:01<00:01, 67.52batch/s]

Training batches on cuda:  42%|████▏     | 85/203 [00:01<00:01, 68.67batch/s]

Training batches on cuda:  45%|████▌     | 92/203 [00:01<00:01, 68.79batch/s]

Training batches on cuda:  49%|████▉     | 100/203 [00:01<00:01, 69.59batch/s]

Training batches on cuda:  53%|█████▎    | 108/203 [00:01<00:01, 70.00batch/s]

Training batches on cuda:  57%|█████▋    | 116/203 [00:01<00:01, 70.70batch/s]

Training batches on cuda:  61%|██████    | 124/203 [00:01<00:01, 70.62batch/s]

Training batches on cuda:  65%|██████▌   | 132/203 [00:01<00:01, 69.30batch/s]

Training batches on cuda:  69%|██████▉   | 140/203 [00:02<00:00, 70.15batch/s]

Training batches on cuda:  73%|███████▎  | 148/203 [00:02<00:00, 72.35batch/s]

Training batches on cuda:  77%|███████▋  | 156/203 [00:02<00:00, 73.70batch/s]

Training batches on cuda:  81%|████████  | 164/203 [00:02<00:00, 67.19batch/s]

Training batches on cuda:  84%|████████▍ | 171/203 [00:02<00:00, 67.01batch/s]

Training batches on cuda:  88%|████████▊ | 179/203 [00:02<00:00, 68.83batch/s]

Training batches on cuda:  92%|█████████▏| 187/203 [00:02<00:00, 70.26batch/s]

Training batches on cuda:  96%|█████████▌| 195/203 [00:02<00:00, 67.03batch/s]

Training batches on cuda: 100%|█████████▉| 202/203 [00:03<00:00, 64.61batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:18,  3.05s/epoch, loss=0.000657, prev_loss=0.000656]
Training epochs on cuda:  50%|█████     | 5/10 [00:15<00:15,  3.11s/epoch, loss=0.000657, prev_loss=0.000656]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/203 [00:00<00:04, 45.14batch/s]

Training batches on cuda:   6%|▌         | 12/203 [00:00<00:03, 50.22batch/s]

Training batches on cuda:   9%|▉         | 18/203 [00:00<00:03, 52.48batch/s]

Training batches on cuda:  11%|█▏        | 23/203 [00:00<00:03, 51.39batch/s]

Training batches on cuda:  15%|█▍        | 30/203 [00:00<00:03, 55.51batch/s]

Training batches on cuda:  18%|█▊        | 36/203 [00:00<00:03, 55.04batch/s]

Training batches on cuda:  21%|██        | 43/203 [00:00<00:02, 57.74batch/s]

Training batches on cuda:  24%|██▍       | 49/203 [00:00<00:02, 57.84batch/s]

Training batches on cuda:  28%|██▊       | 56/203 [00:00<00:02, 60.26batch/s]

Training batches on cuda:  31%|███       | 62/203 [00:01<00:02, 59.42batch/s]

Training batches on cuda:  34%|███▍      | 69/203 [00:01<00:02, 62.00batch/s]

Training batches on cuda:  38%|███▊      | 77/203 [00:01<00:01, 64.32batch/s]

Training batches on cuda:  41%|████▏     | 84/203 [00:01<00:01, 63.99batch/s]

Training batches on cuda:  45%|████▍     | 91/203 [00:01<00:01, 61.50batch/s]

Training batches on cuda:  48%|████▊     | 98/203 [00:01<00:01, 59.91batch/s]

Training batches on cuda:  52%|█████▏    | 105/203 [00:01<00:01, 59.48batch/s]

Training batches on cuda:  55%|█████▌    | 112/203 [00:01<00:01, 61.02batch/s]

Training batches on cuda:  59%|█████▉    | 120/203 [00:01<00:01, 63.30batch/s]

Training batches on cuda:  63%|██████▎   | 127/203 [00:02<00:01, 63.44batch/s]

Training batches on cuda:  66%|██████▌   | 134/203 [00:02<00:01, 64.25batch/s]

Training batches on cuda:  69%|██████▉   | 141/203 [00:02<00:00, 62.70batch/s]

Training batches on cuda:  73%|███████▎  | 148/203 [00:02<00:00, 62.90batch/s]

Training batches on cuda:  76%|███████▋  | 155/203 [00:02<00:00, 63.02batch/s]

Training batches on cuda:  80%|████████  | 163/203 [00:02<00:00, 65.71batch/s]

Training batches on cuda:  84%|████████▍ | 171/203 [00:02<00:00, 67.85batch/s]

Training batches on cuda:  88%|████████▊ | 178/203 [00:02<00:00, 67.05batch/s]

Training batches on cuda:  91%|█████████ | 185/203 [00:02<00:00, 66.70batch/s]

Training batches on cuda:  95%|█████████▍| 192/203 [00:03<00:00, 65.87batch/s]

Training batches on cuda:  99%|█████████▊| 200/203 [00:03<00:00, 67.14batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:18<00:15,  3.11s/epoch, loss=0.000657, prev_loss=0.000657]
Training epochs on cuda:  60%|██████    | 6/10 [00:18<00:12,  3.20s/epoch, loss=0.000657, prev_loss=0.000657]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/203 [00:00<00:04, 47.78batch/s]

Training batches on cuda:   6%|▋         | 13/203 [00:00<00:03, 53.37batch/s]

Training batches on cuda:  10%|█         | 21/203 [00:00<00:03, 57.69batch/s]

Training batches on cuda:  14%|█▍        | 29/203 [00:00<00:02, 62.04batch/s]

Training batches on cuda:  18%|█▊        | 37/203 [00:00<00:02, 65.24batch/s]

Training batches on cuda:  23%|██▎       | 46/203 [00:00<00:02, 69.11batch/s]

Training batches on cuda:  27%|██▋       | 55/203 [00:00<00:02, 72.98batch/s]

Training batches on cuda:  31%|███       | 63/203 [00:00<00:01, 74.81batch/s]

Training batches on cuda:  35%|███▍      | 71/203 [00:00<00:01, 76.00batch/s]

Training batches on cuda:  39%|███▉      | 79/203 [00:01<00:01, 76.71batch/s]

Training batches on cuda:  43%|████▎     | 87/203 [00:01<00:01, 75.55batch/s]

Training batches on cuda:  47%|████▋     | 95/203 [00:01<00:01, 75.39batch/s]

Training batches on cuda:  51%|█████     | 103/203 [00:01<00:01, 73.62batch/s]

Training batches on cuda:  55%|█████▍    | 111/203 [00:01<00:01, 74.85batch/s]

Training batches on cuda:  59%|█████▊    | 119/203 [00:01<00:01, 75.30batch/s]

Training batches on cuda:  63%|██████▎   | 127/203 [00:01<00:00, 76.05batch/s]

Training batches on cuda:  67%|██████▋   | 135/203 [00:01<00:00, 76.44batch/s]

Training batches on cuda:  70%|███████   | 143/203 [00:01<00:00, 75.38batch/s]

Training batches on cuda:  74%|███████▍  | 151/203 [00:02<00:00, 69.00batch/s]

Training batches on cuda:  78%|███████▊  | 159/203 [00:02<00:00, 70.42batch/s]

Training batches on cuda:  82%|████████▏ | 167/203 [00:02<00:00, 71.61batch/s]

Training batches on cuda:  86%|████████▌ | 175/203 [00:02<00:00, 72.07batch/s]

Training batches on cuda:  90%|█████████ | 183/203 [00:02<00:00, 72.16batch/s]

Training batches on cuda:  94%|█████████▍| 191/203 [00:02<00:00, 70.90batch/s]

Training batches on cuda:  98%|█████████▊| 199/203 [00:02<00:00, 69.86batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:21<00:12,  3.20s/epoch, loss=0.000657, prev_loss=0.000657]
Training epochs on cuda:  70%|███████   | 7/10 [00:21<00:09,  3.15s/epoch, loss=0.000657, prev_loss=0.000657]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/203 [00:00<00:05, 35.47batch/s]

Training batches on cuda:   6%|▌         | 12/203 [00:00<00:04, 41.96batch/s]

Training batches on cuda:   9%|▉         | 19/203 [00:00<00:03, 47.10batch/s]

Training batches on cuda:  13%|█▎        | 27/203 [00:00<00:03, 53.64batch/s]

Training batches on cuda:  18%|█▊        | 36/203 [00:00<00:02, 60.04batch/s]

Training batches on cuda:  22%|██▏       | 45/203 [00:00<00:02, 65.15batch/s]

Training batches on cuda:  27%|██▋       | 54/203 [00:00<00:02, 69.28batch/s]

Training batches on cuda:  31%|███       | 63/203 [00:00<00:01, 72.91batch/s]

Training batches on cuda:  35%|███▌      | 72/203 [00:00<00:01, 75.66batch/s]

Training batches on cuda:  39%|███▉      | 80/203 [00:01<00:01, 76.67batch/s]

Training batches on cuda:  43%|████▎     | 88/203 [00:01<00:01, 77.60batch/s]

Training batches on cuda:  48%|████▊     | 97/203 [00:01<00:01, 78.68batch/s]

Training batches on cuda:  52%|█████▏    | 106/203 [00:01<00:01, 79.16batch/s]

Training batches on cuda:  56%|█████▌    | 114/203 [00:01<00:01, 78.44batch/s]

Training batches on cuda:  60%|██████    | 122/203 [00:01<00:01, 77.32batch/s]

Training batches on cuda:  64%|██████▍   | 130/203 [00:01<00:00, 76.52batch/s]

Training batches on cuda:  68%|██████▊   | 138/203 [00:01<00:00, 74.81batch/s]

Training batches on cuda:  72%|███████▏  | 146/203 [00:01<00:00, 74.94batch/s]

Training batches on cuda:  76%|███████▌  | 154/203 [00:02<00:00, 75.44batch/s]

Training batches on cuda:  80%|███████▉  | 162/203 [00:02<00:00, 76.56batch/s]

Training batches on cuda:  84%|████████▍ | 171/203 [00:02<00:00, 77.57batch/s]

Training batches on cuda:  88%|████████▊ | 179/203 [00:02<00:00, 77.58batch/s]

Training batches on cuda:  92%|█████████▏| 187/203 [00:02<00:00, 73.53batch/s]

Training batches on cuda:  96%|█████████▌| 195/203 [00:02<00:00, 72.09batch/s]

Training batches on cuda: 100%|██████████| 203/203 [00:02<00:00, 69.66batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:24<00:09,  3.15s/epoch, loss=0.000656, prev_loss=0.000657]
Training epochs on cuda:  80%|████████  | 8/10 [00:24<00:06,  3.09s/epoch, loss=0.000656, prev_loss=0.000657]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/203 [00:00<00:05, 38.56batch/s]

Training batches on cuda:   5%|▌         | 11/203 [00:00<00:04, 44.37batch/s]

Training batches on cuda:   9%|▉         | 18/203 [00:00<00:03, 49.72batch/s]

Training batches on cuda:  12%|█▏        | 25/203 [00:00<00:03, 54.34batch/s]

Training batches on cuda:  16%|█▌        | 32/203 [00:00<00:02, 57.85batch/s]

Training batches on cuda:  19%|█▉        | 39/203 [00:00<00:02, 60.29batch/s]

Training batches on cuda:  23%|██▎       | 47/203 [00:00<00:02, 62.97batch/s]

Training batches on cuda:  27%|██▋       | 55/203 [00:00<00:02, 65.09batch/s]

Training batches on cuda:  31%|███       | 62/203 [00:00<00:02, 65.27batch/s]

Training batches on cuda:  34%|███▍      | 69/203 [00:01<00:02, 66.59batch/s]

Training batches on cuda:  38%|███▊      | 77/203 [00:01<00:01, 67.81batch/s]

Training batches on cuda:  42%|████▏     | 85/203 [00:01<00:01, 68.75batch/s]

Training batches on cuda:  46%|████▌     | 93/203 [00:01<00:01, 69.11batch/s]

Training batches on cuda:  50%|████▉     | 101/203 [00:01<00:01, 69.46batch/s]

Training batches on cuda:  54%|█████▎    | 109/203 [00:01<00:01, 69.90batch/s]

Training batches on cuda:  57%|█████▋    | 116/203 [00:01<00:01, 69.34batch/s]

Training batches on cuda:  61%|██████    | 124/203 [00:01<00:01, 69.75batch/s]

Training batches on cuda:  65%|██████▌   | 132/203 [00:01<00:01, 70.35batch/s]

Training batches on cuda:  69%|██████▉   | 140/203 [00:02<00:00, 69.60batch/s]

Training batches on cuda:  72%|███████▏  | 147/203 [00:02<00:00, 67.62batch/s]

Training batches on cuda:  76%|███████▋  | 155/203 [00:02<00:00, 68.60batch/s]

Training batches on cuda:  80%|████████  | 163/203 [00:02<00:00, 69.20batch/s]

Training batches on cuda:  84%|████████▎ | 170/203 [00:02<00:00, 68.87batch/s]

Training batches on cuda:  87%|████████▋ | 177/203 [00:02<00:00, 68.18batch/s]

Training batches on cuda:  91%|█████████ | 184/203 [00:02<00:00, 67.73batch/s]

Training batches on cuda:  94%|█████████▍| 191/203 [00:02<00:00, 67.45batch/s]

Training batches on cuda:  98%|█████████▊| 198/203 [00:02<00:00, 67.59batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:27<00:06,  3.09s/epoch, loss=0.000657, prev_loss=0.000656]
Training epochs on cuda:  90%|█████████ | 9/10 [00:27<00:03,  3.12s/epoch, loss=0.000657, prev_loss=0.000656]

Training batches on cuda:   0%|          | 0/203 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 5/203 [00:00<00:04, 49.06batch/s]

Training batches on cuda:   6%|▌         | 12/203 [00:00<00:03, 52.03batch/s]

Training batches on cuda:   9%|▉         | 18/203 [00:00<00:03, 52.95batch/s]

Training batches on cuda:  12%|█▏        | 25/203 [00:00<00:03, 55.86batch/s]

Training batches on cuda:  16%|█▌        | 32/203 [00:00<00:02, 58.81batch/s]

Training batches on cuda:  19%|█▉        | 39/203 [00:00<00:02, 61.42batch/s]

Training batches on cuda:  23%|██▎       | 46/203 [00:00<00:02, 62.91batch/s]

Training batches on cuda:  26%|██▌       | 52/203 [00:00<00:02, 59.21batch/s]

Training batches on cuda:  29%|██▊       | 58/203 [00:00<00:02, 59.16batch/s]

Training batches on cuda:  33%|███▎      | 66/203 [00:01<00:02, 62.11batch/s]

Training batches on cuda:  36%|███▋      | 74/203 [00:01<00:01, 65.46batch/s]

Training batches on cuda:  40%|████      | 82/203 [00:01<00:01, 68.02batch/s]

Training batches on cuda:  45%|████▍     | 91/203 [00:01<00:01, 71.40batch/s]

Training batches on cuda:  49%|████▉     | 100/203 [00:01<00:01, 73.99batch/s]

Training batches on cuda:  53%|█████▎    | 108/203 [00:01<00:01, 74.21batch/s]

Training batches on cuda:  57%|█████▋    | 116/203 [00:01<00:01, 72.22batch/s]

Training batches on cuda:  61%|██████    | 124/203 [00:01<00:01, 64.86batch/s]

Training batches on cuda:  65%|██████▍   | 131/203 [00:01<00:01, 65.70batch/s]

Training batches on cuda:  68%|██████▊   | 138/203 [00:02<00:00, 66.21batch/s]

Training batches on cuda:  72%|███████▏  | 146/203 [00:02<00:00, 67.98batch/s]

Training batches on cuda:  76%|███████▌  | 154/203 [00:02<00:00, 71.00batch/s]

Training batches on cuda:  80%|███████▉  | 162/203 [00:02<00:00, 72.43batch/s]

Training batches on cuda:  84%|████████▎ | 170/203 [00:02<00:00, 72.78batch/s]

Training batches on cuda:  88%|████████▊ | 178/203 [00:02<00:00, 72.97batch/s]

Training batches on cuda:  92%|█████████▏| 186/203 [00:02<00:00, 72.19batch/s]

Training batches on cuda:  96%|█████████▌| 194/203 [00:02<00:00, 64.53batch/s]

Training batches on cuda: 100%|█████████▉| 202/203 [00:03<00:00, 66.22batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:31<00:03,  3.12s/epoch, loss=0.000654, prev_loss=0.000657]2021-09-16:19:28:05,428 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:28:06,460 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:28:21,429 INFO     [evaluator.py:657] Evaluation took 14.97s seconds
2021-09-16:19:28:21,540 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00,  8.01s/epoch, loss=0.000654, prev_loss=0.000657]
Training epochs on cuda: 100%|██████████| 10/10 [00:47<00:00,  4.74s/epoch, loss=0.000654, prev_loss=0.000657]
2021-09-16:19:28:21,540 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpe2j_yfvt'
2021-09-16:19:28:21,641 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpe2j_yfvt' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.62ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:28:36,423 INFO     [evaluator.py:657] Evaluation took 14.76s seconds
[I 2021-09-16 19:28:36,429] Trial 85 finished with value: 0.0166960894550634 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.06306592919507666, 'optimizer.lr': 0.0500964363927127, 'negative_sampler.num_negs_per_pos': 14, 'training.batch_size': 1532}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:28:36,451 WARNING  [api.py:823] No random seed is specified. Setting to 2756475732.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 47.97batch/s]

Training batches on cuda:   7%|▋         | 12/161 [00:00<00:02, 51.76batch/s]

Training batches on cuda:  11%|█         | 18/161 [00:00<00:02, 52.74batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:02, 54.92batch/s]

Training batches on cuda:  20%|█▉        | 32/161 [00:00<00:02, 57.06batch/s]

Training batches on cuda:  24%|██▍       | 39/161 [00:00<00:02, 58.39batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:01, 59.47batch/s]

Training batches on cuda:  33%|███▎      | 53/161 [00:00<00:01, 60.22batch/s]

Training batches on cuda:  37%|███▋      | 60/161 [00:01<00:01, 60.77batch/s]

Training batches on cuda:  42%|████▏     | 67/161 [00:01<00:01, 62.94batch/s]

Training batches on cuda:  46%|████▌     | 74/161 [00:01<00:01, 64.71batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:01<00:01, 65.79batch/s]

Training batches on cuda:  55%|█████▍    | 88/161 [00:01<00:01, 66.31batch/s]

Training batches on cuda:  59%|█████▉    | 95/161 [00:01<00:00, 66.49batch/s]

Training batches on cuda:  63%|██████▎   | 102/161 [00:01<00:00, 66.72batch/s]

Training batches on cuda:  68%|██████▊   | 109/161 [00:01<00:00, 67.12batch/s]

Training batches on cuda:  72%|███████▏  | 116/161 [00:01<00:00, 67.31batch/s]

Training batches on cuda:  76%|███████▋  | 123/161 [00:01<00:00, 67.12batch/s]

Training batches on cuda:  81%|████████  | 130/161 [00:02<00:00, 67.14batch/s]

Training batches on cuda:  85%|████████▌ | 137/161 [00:02<00:00, 67.73batch/s]

Training batches on cuda:  89%|████████▉ | 144/161 [00:02<00:00, 67.99batch/s]

Training batches on cuda:  94%|█████████▍| 151/161 [00:02<00:00, 67.86batch/s]

Training batches on cuda:  98%|█████████▊| 158/161 [00:02<00:00, 67.10batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000571, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:24,  2.68s/epoch, loss=0.000571, prev_loss=nan]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/161 [00:00<00:04, 35.93batch/s]

Training batches on cuda:   7%|▋         | 11/161 [00:00<00:03, 41.60batch/s]

Training batches on cuda:  11%|█         | 18/161 [00:00<00:03, 46.91batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:02, 51.41batch/s]

Training batches on cuda:  20%|█▉        | 32/161 [00:00<00:02, 55.33batch/s]

Training batches on cuda:  24%|██▍       | 39/161 [00:00<00:02, 58.36batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:01, 60.44batch/s]

Training batches on cuda:  33%|███▎      | 53/161 [00:00<00:01, 62.02batch/s]

Training batches on cuda:  37%|███▋      | 60/161 [00:00<00:01, 63.34batch/s]

Training batches on cuda:  42%|████▏     | 67/161 [00:01<00:01, 64.84batch/s]

Training batches on cuda:  46%|████▌     | 74/161 [00:01<00:01, 63.53batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:01<00:01, 61.36batch/s]

Training batches on cuda:  55%|█████▍    | 88/161 [00:01<00:01, 60.02batch/s]

Training batches on cuda:  59%|█████▉    | 95/161 [00:01<00:01, 59.42batch/s]

Training batches on cuda:  63%|██████▎   | 101/161 [00:01<00:01, 58.95batch/s]

Training batches on cuda:  66%|██████▋   | 107/161 [00:01<00:00, 58.69batch/s]

Training batches on cuda:  70%|███████   | 113/161 [00:01<00:00, 57.33batch/s]

Training batches on cuda:  74%|███████▍  | 119/161 [00:01<00:00, 55.08batch/s]

Training batches on cuda:  78%|███████▊  | 126/161 [00:02<00:00, 58.52batch/s]

Training batches on cuda:  83%|████████▎ | 133/161 [00:02<00:00, 60.95batch/s]

Training batches on cuda:  87%|████████▋ | 140/161 [00:02<00:00, 62.88batch/s]

Training batches on cuda:  91%|█████████▏| 147/161 [00:02<00:00, 63.73batch/s]

Training batches on cuda:  96%|█████████▌| 154/161 [00:02<00:00, 65.06batch/s]

Training batches on cuda: 100%|██████████| 161/161 [00:02<00:00, 64.97batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:24,  2.68s/epoch, loss=0.00054, prev_loss=nan] 
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:21,  2.72s/epoch, loss=0.00054, prev_loss=nan]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/161 [00:00<00:08, 17.73batch/s]

Training batches on cuda:   6%|▌         | 9/161 [00:00<00:06, 22.65batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:05, 28.28batch/s]

Training batches on cuda:  14%|█▍        | 23/161 [00:00<00:04, 34.15batch/s]

Training batches on cuda:  19%|█▊        | 30/161 [00:00<00:03, 40.16batch/s]

Training batches on cuda:  23%|██▎       | 37/161 [00:00<00:02, 45.64batch/s]

Training batches on cuda:  27%|██▋       | 44/161 [00:00<00:02, 50.52batch/s]

Training batches on cuda:  32%|███▏      | 51/161 [00:00<00:02, 53.84batch/s]

Training batches on cuda:  36%|███▌      | 58/161 [00:00<00:01, 57.02batch/s]

Training batches on cuda:  40%|████      | 65/161 [00:01<00:01, 58.82batch/s]

Training batches on cuda:  45%|████▍     | 72/161 [00:01<00:01, 60.84batch/s]

Training batches on cuda:  49%|████▉     | 79/161 [00:01<00:01, 63.16batch/s]

Training batches on cuda:  53%|█████▎    | 86/161 [00:01<00:01, 64.55batch/s]

Training batches on cuda:  58%|█████▊    | 93/161 [00:01<00:01, 65.92batch/s]

Training batches on cuda:  62%|██████▏   | 100/161 [00:01<00:00, 66.62batch/s]

Training batches on cuda:  67%|██████▋   | 108/161 [00:01<00:00, 67.72batch/s]

Training batches on cuda:  72%|███████▏  | 116/161 [00:01<00:00, 68.58batch/s]

Training batches on cuda:  76%|███████▋  | 123/161 [00:01<00:00, 68.53batch/s]

Training batches on cuda:  81%|████████  | 130/161 [00:02<00:00, 67.77batch/s]

Training batches on cuda:  86%|████████▌ | 138/161 [00:02<00:00, 68.49batch/s]

Training batches on cuda:  90%|█████████ | 145/161 [00:02<00:00, 68.62batch/s]

Training batches on cuda:  94%|█████████▍| 152/161 [00:02<00:00, 68.88batch/s]

Training batches on cuda:  99%|█████████▉| 160/161 [00:02<00:00, 69.28batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:21,  2.72s/epoch, loss=0.000541, prev_loss=0.00054]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:18,  2.69s/epoch, loss=0.000541, prev_loss=0.00054]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 52.79batch/s]

Training batches on cuda:   9%|▊         | 14/161 [00:00<00:02, 57.18batch/s]

Training batches on cuda:  13%|█▎        | 21/161 [00:00<00:02, 60.22batch/s]

Training batches on cuda:  17%|█▋        | 28/161 [00:00<00:02, 62.39batch/s]

Training batches on cuda:  21%|██        | 34/161 [00:00<00:02, 57.88batch/s]

Training batches on cuda:  25%|██▌       | 41/161 [00:00<00:01, 60.69batch/s]

Training batches on cuda:  30%|██▉       | 48/161 [00:00<00:01, 62.73batch/s]

Training batches on cuda:  34%|███▍      | 55/161 [00:00<00:01, 64.60batch/s]

Training batches on cuda:  39%|███▉      | 63/161 [00:00<00:01, 66.33batch/s]

Training batches on cuda:  44%|████▍     | 71/161 [00:01<00:01, 67.51batch/s]

Training batches on cuda:  49%|████▉     | 79/161 [00:01<00:01, 68.10batch/s]

Training batches on cuda:  54%|█████▍    | 87/161 [00:01<00:01, 68.81batch/s]

Training batches on cuda:  59%|█████▉    | 95/161 [00:01<00:00, 69.50batch/s]

Training batches on cuda:  64%|██████▍   | 103/161 [00:01<00:00, 69.75batch/s]

Training batches on cuda:  69%|██████▉   | 111/161 [00:01<00:00, 70.17batch/s]

Training batches on cuda:  74%|███████▍  | 119/161 [00:01<00:00, 70.17batch/s]

Training batches on cuda:  79%|███████▉  | 127/161 [00:01<00:00, 70.54batch/s]

Training batches on cuda:  84%|████████▍ | 135/161 [00:01<00:00, 70.68batch/s]

Training batches on cuda:  89%|████████▉ | 143/161 [00:02<00:00, 70.94batch/s]

Training batches on cuda:  94%|█████████▍| 151/161 [00:02<00:00, 70.81batch/s]

Training batches on cuda:  99%|█████████▉| 159/161 [00:02<00:00, 70.71batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:18,  2.69s/epoch, loss=0.000543, prev_loss=0.000541]
Training epochs on cuda:  40%|████      | 4/10 [00:10<00:15,  2.65s/epoch, loss=0.000543, prev_loss=0.000541]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 6/161 [00:00<00:02, 52.26batch/s]

Training batches on cuda:   9%|▊         | 14/161 [00:00<00:02, 56.74batch/s]

Training batches on cuda:  13%|█▎        | 21/161 [00:00<00:02, 59.98batch/s]

Training batches on cuda:  18%|█▊        | 29/161 [00:00<00:02, 62.81batch/s]

Training batches on cuda:  22%|██▏       | 36/161 [00:00<00:01, 64.55batch/s]

Training batches on cuda:  27%|██▋       | 44/161 [00:00<00:01, 66.17batch/s]

Training batches on cuda:  32%|███▏      | 51/161 [00:00<00:01, 67.27batch/s]

Training batches on cuda:  37%|███▋      | 59/161 [00:00<00:01, 68.15batch/s]

Training batches on cuda:  41%|████      | 66/161 [00:00<00:01, 68.61batch/s]

Training batches on cuda:  45%|████▌     | 73/161 [00:01<00:01, 68.97batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:01<00:01, 69.48batch/s]

Training batches on cuda:  55%|█████▌    | 89/161 [00:01<00:01, 69.84batch/s]

Training batches on cuda:  60%|█████▉    | 96/161 [00:01<00:00, 66.29batch/s]

Training batches on cuda:  64%|██████▍   | 103/161 [00:01<00:00, 62.07batch/s]

Training batches on cuda:  68%|██████▊   | 110/161 [00:01<00:00, 60.49batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:01<00:00, 58.28batch/s]

Training batches on cuda:  76%|███████▋  | 123/161 [00:01<00:00, 57.45batch/s]

Training batches on cuda:  80%|████████  | 129/161 [00:02<00:00, 58.06batch/s]

Training batches on cuda:  84%|████████▍ | 135/161 [00:02<00:00, 58.08batch/s]

Training batches on cuda:  88%|████████▊ | 141/161 [00:02<00:00, 58.29batch/s]

Training batches on cuda:  92%|█████████▏| 148/161 [00:02<00:00, 58.97batch/s]

Training batches on cuda:  96%|█████████▌| 154/161 [00:02<00:00, 58.90batch/s]

Training batches on cuda: 100%|██████████| 161/161 [00:02<00:00, 59.09batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:13<00:15,  2.65s/epoch, loss=0.000543, prev_loss=0.000543]
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:13,  2.68s/epoch, loss=0.000543, prev_loss=0.000543]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/161 [00:00<00:04, 38.28batch/s]

Training batches on cuda:   6%|▌         | 10/161 [00:00<00:03, 42.20batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:03, 45.69batch/s]

Training batches on cuda:  14%|█▎        | 22/161 [00:00<00:02, 49.09batch/s]

Training batches on cuda:  17%|█▋        | 28/161 [00:00<00:02, 50.96batch/s]

Training batches on cuda:  21%|██        | 34/161 [00:00<00:02, 52.93batch/s]

Training batches on cuda:  25%|██▍       | 40/161 [00:00<00:02, 54.18batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:02, 55.53batch/s]

Training batches on cuda:  32%|███▏      | 52/161 [00:00<00:01, 56.18batch/s]

Training batches on cuda:  36%|███▌      | 58/161 [00:01<00:01, 57.11batch/s]

Training batches on cuda:  40%|███▉      | 64/161 [00:01<00:01, 56.05batch/s]

Training batches on cuda:  44%|████▍     | 71/161 [00:01<00:01, 58.53batch/s]

Training batches on cuda:  48%|████▊     | 78/161 [00:01<00:01, 59.96batch/s]

Training batches on cuda:  53%|█████▎    | 85/161 [00:01<00:01, 60.94batch/s]

Training batches on cuda:  57%|█████▋    | 92/161 [00:01<00:01, 62.33batch/s]

Training batches on cuda:  61%|██████▏   | 99/161 [00:01<00:00, 63.62batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 64.82batch/s]

Training batches on cuda:  70%|███████   | 113/161 [00:01<00:00, 65.75batch/s]

Training batches on cuda:  75%|███████▍  | 120/161 [00:02<00:00, 66.37batch/s]

Training batches on cuda:  79%|███████▉  | 127/161 [00:02<00:00, 66.45batch/s]

Training batches on cuda:  83%|████████▎ | 134/161 [00:02<00:00, 66.97batch/s]

Training batches on cuda:  88%|████████▊ | 141/161 [00:02<00:00, 67.34batch/s]

Training batches on cuda:  92%|█████████▏| 148/161 [00:02<00:00, 67.28batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:02<00:00, 67.85batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:16<00:13,  2.68s/epoch, loss=0.000542, prev_loss=0.000543]
Training epochs on cuda:  60%|██████    | 6/10 [00:16<00:10,  2.71s/epoch, loss=0.000542, prev_loss=0.000543]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 45.39batch/s]

Training batches on cuda:   7%|▋         | 12/161 [00:00<00:02, 49.81batch/s]

Training batches on cuda:  11%|█         | 18/161 [00:00<00:02, 51.87batch/s]

Training batches on cuda:  16%|█▌        | 25/161 [00:00<00:02, 54.16batch/s]

Training batches on cuda:  20%|█▉        | 32/161 [00:00<00:02, 55.90batch/s]

Training batches on cuda:  24%|██▎       | 38/161 [00:00<00:02, 56.97batch/s]

Training batches on cuda:  27%|██▋       | 44/161 [00:00<00:02, 57.67batch/s]

Training batches on cuda:  31%|███       | 50/161 [00:00<00:01, 56.54batch/s]

Training batches on cuda:  35%|███▍      | 56/161 [00:00<00:01, 57.53batch/s]

Training batches on cuda:  39%|███▊      | 62/161 [00:01<00:01, 58.23batch/s]

Training batches on cuda:  42%|████▏     | 68/161 [00:01<00:01, 58.61batch/s]

Training batches on cuda:  46%|████▌     | 74/161 [00:01<00:01, 59.00batch/s]

Training batches on cuda:  50%|█████     | 81/161 [00:01<00:01, 59.41batch/s]

Training batches on cuda:  55%|█████▍    | 88/161 [00:01<00:01, 59.67batch/s]

Training batches on cuda:  58%|█████▊    | 94/161 [00:01<00:01, 59.51batch/s]

Training batches on cuda:  63%|██████▎   | 101/161 [00:01<00:01, 59.72batch/s]

Training batches on cuda:  66%|██████▋   | 107/161 [00:01<00:00, 59.74batch/s]

Training batches on cuda:  71%|███████   | 114/161 [00:01<00:00, 59.93batch/s]

Training batches on cuda:  75%|███████▌  | 121/161 [00:02<00:00, 60.06batch/s]

Training batches on cuda:  80%|███████▉  | 128/161 [00:02<00:00, 59.88batch/s]

Training batches on cuda:  84%|████████▍ | 135/161 [00:02<00:00, 59.99batch/s]

Training batches on cuda:  88%|████████▊ | 142/161 [00:02<00:00, 60.10batch/s]

Training batches on cuda:  93%|█████████▎| 149/161 [00:02<00:00, 59.96batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:02<00:00, 59.69batch/s]

Training batches on cuda: 100%|██████████| 161/161 [00:02<00:00, 59.57batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:19<00:10,  2.71s/epoch, loss=0.000542, prev_loss=0.000542]
Training epochs on cuda:  70%|███████   | 7/10 [00:19<00:08,  2.77s/epoch, loss=0.000542, prev_loss=0.000542]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/161 [00:00<00:04, 38.96batch/s]

Training batches on cuda:   6%|▌         | 10/161 [00:00<00:03, 43.23batch/s]

Training batches on cuda:  10%|▉         | 16/161 [00:00<00:03, 47.06batch/s]

Training batches on cuda:  14%|█▎        | 22/161 [00:00<00:02, 50.08batch/s]

Training batches on cuda:  17%|█▋        | 28/161 [00:00<00:02, 52.45batch/s]

Training batches on cuda:  21%|██        | 34/161 [00:00<00:02, 53.83batch/s]

Training batches on cuda:  25%|██▍       | 40/161 [00:00<00:02, 55.23batch/s]

Training batches on cuda:  29%|██▊       | 46/161 [00:00<00:02, 55.99batch/s]

Training batches on cuda:  32%|███▏      | 52/161 [00:00<00:01, 56.81batch/s]

Training batches on cuda:  36%|███▌      | 58/161 [00:01<00:01, 57.40batch/s]

Training batches on cuda:  40%|███▉      | 64/161 [00:01<00:01, 57.70batch/s]

Training batches on cuda:  43%|████▎     | 70/161 [00:01<00:01, 58.02batch/s]

Training batches on cuda:  47%|████▋     | 76/161 [00:01<00:01, 58.03batch/s]

Training batches on cuda:  51%|█████     | 82/161 [00:01<00:01, 58.42batch/s]

Training batches on cuda:  55%|█████▍    | 88/161 [00:01<00:01, 58.61batch/s]

Training batches on cuda:  58%|█████▊    | 94/161 [00:01<00:01, 58.96batch/s]

Training batches on cuda:  62%|██████▏   | 100/161 [00:01<00:01, 58.76batch/s]

Training batches on cuda:  66%|██████▌   | 106/161 [00:01<00:00, 58.88batch/s]

Training batches on cuda:  70%|██████▉   | 112/161 [00:01<00:00, 58.68batch/s]

Training batches on cuda:  73%|███████▎  | 118/161 [00:02<00:00, 58.40batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:02<00:00, 58.47batch/s]

Training batches on cuda:  81%|████████  | 130/161 [00:02<00:00, 58.34batch/s]

Training batches on cuda:  84%|████████▍ | 136/161 [00:02<00:00, 56.84batch/s]

Training batches on cuda:  88%|████████▊ | 142/161 [00:02<00:00, 56.13batch/s]

Training batches on cuda:  92%|█████████▏| 148/161 [00:02<00:00, 56.05batch/s]

Training batches on cuda:  96%|█████████▌| 154/161 [00:02<00:00, 56.02batch/s]

Training batches on cuda:  99%|█████████▉| 160/161 [00:02<00:00, 56.82batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:22<00:08,  2.77s/epoch, loss=0.000543, prev_loss=0.000542]
Training epochs on cuda:  80%|████████  | 8/10 [00:22<00:05,  2.87s/epoch, loss=0.000543, prev_loss=0.000542]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 2/161 [00:00<00:08, 19.71batch/s]

Training batches on cuda:   5%|▍         | 8/161 [00:00<00:06, 24.42batch/s]

Training batches on cuda:   9%|▊         | 14/161 [00:00<00:04, 29.57batch/s]

Training batches on cuda:  12%|█▏        | 20/161 [00:00<00:04, 34.62batch/s]

Training batches on cuda:  16%|█▌        | 26/161 [00:00<00:03, 39.45batch/s]

Training batches on cuda:  20%|█▉        | 32/161 [00:00<00:02, 43.65batch/s]

Training batches on cuda:  24%|██▎       | 38/161 [00:00<00:02, 47.25batch/s]

Training batches on cuda:  27%|██▋       | 44/161 [00:00<00:02, 50.24batch/s]

Training batches on cuda:  31%|███       | 50/161 [00:00<00:02, 52.34batch/s]

Training batches on cuda:  35%|███▍      | 56/161 [00:01<00:01, 53.62batch/s]

Training batches on cuda:  39%|███▊      | 62/161 [00:01<00:01, 54.86batch/s]

Training batches on cuda:  42%|████▏     | 68/161 [00:01<00:01, 55.53batch/s]

Training batches on cuda:  46%|████▌     | 74/161 [00:01<00:01, 56.30batch/s]

Training batches on cuda:  50%|████▉     | 80/161 [00:01<00:01, 56.79batch/s]

Training batches on cuda:  53%|█████▎    | 86/161 [00:01<00:01, 57.28batch/s]

Training batches on cuda:  57%|█████▋    | 92/161 [00:01<00:01, 50.96batch/s]

Training batches on cuda:  61%|██████    | 98/161 [00:01<00:01, 52.70batch/s]

Training batches on cuda:  65%|██████▍   | 104/161 [00:01<00:01, 54.49batch/s]

Training batches on cuda:  69%|██████▉   | 111/161 [00:02<00:00, 56.07batch/s]

Training batches on cuda:  73%|███████▎  | 118/161 [00:02<00:00, 57.25batch/s]

Training batches on cuda:  78%|███████▊  | 125/161 [00:02<00:00, 58.06batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:02<00:00, 56.99batch/s]

Training batches on cuda:  85%|████████▌ | 137/161 [00:02<00:00, 57.72batch/s]

Training batches on cuda:  89%|████████▉ | 143/161 [00:02<00:00, 57.98batch/s]

Training batches on cuda:  93%|█████████▎| 149/161 [00:02<00:00, 56.26batch/s]

Training batches on cuda:  96%|█████████▋| 155/161 [00:02<00:00, 55.19batch/s]

Training batches on cuda: 100%|██████████| 161/161 [00:02<00:00, 54.41batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:25<00:05,  2.87s/epoch, loss=0.000543, prev_loss=0.000543]
Training epochs on cuda:  90%|█████████ | 9/10 [00:25<00:02,  2.95s/epoch, loss=0.000543, prev_loss=0.000543]

Training batches on cuda:   0%|          | 0/161 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/161 [00:00<00:03, 45.03batch/s]

Training batches on cuda:   7%|▋         | 12/161 [00:00<00:02, 49.97batch/s]

Training batches on cuda:  12%|█▏        | 19/161 [00:00<00:02, 54.21batch/s]

Training batches on cuda:  16%|█▌        | 26/161 [00:00<00:02, 57.73batch/s]

Training batches on cuda:  20%|██        | 33/161 [00:00<00:02, 59.97batch/s]

Training batches on cuda:  25%|██▍       | 40/161 [00:00<00:01, 62.43batch/s]

Training batches on cuda:  29%|██▉       | 47/161 [00:00<00:01, 64.31batch/s]

Training batches on cuda:  34%|███▎      | 54/161 [00:00<00:01, 65.68batch/s]

Training batches on cuda:  38%|███▊      | 61/161 [00:00<00:01, 66.68batch/s]

Training batches on cuda:  42%|████▏     | 68/161 [00:01<00:01, 67.31batch/s]

Training batches on cuda:  47%|████▋     | 75/161 [00:01<00:01, 67.48batch/s]

Training batches on cuda:  51%|█████     | 82/161 [00:01<00:01, 67.97batch/s]

Training batches on cuda:  55%|█████▌    | 89/161 [00:01<00:01, 68.33batch/s]

Training batches on cuda:  60%|█████▉    | 96/161 [00:01<00:00, 68.40batch/s]

Training batches on cuda:  64%|██████▍   | 103/161 [00:01<00:00, 68.43batch/s]

Training batches on cuda:  68%|██████▊   | 110/161 [00:01<00:00, 68.60batch/s]

Training batches on cuda:  73%|███████▎  | 117/161 [00:01<00:00, 67.98batch/s]

Training batches on cuda:  77%|███████▋  | 124/161 [00:01<00:00, 67.80batch/s]

Training batches on cuda:  81%|████████▏ | 131/161 [00:01<00:00, 66.17batch/s]

Training batches on cuda:  86%|████████▌ | 138/161 [00:02<00:00, 66.43batch/s]

Training batches on cuda:  90%|█████████ | 145/161 [00:02<00:00, 66.72batch/s]

Training batches on cuda:  94%|█████████▍| 152/161 [00:02<00:00, 66.96batch/s]

Training batches on cuda:  99%|█████████▉| 159/161 [00:02<00:00, 67.23batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:27<00:02,  2.95s/epoch, loss=0.000542, prev_loss=0.000543]2021-09-16:19:29:04,826 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:29:05,813 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:29:20,746 INFO     [evaluator.py:657] Evaluation took 14.93s seconds
2021-09-16:19:29:20,879 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00,  7.66s/epoch, loss=0.000542, prev_loss=0.000543]
Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00,  4.40s/epoch, loss=0.000542, prev_loss=0.000543]
2021-09-16:19:29:20,893 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpa942yjpq'
2021-09-16:19:29:20,967 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpa942yjpq' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.38ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.54ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.64ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.64ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.64ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.64ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.58ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.60ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.63ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:19:29:35,812 INFO     [evaluator.py:657] Evaluation took 14.82s seconds
[I 2021-09-16 19:29:35,817] Trial 86 finished with value: 0.0020761053861099166 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.7988985552242424, 'optimizer.lr': 0.056770047208468684, 'negative_sampler.num_negs_per_pos': 12, 'training.batch_size': 1932}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:29:35,840 WARNING  [api.py:823] No random seed is specified. Setting to 1404255336.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 26.27batch/s]

Training batches on cuda:   7%|▋         | 8/110 [00:00<00:03, 29.72batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:02, 32.74batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:00<00:02, 33.89batch/s]

Training batches on cuda:  20%|██        | 22/110 [00:00<00:02, 35.86batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:00<00:02, 37.34batch/s]

Training batches on cuda:  29%|██▉       | 32/110 [00:00<00:02, 38.55batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:00<00:01, 39.54batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:01<00:01, 39.56batch/s]

Training batches on cuda:  42%|████▏     | 46/110 [00:01<00:01, 40.20batch/s]

Training batches on cuda:  46%|████▋     | 51/110 [00:01<00:01, 40.40batch/s]

Training batches on cuda:  51%|█████     | 56/110 [00:01<00:01, 40.79batch/s]

Training batches on cuda:  55%|█████▌    | 61/110 [00:01<00:01, 40.97batch/s]

Training batches on cuda:  60%|██████    | 66/110 [00:01<00:01, 41.72batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:01<00:00, 42.59batch/s]

Training batches on cuda:  69%|██████▉   | 76/110 [00:01<00:00, 42.55batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:01<00:00, 42.35batch/s]

Training batches on cuda:  78%|███████▊  | 86/110 [00:02<00:00, 43.00batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:02<00:00, 43.58batch/s]

Training batches on cuda:  87%|████████▋ | 96/110 [00:02<00:00, 43.67batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:02<00:00, 43.78batch/s]

Training batches on cuda:  96%|█████████▋| 106/110 [00:02<00:00, 43.81batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000356, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:25,  2.84s/epoch, loss=0.000356, prev_loss=nan]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/110 [00:00<00:05, 19.29batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:04, 23.14batch/s]

Training batches on cuda:  11%|█         | 12/110 [00:00<00:03, 26.86batch/s]

Training batches on cuda:  15%|█▌        | 17/110 [00:00<00:03, 30.35batch/s]

Training batches on cuda:  20%|██        | 22/110 [00:00<00:02, 33.01batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:00<00:02, 35.20batch/s]

Training batches on cuda:  29%|██▉       | 32/110 [00:00<00:02, 37.01batch/s]

Training batches on cuda:  34%|███▎      | 37/110 [00:00<00:01, 37.99batch/s]

Training batches on cuda:  38%|███▊      | 42/110 [00:01<00:01, 38.22batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:01<00:01, 39.19batch/s]

Training batches on cuda:  47%|████▋     | 52/110 [00:01<00:01, 40.33batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:01<00:01, 40.42batch/s]

Training batches on cuda:  56%|█████▋    | 62/110 [00:01<00:01, 41.48batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:01<00:01, 42.21batch/s]

Training batches on cuda:  65%|██████▌   | 72/110 [00:01<00:00, 42.29batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:01<00:00, 42.34batch/s]

Training batches on cuda:  75%|███████▍  | 82/110 [00:02<00:00, 38.03batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:02<00:00, 38.47batch/s]

Training batches on cuda:  84%|████████▎ | 92/110 [00:02<00:00, 39.56batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:02<00:00, 40.69batch/s]

Training batches on cuda:  93%|█████████▎| 102/110 [00:02<00:00, 41.79batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:02<00:00, 41.83batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:25,  2.84s/epoch, loss=0.000354, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:22,  2.86s/epoch, loss=0.000354, prev_loss=nan]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:11,  9.54batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:08, 12.35batch/s]

Training batches on cuda:   9%|▉         | 10/110 [00:00<00:06, 15.59batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:04, 19.13batch/s]

Training batches on cuda:  18%|█▊        | 20/110 [00:00<00:03, 22.79batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:00<00:03, 26.37batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:00<00:02, 29.31batch/s]

Training batches on cuda:  31%|███       | 34/110 [00:00<00:02, 32.17batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:01<00:02, 34.66batch/s]

Training batches on cuda:  40%|████      | 44/110 [00:01<00:01, 36.81batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:01<00:01, 38.65batch/s]

Training batches on cuda:  49%|████▉     | 54/110 [00:01<00:01, 39.69batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:01<00:01, 40.48batch/s]

Training batches on cuda:  58%|█████▊    | 64/110 [00:01<00:01, 41.28batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:00, 41.34batch/s]

Training batches on cuda:  67%|██████▋   | 74/110 [00:01<00:00, 41.46batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:01<00:00, 41.48batch/s]

Training batches on cuda:  76%|███████▋  | 84/110 [00:02<00:00, 42.12batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:02<00:00, 42.02batch/s]

Training batches on cuda:  85%|████████▌ | 94/110 [00:02<00:00, 42.63batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:02<00:00, 43.18batch/s]

Training batches on cuda:  95%|█████████▍| 104/110 [00:02<00:00, 43.17batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:02<00:00, 42.79batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:22,  2.86s/epoch, loss=0.000352, prev_loss=0.000354]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:20,  2.87s/epoch, loss=0.000352, prev_loss=0.000354]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 4/110 [00:00<00:03, 31.66batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:02, 33.80batch/s]

Training batches on cuda:  13%|█▎        | 14/110 [00:00<00:02, 36.51batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:02, 38.48batch/s]

Training batches on cuda:  22%|██▏       | 24/110 [00:00<00:02, 40.13batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:00<00:01, 41.26batch/s]

Training batches on cuda:  31%|███       | 34/110 [00:00<00:01, 42.24batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:00<00:01, 42.72batch/s]

Training batches on cuda:  40%|████      | 44/110 [00:01<00:01, 42.75batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:01<00:01, 43.27batch/s]

Training batches on cuda:  49%|████▉     | 54/110 [00:01<00:01, 43.74batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:01<00:01, 42.22batch/s]

Training batches on cuda:  58%|█████▊    | 64/110 [00:01<00:01, 42.32batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:00, 43.26batch/s]

Training batches on cuda:  67%|██████▋   | 74/110 [00:01<00:00, 43.45batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:01<00:00, 43.59batch/s]

Training batches on cuda:  76%|███████▋  | 84/110 [00:01<00:00, 43.44batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:02<00:00, 43.49batch/s]

Training batches on cuda:  85%|████████▌ | 94/110 [00:02<00:00, 42.87batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:02<00:00, 42.55batch/s]

Training batches on cuda:  95%|█████████▍| 104/110 [00:02<00:00, 41.07batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:02<00:00, 41.09batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:20,  2.87s/epoch, loss=0.000349, prev_loss=0.000352]
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:17,  2.85s/epoch, loss=0.000349, prev_loss=0.000352]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   4%|▎         | 4/110 [00:00<00:02, 37.08batch/s]

Training batches on cuda:   8%|▊         | 9/110 [00:00<00:02, 39.58batch/s]

Training batches on cuda:  13%|█▎        | 14/110 [00:00<00:02, 42.18batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:02, 42.68batch/s]

Training batches on cuda:  22%|██▏       | 24/110 [00:00<00:02, 41.70batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:00<00:01, 42.65batch/s]

Training batches on cuda:  31%|███       | 34/110 [00:00<00:01, 43.11batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:00<00:01, 43.68batch/s]

Training batches on cuda:  40%|████      | 44/110 [00:01<00:01, 43.15batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:01<00:01, 43.33batch/s]

Training batches on cuda:  49%|████▉     | 54/110 [00:01<00:01, 43.15batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:01<00:01, 43.45batch/s]

Training batches on cuda:  58%|█████▊    | 64/110 [00:01<00:01, 43.21batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:00, 43.62batch/s]

Training batches on cuda:  67%|██████▋   | 74/110 [00:01<00:00, 43.85batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:01<00:00, 44.02batch/s]

Training batches on cuda:  76%|███████▋  | 84/110 [00:01<00:00, 43.42batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:02<00:00, 43.43batch/s]

Training batches on cuda:  85%|████████▌ | 94/110 [00:02<00:00, 43.31batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:02<00:00, 43.52batch/s]

Training batches on cuda:  95%|█████████▍| 104/110 [00:02<00:00, 43.78batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:02<00:00, 43.53batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:14<00:17,  2.85s/epoch, loss=0.000333, prev_loss=0.000349]
Training epochs on cuda:  50%|█████     | 5/10 [00:14<00:14,  2.84s/epoch, loss=0.000333, prev_loss=0.000349]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:03, 28.49batch/s]

Training batches on cuda:   7%|▋         | 8/110 [00:00<00:03, 31.58batch/s]

Training batches on cuda:  12%|█▏        | 13/110 [00:00<00:02, 34.20batch/s]

Training batches on cuda:  16%|█▋        | 18/110 [00:00<00:02, 36.03batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:00<00:02, 37.58batch/s]

Training batches on cuda:  25%|██▌       | 28/110 [00:00<00:02, 38.73batch/s]

Training batches on cuda:  30%|███       | 33/110 [00:00<00:01, 39.59batch/s]

Training batches on cuda:  35%|███▍      | 38/110 [00:00<00:01, 40.18batch/s]

Training batches on cuda:  39%|███▉      | 43/110 [00:01<00:01, 40.61batch/s]

Training batches on cuda:  43%|████▎     | 47/110 [00:01<00:01, 40.08batch/s]

Training batches on cuda:  47%|████▋     | 52/110 [00:01<00:01, 41.28batch/s]

Training batches on cuda:  52%|█████▏    | 57/110 [00:01<00:01, 42.21batch/s]

Training batches on cuda:  56%|█████▋    | 62/110 [00:01<00:01, 42.21batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:01<00:01, 42.35batch/s]

Training batches on cuda:  65%|██████▌   | 72/110 [00:01<00:00, 42.59batch/s]

Training batches on cuda:  70%|███████   | 77/110 [00:01<00:00, 42.52batch/s]

Training batches on cuda:  75%|███████▍  | 82/110 [00:01<00:00, 42.69batch/s]

Training batches on cuda:  79%|███████▉  | 87/110 [00:02<00:00, 42.57batch/s]

Training batches on cuda:  84%|████████▎ | 92/110 [00:02<00:00, 42.43batch/s]

Training batches on cuda:  88%|████████▊ | 97/110 [00:02<00:00, 42.54batch/s]

Training batches on cuda:  93%|█████████▎| 102/110 [00:02<00:00, 42.86batch/s]

Training batches on cuda:  97%|█████████▋| 107/110 [00:02<00:00, 43.26batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:17<00:14,  2.84s/epoch, loss=0.000322, prev_loss=0.000333]
Training epochs on cuda:  60%|██████    | 6/10 [00:17<00:11,  2.85s/epoch, loss=0.000322, prev_loss=0.000333]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 26.17batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 28.92batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 31.35batch/s]

Training batches on cuda:  15%|█▍        | 16/110 [00:00<00:02, 33.70batch/s]

Training batches on cuda:  19%|█▉        | 21/110 [00:00<00:02, 36.51batch/s]

Training batches on cuda:  24%|██▎       | 26/110 [00:00<00:02, 38.48batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:00<00:02, 39.24batch/s]

Training batches on cuda:  33%|███▎      | 36/110 [00:00<00:01, 39.54batch/s]

Training batches on cuda:  36%|███▋      | 40/110 [00:01<00:01, 39.56batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:01<00:01, 41.19batch/s]

Training batches on cuda:  45%|████▌     | 50/110 [00:01<00:01, 41.19batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:01<00:01, 40.80batch/s]

Training batches on cuda:  55%|█████▍    | 60/110 [00:01<00:01, 40.52batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:01<00:01, 41.42batch/s]

Training batches on cuda:  64%|██████▎   | 70/110 [00:01<00:00, 41.05batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:01<00:00, 41.04batch/s]

Training batches on cuda:  73%|███████▎  | 80/110 [00:01<00:00, 40.78batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:02<00:00, 41.06batch/s]

Training batches on cuda:  82%|████████▏ | 90/110 [00:02<00:00, 41.02batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:02<00:00, 40.57batch/s]

Training batches on cuda:  91%|█████████ | 100/110 [00:02<00:00, 40.51batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:02<00:00, 40.45batch/s]

Training batches on cuda: 100%|██████████| 110/110 [00:02<00:00, 36.39batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:20<00:11,  2.85s/epoch, loss=0.00032, prev_loss=0.000322] 
Training epochs on cuda:  70%|███████   | 7/10 [00:20<00:08,  2.88s/epoch, loss=0.00032, prev_loss=0.000322]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 23.51batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 26.51batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 28.95batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:03, 31.13batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:02, 32.31batch/s]

Training batches on cuda:  21%|██        | 23/110 [00:00<00:02, 33.25batch/s]

Training batches on cuda:  25%|██▍       | 27/110 [00:00<00:02, 34.47batch/s]

Training batches on cuda:  28%|██▊       | 31/110 [00:00<00:02, 35.14batch/s]

Training batches on cuda:  33%|███▎      | 36/110 [00:01<00:02, 36.70batch/s]

Training batches on cuda:  37%|███▋      | 41/110 [00:01<00:01, 38.21batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:01<00:01, 38.22batch/s]

Training batches on cuda:  45%|████▌     | 50/110 [00:01<00:01, 38.86batch/s]

Training batches on cuda:  49%|████▉     | 54/110 [00:01<00:01, 37.49batch/s]

Training batches on cuda:  53%|█████▎    | 58/110 [00:01<00:01, 37.84batch/s]

Training batches on cuda:  57%|█████▋    | 63/110 [00:01<00:01, 39.44batch/s]

Training batches on cuda:  61%|██████    | 67/110 [00:01<00:01, 39.35batch/s]

Training batches on cuda:  65%|██████▍   | 71/110 [00:01<00:00, 39.45batch/s]

Training batches on cuda:  69%|██████▉   | 76/110 [00:02<00:00, 40.23batch/s]

Training batches on cuda:  74%|███████▎  | 81/110 [00:02<00:00, 41.26batch/s]

Training batches on cuda:  78%|███████▊  | 86/110 [00:02<00:00, 42.11batch/s]

Training batches on cuda:  83%|████████▎ | 91/110 [00:02<00:00, 42.37batch/s]

Training batches on cuda:  87%|████████▋ | 96/110 [00:02<00:00, 40.81batch/s]

Training batches on cuda:  92%|█████████▏| 101/110 [00:02<00:00, 40.47batch/s]

Training batches on cuda:  96%|█████████▋| 106/110 [00:02<00:00, 39.85batch/s]

Training batches on cuda: 100%|██████████| 110/110 [00:02<00:00, 39.18batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:23<00:08,  2.88s/epoch, loss=0.000304, prev_loss=0.00032]
Training epochs on cuda:  80%|████████  | 8/10 [00:23<00:05,  2.94s/epoch, loss=0.000304, prev_loss=0.00032]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/110 [00:00<00:04, 24.14batch/s]

Training batches on cuda:   6%|▋         | 7/110 [00:00<00:03, 27.12batch/s]

Training batches on cuda:  10%|█         | 11/110 [00:00<00:03, 29.79batch/s]

Training batches on cuda:  14%|█▎        | 15/110 [00:00<00:02, 32.08batch/s]

Training batches on cuda:  18%|█▊        | 20/110 [00:00<00:02, 34.56batch/s]

Training batches on cuda:  23%|██▎       | 25/110 [00:00<00:02, 37.50batch/s]

Training batches on cuda:  27%|██▋       | 30/110 [00:00<00:01, 40.19batch/s]

Training batches on cuda:  32%|███▏      | 35/110 [00:00<00:01, 41.33batch/s]

Training batches on cuda:  36%|███▋      | 40/110 [00:00<00:01, 42.79batch/s]

Training batches on cuda:  41%|████      | 45/110 [00:01<00:01, 43.51batch/s]

Training batches on cuda:  45%|████▌     | 50/110 [00:01<00:01, 43.55batch/s]

Training batches on cuda:  50%|█████     | 55/110 [00:01<00:01, 44.41batch/s]

Training batches on cuda:  55%|█████▍    | 60/110 [00:01<00:01, 44.74batch/s]

Training batches on cuda:  59%|█████▉    | 65/110 [00:01<00:01, 44.52batch/s]

Training batches on cuda:  64%|██████▎   | 70/110 [00:01<00:00, 44.27batch/s]

Training batches on cuda:  68%|██████▊   | 75/110 [00:01<00:00, 39.11batch/s]

Training batches on cuda:  73%|███████▎  | 80/110 [00:01<00:00, 40.00batch/s]

Training batches on cuda:  77%|███████▋  | 85/110 [00:02<00:00, 41.11batch/s]

Training batches on cuda:  82%|████████▏ | 90/110 [00:02<00:00, 40.94batch/s]

Training batches on cuda:  86%|████████▋ | 95/110 [00:02<00:00, 41.71batch/s]

Training batches on cuda:  91%|█████████ | 100/110 [00:02<00:00, 42.32batch/s]

Training batches on cuda:  95%|█████████▌| 105/110 [00:02<00:00, 42.77batch/s]

Training batches on cuda: 100%|██████████| 110/110 [00:02<00:00, 39.13batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:26<00:05,  2.94s/epoch, loss=0.000252, prev_loss=0.000304]
Training epochs on cuda:  90%|█████████ | 9/10 [00:26<00:02,  2.92s/epoch, loss=0.000252, prev_loss=0.000304]

Training batches on cuda:   0%|          | 0/110 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/110 [00:00<00:11,  9.90batch/s]

Training batches on cuda:   5%|▍         | 5/110 [00:00<00:08, 12.65batch/s]

Training batches on cuda:   9%|▉         | 10/110 [00:00<00:06, 15.99batch/s]

Training batches on cuda:  13%|█▎        | 14/110 [00:00<00:04, 19.49batch/s]

Training batches on cuda:  17%|█▋        | 19/110 [00:00<00:03, 23.23batch/s]

Training batches on cuda:  22%|██▏       | 24/110 [00:00<00:03, 27.10batch/s]

Training batches on cuda:  26%|██▋       | 29/110 [00:00<00:02, 30.84batch/s]

Training batches on cuda:  31%|███       | 34/110 [00:00<00:02, 34.48batch/s]

Training batches on cuda:  35%|███▌      | 39/110 [00:00<00:01, 37.12batch/s]

Training batches on cuda:  40%|████      | 44/110 [00:01<00:01, 39.49batch/s]

Training batches on cuda:  45%|████▍     | 49/110 [00:01<00:01, 39.26batch/s]

Training batches on cuda:  49%|████▉     | 54/110 [00:01<00:01, 39.60batch/s]

Training batches on cuda:  54%|█████▎    | 59/110 [00:01<00:01, 39.24batch/s]

Training batches on cuda:  58%|█████▊    | 64/110 [00:01<00:01, 40.27batch/s]

Training batches on cuda:  63%|██████▎   | 69/110 [00:01<00:00, 41.58batch/s]

Training batches on cuda:  67%|██████▋   | 74/110 [00:01<00:00, 42.42batch/s]

Training batches on cuda:  72%|███████▏  | 79/110 [00:01<00:00, 41.99batch/s]

Training batches on cuda:  76%|███████▋  | 84/110 [00:02<00:00, 42.38batch/s]

Training batches on cuda:  81%|████████  | 89/110 [00:02<00:00, 42.79batch/s]

Training batches on cuda:  85%|████████▌ | 94/110 [00:02<00:00, 42.01batch/s]

Training batches on cuda:  90%|█████████ | 99/110 [00:02<00:00, 42.36batch/s]

Training batches on cuda:  95%|█████████▍| 104/110 [00:02<00:00, 42.93batch/s]

Training batches on cuda:  99%|█████████▉| 109/110 [00:02<00:00, 43.14batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:28<00:02,  2.92s/epoch, loss=0.000175, prev_loss=0.000252]2021-09-16:19:30:05,262 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:30:06,307 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:30:21,269 INFO     [evaluator.py:657] Evaluation took 14.96s seconds
2021-09-16:19:30:21,399 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  7.75s/epoch, loss=0.000175, prev_loss=0.000252]
Training epochs on cuda: 100%|██████████| 10/10 [00:45<00:00,  4.50s/epoch, loss=0.000175, prev_loss=0.000252]
2021-09-16:19:30:21,399 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpax7o0sxa'
2021-09-16:19:30:21,474 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpax7o0sxa' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.33ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.41ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.52ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:13, 2.56ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.59ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:30:36,250 INFO     [evaluator.py:657] Evaluation took 14.76s seconds
[I 2021-09-16 19:30:36,257] Trial 87 finished with value: 0.903717813386577 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.030267548342956502, 'optimizer.lr': 0.04664970992276009, 'negative_sampler.num_negs_per_pos': 8, 'training.batch_size': 2832}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:30:36,279 WARNING  [api.py:823] No random seed is specified. Setting to 3415582997.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/119 [00:00<00:04, 27.13batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:03, 29.11batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:03, 30.75batch/s]

Training batches on cuda:  13%|█▎        | 15/119 [00:00<00:03, 32.26batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:02, 33.40batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:00<00:02, 34.28batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:00<00:02, 34.50batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:00<00:02, 33.38batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:02, 33.56batch/s]

Training batches on cuda:  33%|███▎      | 39/119 [00:01<00:02, 34.12batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:01<00:02, 34.12batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:01<00:02, 33.90batch/s]

Training batches on cuda:  43%|████▎     | 51/119 [00:01<00:02, 31.08batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:01<00:01, 32.06batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:01<00:01, 32.54batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:01<00:01, 33.05batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:01<00:01, 33.43batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:02<00:01, 33.48batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:02<00:01, 33.64batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:02<00:01, 33.69batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:02<00:01, 33.84batch/s]

Training batches on cuda:  73%|███████▎  | 87/119 [00:02<00:00, 33.60batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:02<00:00, 33.67batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:02<00:00, 33.83batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:02<00:00, 34.00batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:03<00:00, 33.94batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:03<00:00, 34.04batch/s]

Training batches on cuda:  93%|█████████▎| 111/119 [00:03<00:00, 34.11batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:03<00:00, 34.11batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000392, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:33,  3.78s/epoch, loss=0.000392, prev_loss=nan]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/119 [00:00<00:04, 24.62batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:04, 27.36batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:03, 28.96batch/s]

Training batches on cuda:  13%|█▎        | 15/119 [00:00<00:03, 29.96batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:03, 31.43batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:00<00:02, 32.69batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:00<00:02, 32.00batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:00<00:02, 31.74batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:02, 32.94batch/s]

Training batches on cuda:  33%|███▎      | 39/119 [00:01<00:02, 34.19batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:01<00:02, 34.93batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:01<00:02, 35.71batch/s]

Training batches on cuda:  43%|████▎     | 51/119 [00:01<00:01, 35.62batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:01<00:01, 34.78batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:01<00:01, 33.56batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:01<00:01, 33.04batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:02<00:01, 32.77batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:02<00:01, 32.34batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:02<00:01, 31.97batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:02<00:01, 32.60batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:02<00:01, 33.16batch/s]

Training batches on cuda:  73%|███████▎  | 87/119 [00:02<00:00, 33.52batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:02<00:00, 31.13batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:02<00:00, 31.98batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:02<00:00, 32.55batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:03<00:00, 33.12batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:03<00:00, 33.33batch/s]

Training batches on cuda:  93%|█████████▎| 111/119 [00:03<00:00, 33.46batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:03<00:00, 33.79batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:07<00:33,  3.78s/epoch, loss=0.000386, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:30,  3.78s/epoch, loss=0.000386, prev_loss=nan]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/119 [00:00<00:07, 16.16batch/s]

Training batches on cuda:   5%|▌         | 6/119 [00:00<00:05, 19.13batch/s]

Training batches on cuda:   8%|▊         | 10/119 [00:00<00:04, 22.09batch/s]

Training batches on cuda:  12%|█▏        | 14/119 [00:00<00:04, 24.74batch/s]

Training batches on cuda:  15%|█▌        | 18/119 [00:00<00:03, 27.02batch/s]

Training batches on cuda:  18%|█▊        | 22/119 [00:00<00:03, 28.69batch/s]

Training batches on cuda:  22%|██▏       | 26/119 [00:00<00:03, 30.14batch/s]

Training batches on cuda:  25%|██▌       | 30/119 [00:00<00:02, 31.28batch/s]

Training batches on cuda:  29%|██▊       | 34/119 [00:01<00:02, 31.90batch/s]

Training batches on cuda:  32%|███▏      | 38/119 [00:01<00:02, 32.54batch/s]

Training batches on cuda:  35%|███▌      | 42/119 [00:01<00:02, 33.03batch/s]

Training batches on cuda:  39%|███▊      | 46/119 [00:01<00:02, 33.28batch/s]

Training batches on cuda:  42%|████▏     | 50/119 [00:01<00:02, 33.45batch/s]

Training batches on cuda:  45%|████▌     | 54/119 [00:01<00:01, 33.72batch/s]

Training batches on cuda:  49%|████▊     | 58/119 [00:01<00:01, 33.89batch/s]

Training batches on cuda:  52%|█████▏    | 62/119 [00:01<00:01, 34.03batch/s]

Training batches on cuda:  55%|█████▌    | 66/119 [00:02<00:01, 34.00batch/s]

Training batches on cuda:  59%|█████▉    | 70/119 [00:02<00:01, 33.93batch/s]

Training batches on cuda:  62%|██████▏   | 74/119 [00:02<00:01, 34.13batch/s]

Training batches on cuda:  66%|██████▌   | 78/119 [00:02<00:01, 34.12batch/s]

Training batches on cuda:  69%|██████▉   | 82/119 [00:02<00:01, 34.09batch/s]

Training batches on cuda:  72%|███████▏  | 86/119 [00:02<00:00, 34.23batch/s]

Training batches on cuda:  76%|███████▌  | 90/119 [00:02<00:00, 34.25batch/s]

Training batches on cuda:  79%|███████▉  | 94/119 [00:02<00:00, 34.12batch/s]

Training batches on cuda:  82%|████████▏ | 98/119 [00:02<00:00, 34.25batch/s]

Training batches on cuda:  86%|████████▌ | 102/119 [00:03<00:00, 33.55batch/s]

Training batches on cuda:  89%|████████▉ | 106/119 [00:03<00:00, 33.59batch/s]

Training batches on cuda:  92%|█████████▏| 110/119 [00:03<00:00, 33.28batch/s]

Training batches on cuda:  96%|█████████▌| 114/119 [00:03<00:00, 33.21batch/s]

Training batches on cuda:  99%|█████████▉| 118/119 [00:03<00:00, 33.14batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:30,  3.78s/epoch, loss=0.000386, prev_loss=0.000386]
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:26,  3.78s/epoch, loss=0.000386, prev_loss=0.000386]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/119 [00:00<00:05, 19.92batch/s]

Training batches on cuda:   3%|▎         | 4/119 [00:00<00:05, 19.83batch/s]

Training batches on cuda:   7%|▋         | 8/119 [00:00<00:04, 22.44batch/s]

Training batches on cuda:  10%|█         | 12/119 [00:00<00:04, 24.89batch/s]

Training batches on cuda:  13%|█▎        | 16/119 [00:00<00:03, 27.02batch/s]

Training batches on cuda:  17%|█▋        | 20/119 [00:00<00:03, 28.38batch/s]

Training batches on cuda:  20%|██        | 24/119 [00:00<00:03, 29.37batch/s]

Training batches on cuda:  24%|██▎       | 28/119 [00:00<00:02, 30.65batch/s]

Training batches on cuda:  27%|██▋       | 32/119 [00:01<00:02, 30.83batch/s]

Training batches on cuda:  30%|███       | 36/119 [00:01<00:02, 31.66batch/s]

Training batches on cuda:  34%|███▎      | 40/119 [00:01<00:02, 32.35batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:01<00:02, 32.80batch/s]

Training batches on cuda:  40%|████      | 48/119 [00:01<00:02, 33.09batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:01<00:02, 33.40batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:01<00:01, 33.67batch/s]

Training batches on cuda:  50%|█████     | 60/119 [00:01<00:01, 33.87batch/s]

Training batches on cuda:  54%|█████▍    | 64/119 [00:01<00:01, 34.23batch/s]

Training batches on cuda:  57%|█████▋    | 68/119 [00:02<00:01, 34.13batch/s]

Training batches on cuda:  61%|██████    | 72/119 [00:02<00:01, 34.06batch/s]

Training batches on cuda:  64%|██████▍   | 76/119 [00:02<00:01, 34.16batch/s]

Training batches on cuda:  67%|██████▋   | 80/119 [00:02<00:01, 33.95batch/s]

Training batches on cuda:  71%|███████   | 84/119 [00:02<00:01, 34.00batch/s]

Training batches on cuda:  74%|███████▍  | 88/119 [00:02<00:00, 33.93batch/s]

Training batches on cuda:  77%|███████▋  | 92/119 [00:02<00:00, 33.84batch/s]

Training batches on cuda:  81%|████████  | 96/119 [00:02<00:00, 33.93batch/s]

Training batches on cuda:  84%|████████▍ | 100/119 [00:03<00:00, 34.17batch/s]

Training batches on cuda:  87%|████████▋ | 104/119 [00:03<00:00, 34.25batch/s]

Training batches on cuda:  91%|█████████ | 108/119 [00:03<00:00, 34.09batch/s]

Training batches on cuda:  94%|█████████▍| 112/119 [00:03<00:00, 33.87batch/s]

Training batches on cuda:  97%|█████████▋| 116/119 [00:03<00:00, 33.93batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:26,  3.78s/epoch, loss=0.000386, prev_loss=0.000386]
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:22,  3.78s/epoch, loss=0.000386, prev_loss=0.000386]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/119 [00:00<00:04, 23.24batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:04, 25.20batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:03, 27.69batch/s]

Training batches on cuda:  13%|█▎        | 15/119 [00:00<00:03, 29.50batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:03, 30.66batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:00<00:03, 31.52batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:00<00:02, 31.96batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:00<00:02, 32.54batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:02, 33.37batch/s]

Training batches on cuda:  33%|███▎      | 39/119 [00:01<00:02, 33.03batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:01<00:02, 32.92batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:01<00:02, 30.36batch/s]

Training batches on cuda:  43%|████▎     | 51/119 [00:01<00:02, 30.75batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:01<00:02, 31.30batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:01<00:01, 31.48batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:01<00:01, 32.24batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:02<00:01, 32.47batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:02<00:01, 33.00batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:02<00:01, 33.18batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:02<00:01, 32.83batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:02<00:01, 32.76batch/s]

Training batches on cuda:  73%|███████▎  | 87/119 [00:02<00:00, 32.25batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:02<00:00, 31.48batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:02<00:00, 31.92batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:03<00:00, 32.54batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:03<00:00, 32.58batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:03<00:00, 32.69batch/s]

Training batches on cuda:  93%|█████████▎| 111/119 [00:03<00:00, 32.71batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:03<00:00, 32.67batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:19<00:22,  3.78s/epoch, loss=0.000387, prev_loss=0.000386]
Training epochs on cuda:  50%|█████     | 5/10 [00:19<00:19,  3.81s/epoch, loss=0.000387, prev_loss=0.000386]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/119 [00:00<00:13,  8.73batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:10, 11.22batch/s]

Training batches on cuda:   8%|▊         | 9/119 [00:00<00:07, 13.91batch/s]

Training batches on cuda:  11%|█         | 13/119 [00:00<00:06, 16.95batch/s]

Training batches on cuda:  14%|█▍        | 17/119 [00:00<00:05, 19.82batch/s]

Training batches on cuda:  18%|█▊        | 21/119 [00:00<00:04, 22.40batch/s]

Training batches on cuda:  21%|██        | 25/119 [00:00<00:03, 24.92batch/s]

Training batches on cuda:  24%|██▍       | 29/119 [00:00<00:03, 27.11batch/s]

Training batches on cuda:  28%|██▊       | 33/119 [00:01<00:02, 28.84batch/s]

Training batches on cuda:  31%|███       | 37/119 [00:01<00:02, 29.96batch/s]

Training batches on cuda:  34%|███▍      | 41/119 [00:01<00:02, 31.03batch/s]

Training batches on cuda:  38%|███▊      | 45/119 [00:01<00:02, 31.46batch/s]

Training batches on cuda:  41%|████      | 49/119 [00:01<00:02, 31.30batch/s]

Training batches on cuda:  45%|████▍     | 53/119 [00:01<00:02, 31.94batch/s]

Training batches on cuda:  48%|████▊     | 57/119 [00:01<00:01, 32.54batch/s]

Training batches on cuda:  51%|█████▏    | 61/119 [00:01<00:01, 32.81batch/s]

Training batches on cuda:  55%|█████▍    | 65/119 [00:02<00:01, 32.95batch/s]

Training batches on cuda:  58%|█████▊    | 69/119 [00:02<00:01, 33.03batch/s]

Training batches on cuda:  61%|██████▏   | 73/119 [00:02<00:01, 32.98batch/s]

Training batches on cuda:  65%|██████▍   | 77/119 [00:02<00:01, 33.08batch/s]

Training batches on cuda:  68%|██████▊   | 81/119 [00:02<00:01, 32.52batch/s]

Training batches on cuda:  71%|███████▏  | 85/119 [00:02<00:01, 32.29batch/s]

Training batches on cuda:  75%|███████▍  | 89/119 [00:02<00:00, 31.87batch/s]

Training batches on cuda:  78%|███████▊  | 93/119 [00:02<00:00, 31.51batch/s]

Training batches on cuda:  82%|████████▏ | 97/119 [00:03<00:00, 31.46batch/s]

Training batches on cuda:  85%|████████▍ | 101/119 [00:03<00:00, 31.37batch/s]

Training batches on cuda:  88%|████████▊ | 105/119 [00:03<00:00, 32.59batch/s]

Training batches on cuda:  92%|█████████▏| 109/119 [00:03<00:00, 32.51batch/s]

Training batches on cuda:  95%|█████████▍| 113/119 [00:03<00:00, 31.10batch/s]

Training batches on cuda:  98%|█████████▊| 117/119 [00:03<00:00, 31.31batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:23<00:19,  3.81s/epoch, loss=0.000387, prev_loss=0.000387]
Training epochs on cuda:  60%|██████    | 6/10 [00:23<00:15,  3.86s/epoch, loss=0.000387, prev_loss=0.000387]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/119 [00:00<00:06, 19.05batch/s]

Training batches on cuda:   4%|▍         | 5/119 [00:00<00:05, 21.32batch/s]

Training batches on cuda:   7%|▋         | 8/119 [00:00<00:04, 23.29batch/s]

Training batches on cuda:  10%|█         | 12/119 [00:00<00:04, 25.64batch/s]

Training batches on cuda:  13%|█▎        | 16/119 [00:00<00:03, 27.55batch/s]

Training batches on cuda:  17%|█▋        | 20/119 [00:00<00:03, 28.51batch/s]

Training batches on cuda:  20%|██        | 24/119 [00:00<00:03, 29.51batch/s]

Training batches on cuda:  24%|██▎       | 28/119 [00:00<00:02, 30.97batch/s]

Training batches on cuda:  27%|██▋       | 32/119 [00:01<00:02, 32.08batch/s]

Training batches on cuda:  30%|███       | 36/119 [00:01<00:02, 32.99batch/s]

Training batches on cuda:  34%|███▎      | 40/119 [00:01<00:02, 33.64batch/s]

Training batches on cuda:  37%|███▋      | 44/119 [00:01<00:02, 34.14batch/s]

Training batches on cuda:  40%|████      | 48/119 [00:01<00:02, 34.44batch/s]

Training batches on cuda:  44%|████▎     | 52/119 [00:01<00:01, 34.59batch/s]

Training batches on cuda:  47%|████▋     | 56/119 [00:01<00:01, 34.61batch/s]

Training batches on cuda:  50%|█████     | 60/119 [00:01<00:01, 34.74batch/s]

Training batches on cuda:  54%|█████▍    | 64/119 [00:01<00:01, 34.93batch/s]

Training batches on cuda:  57%|█████▋    | 68/119 [00:02<00:01, 35.17batch/s]

Training batches on cuda:  61%|██████    | 72/119 [00:02<00:01, 35.12batch/s]

Training batches on cuda:  64%|██████▍   | 76/119 [00:02<00:01, 35.02batch/s]

Training batches on cuda:  67%|██████▋   | 80/119 [00:02<00:01, 35.18batch/s]

Training batches on cuda:  71%|███████   | 84/119 [00:02<00:00, 35.28batch/s]

Training batches on cuda:  74%|███████▍  | 88/119 [00:02<00:00, 35.07batch/s]

Training batches on cuda:  77%|███████▋  | 92/119 [00:02<00:00, 34.99batch/s]

Training batches on cuda:  81%|████████  | 96/119 [00:02<00:00, 35.01batch/s]

Training batches on cuda:  84%|████████▍ | 100/119 [00:02<00:00, 35.09batch/s]

Training batches on cuda:  87%|████████▋ | 104/119 [00:03<00:00, 35.05batch/s]

Training batches on cuda:  91%|█████████ | 108/119 [00:03<00:00, 35.03batch/s]

Training batches on cuda:  94%|█████████▍| 112/119 [00:03<00:00, 35.11batch/s]

Training batches on cuda:  97%|█████████▋| 116/119 [00:03<00:00, 35.51batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:26<00:15,  3.86s/epoch, loss=0.000387, prev_loss=0.000387]
Training epochs on cuda:  70%|███████   | 7/10 [00:26<00:11,  3.81s/epoch, loss=0.000387, prev_loss=0.000387]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/119 [00:00<00:04, 26.97batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:03, 29.01batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:03, 30.93batch/s]

Training batches on cuda:  13%|█▎        | 15/119 [00:00<00:03, 32.06batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:03, 32.37batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:00<00:02, 33.10batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:00<00:02, 33.39batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:00<00:02, 33.34batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:02, 33.68batch/s]

Training batches on cuda:  33%|███▎      | 39/119 [00:01<00:02, 33.31batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:01<00:02, 33.69batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:01<00:02, 34.00batch/s]

Training batches on cuda:  43%|████▎     | 51/119 [00:01<00:02, 33.61batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:01<00:01, 33.97batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:01<00:01, 34.18batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:01<00:01, 33.90batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:01<00:01, 33.85batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:02<00:01, 33.45batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:02<00:01, 33.24batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:02<00:01, 33.21batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:02<00:01, 33.60batch/s]

Training batches on cuda:  73%|███████▎  | 87/119 [00:02<00:00, 33.53batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:02<00:00, 33.66batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:02<00:00, 33.65batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:02<00:00, 33.73batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:03<00:00, 33.94batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:03<00:00, 33.65batch/s]

Training batches on cuda:  93%|█████████▎| 111/119 [00:03<00:00, 33.69batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:03<00:00, 33.50batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:30<00:11,  3.81s/epoch, loss=0.000387, prev_loss=0.000387]
Training epochs on cuda:  80%|████████  | 8/10 [00:30<00:07,  3.79s/epoch, loss=0.000387, prev_loss=0.000387]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/119 [00:00<00:04, 24.58batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:04, 26.65batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:03, 28.54batch/s]

Training batches on cuda:  13%|█▎        | 15/119 [00:00<00:03, 30.03batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:03, 30.68batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:00<00:03, 31.46batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:00<00:02, 32.07batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:00<00:02, 32.47batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:02, 31.77batch/s]

Training batches on cuda:  33%|███▎      | 39/119 [00:01<00:02, 32.58batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:01<00:02, 32.78batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:01<00:02, 30.64batch/s]

Training batches on cuda:  43%|████▎     | 51/119 [00:01<00:02, 31.65batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:01<00:01, 32.38batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:01<00:01, 32.99batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:01<00:01, 33.51batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:02<00:01, 33.55batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:02<00:01, 33.66batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:02<00:01, 33.62batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:02<00:01, 33.79batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:02<00:01, 33.92batch/s]

Training batches on cuda:  73%|███████▎  | 87/119 [00:02<00:00, 33.97batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:02<00:00, 34.13batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:02<00:00, 34.07batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:02<00:00, 34.17batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:03<00:00, 34.23batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:03<00:00, 34.29batch/s]

Training batches on cuda:  93%|█████████▎| 111/119 [00:03<00:00, 33.99batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:03<00:00, 34.06batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:34<00:07,  3.79s/epoch, loss=0.000387, prev_loss=0.000387]
Training epochs on cuda:  90%|█████████ | 9/10 [00:34<00:03,  3.80s/epoch, loss=0.000387, prev_loss=0.000387]

Training batches on cuda:   0%|          | 0/119 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/119 [00:00<00:04, 25.08batch/s]

Training batches on cuda:   6%|▌         | 7/119 [00:00<00:04, 27.13batch/s]

Training batches on cuda:   9%|▉         | 11/119 [00:00<00:03, 28.91batch/s]

Training batches on cuda:  13%|█▎        | 15/119 [00:00<00:03, 30.34batch/s]

Training batches on cuda:  16%|█▌        | 19/119 [00:00<00:03, 31.24batch/s]

Training batches on cuda:  19%|█▉        | 23/119 [00:00<00:02, 32.07batch/s]

Training batches on cuda:  23%|██▎       | 27/119 [00:00<00:02, 32.87batch/s]

Training batches on cuda:  26%|██▌       | 31/119 [00:00<00:02, 33.17batch/s]

Training batches on cuda:  29%|██▉       | 35/119 [00:01<00:02, 33.44batch/s]

Training batches on cuda:  33%|███▎      | 39/119 [00:01<00:02, 33.67batch/s]

Training batches on cuda:  36%|███▌      | 43/119 [00:01<00:02, 32.37batch/s]

Training batches on cuda:  39%|███▉      | 47/119 [00:01<00:02, 32.81batch/s]

Training batches on cuda:  43%|████▎     | 51/119 [00:01<00:02, 33.08batch/s]

Training batches on cuda:  46%|████▌     | 55/119 [00:01<00:01, 33.10batch/s]

Training batches on cuda:  50%|████▉     | 59/119 [00:01<00:01, 33.58batch/s]

Training batches on cuda:  53%|█████▎    | 63/119 [00:01<00:01, 33.90batch/s]

Training batches on cuda:  56%|█████▋    | 67/119 [00:02<00:01, 34.06batch/s]

Training batches on cuda:  60%|█████▉    | 71/119 [00:02<00:01, 34.05batch/s]

Training batches on cuda:  63%|██████▎   | 75/119 [00:02<00:01, 34.14batch/s]

Training batches on cuda:  66%|██████▋   | 79/119 [00:02<00:01, 34.09batch/s]

Training batches on cuda:  70%|██████▉   | 83/119 [00:02<00:01, 33.91batch/s]

Training batches on cuda:  73%|███████▎  | 87/119 [00:02<00:00, 33.73batch/s]

Training batches on cuda:  76%|███████▋  | 91/119 [00:02<00:00, 33.76batch/s]

Training batches on cuda:  80%|███████▉  | 95/119 [00:02<00:00, 33.91batch/s]

Training batches on cuda:  83%|████████▎ | 99/119 [00:02<00:00, 34.08batch/s]

Training batches on cuda:  87%|████████▋ | 103/119 [00:03<00:00, 34.03batch/s]

Training batches on cuda:  90%|████████▉ | 107/119 [00:03<00:00, 34.08batch/s]

Training batches on cuda:  93%|█████████▎| 111/119 [00:03<00:00, 33.97batch/s]

Training batches on cuda:  97%|█████████▋| 115/119 [00:03<00:00, 33.79batch/s]

Training batches on cuda: 100%|██████████| 119/119 [00:03<00:00, 35.37batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:37<00:03,  3.80s/epoch, loss=0.000387, prev_loss=0.000387]2021-09-16:19:31:14,744 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:31:15,794 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:31:30,757 INFO     [evaluator.py:657] Evaluation took 14.96s seconds
2021-09-16:19:31:30,925 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00,  8.64s/epoch, loss=0.000387, prev_loss=0.000387]
Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00,  5.42s/epoch, loss=0.000387, prev_loss=0.000387]
2021-09-16:19:31:30,926 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpk3pak348'
2021-09-16:19:31:30,997 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpk3pak348' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.39ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.57ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.59ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.61ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.64ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.66ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:31:45,788 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 19:31:45,793] Trial 88 finished with value: 0.0038048101216308083 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.1636747807151028, 'optimizer.lr': 0.0545353000021814, 'negative_sampler.num_negs_per_pos': 16, 'training.batch_size': 2632}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:31:45,815 WARNING  [api.py:823] No random seed is specified. Setting to 1080159932.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/180 [00:00<00:04, 39.50batch/s]

Training batches on cuda:   7%|▋         | 12/180 [00:00<00:03, 46.06batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:03, 52.55batch/s]

Training batches on cuda:  16%|█▌        | 28/180 [00:00<00:02, 58.37batch/s]

Training batches on cuda:  20%|██        | 36/180 [00:00<00:02, 61.42batch/s]

Training batches on cuda:  24%|██▍       | 43/180 [00:00<00:02, 63.55batch/s]

Training batches on cuda:  28%|██▊       | 51/180 [00:00<00:01, 65.97batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:00<00:01, 68.94batch/s]

Training batches on cuda:  37%|███▋      | 67/180 [00:00<00:01, 71.14batch/s]

Training batches on cuda:  42%|████▏     | 75/180 [00:01<00:01, 72.64batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:01<00:01, 73.84batch/s]

Training batches on cuda:  51%|█████     | 91/180 [00:01<00:01, 67.99batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:01<00:01, 66.54batch/s]

Training batches on cuda:  58%|█████▊    | 105/180 [00:01<00:01, 66.92batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:01<00:00, 68.67batch/s]

Training batches on cuda:  67%|██████▋   | 121/180 [00:01<00:00, 69.83batch/s]

Training batches on cuda:  72%|███████▏  | 129/180 [00:01<00:00, 71.93batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:01<00:00, 73.56batch/s]

Training batches on cuda:  81%|████████  | 145/180 [00:02<00:00, 73.74batch/s]

Training batches on cuda:  85%|████████▌ | 153/180 [00:02<00:00, 70.54batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:02<00:00, 72.35batch/s]

Training batches on cuda:  94%|█████████▍| 169/180 [00:02<00:00, 73.18batch/s]

Training batches on cuda:  98%|█████████▊| 177/180 [00:02<00:00, 74.40batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:02<?, ?epoch/s, loss=0.000587, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:02<00:24,  2.69s/epoch, loss=0.000587, prev_loss=nan]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:03, 48.26batch/s]

Training batches on cuda:   7%|▋         | 13/180 [00:00<00:03, 54.00batch/s]

Training batches on cuda:  12%|█▏        | 21/180 [00:00<00:02, 59.18batch/s]

Training batches on cuda:  15%|█▌        | 27/180 [00:00<00:02, 59.16batch/s]

Training batches on cuda:  19%|█▉        | 34/180 [00:00<00:02, 60.11batch/s]

Training batches on cuda:  22%|██▏       | 40/180 [00:00<00:02, 58.13batch/s]

Training batches on cuda:  26%|██▌       | 46/180 [00:00<00:02, 54.87batch/s]

Training batches on cuda:  29%|██▉       | 52/180 [00:00<00:02, 50.49batch/s]

Training batches on cuda:  33%|███▎      | 60/180 [00:01<00:02, 55.92batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:01<00:01, 60.22batch/s]

Training batches on cuda:  42%|████▏     | 76/180 [00:01<00:01, 64.61batch/s]

Training batches on cuda:  47%|████▋     | 84/180 [00:01<00:01, 67.61batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:01<00:01, 70.02batch/s]

Training batches on cuda:  56%|█████▌    | 100/180 [00:01<00:01, 71.99batch/s]

Training batches on cuda:  60%|██████    | 108/180 [00:01<00:00, 73.29batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:01<00:00, 74.03batch/s]

Training batches on cuda:  69%|██████▉   | 124/180 [00:01<00:00, 74.57batch/s]

Training batches on cuda:  73%|███████▎  | 132/180 [00:01<00:00, 75.50batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:02<00:00, 76.38batch/s]

Training batches on cuda:  82%|████████▏ | 148/180 [00:02<00:00, 76.78batch/s]

Training batches on cuda:  87%|████████▋ | 156/180 [00:02<00:00, 76.67batch/s]

Training batches on cuda:  91%|█████████ | 164/180 [00:02<00:00, 75.74batch/s]

Training batches on cuda:  96%|█████████▌| 172/180 [00:02<00:00, 73.65batch/s]

Training batches on cuda: 100%|██████████| 180/180 [00:02<00:00, 71.52batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:24,  2.69s/epoch, loss=0.000583, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:05<00:21,  2.73s/epoch, loss=0.000583, prev_loss=nan]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/180 [00:00<00:06, 26.90batch/s]

Training batches on cuda:   5%|▌         | 9/180 [00:00<00:05, 31.71batch/s]

Training batches on cuda:   9%|▉         | 16/180 [00:00<00:04, 37.75batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:03, 43.58batch/s]

Training batches on cuda:  17%|█▋        | 31/180 [00:00<00:03, 49.46batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:00<00:02, 54.10batch/s]

Training batches on cuda:  25%|██▌       | 45/180 [00:00<00:02, 58.04batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:00<00:02, 61.21batch/s]

Training batches on cuda:  33%|███▎      | 60/180 [00:00<00:01, 63.06batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:01<00:01, 64.87batch/s]

Training batches on cuda:  42%|████▏     | 75/180 [00:01<00:01, 65.38batch/s]

Training batches on cuda:  46%|████▌     | 82/180 [00:01<00:01, 65.68batch/s]

Training batches on cuda:  49%|████▉     | 89/180 [00:01<00:01, 65.91batch/s]

Training batches on cuda:  53%|█████▎    | 96/180 [00:01<00:01, 65.76batch/s]

Training batches on cuda:  57%|█████▋    | 103/180 [00:01<00:01, 66.37batch/s]

Training batches on cuda:  61%|██████    | 110/180 [00:01<00:01, 66.04batch/s]

Training batches on cuda:  66%|██████▌   | 118/180 [00:01<00:00, 67.76batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:01<00:00, 68.28batch/s]

Training batches on cuda:  73%|███████▎  | 132/180 [00:02<00:00, 67.95batch/s]

Training batches on cuda:  77%|███████▋  | 139/180 [00:02<00:00, 66.40batch/s]

Training batches on cuda:  81%|████████  | 146/180 [00:02<00:00, 64.81batch/s]

Training batches on cuda:  85%|████████▌ | 153/180 [00:02<00:00, 65.86batch/s]

Training batches on cuda:  89%|████████▉ | 160/180 [00:02<00:00, 66.10batch/s]

Training batches on cuda:  93%|█████████▎| 167/180 [00:02<00:00, 65.30batch/s]

Training batches on cuda:  97%|█████████▋| 174/180 [00:02<00:00, 62.27batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:21,  2.73s/epoch, loss=0.000583, prev_loss=0.000583]
Training epochs on cuda:  30%|███       | 3/10 [00:08<00:19,  2.81s/epoch, loss=0.000583, prev_loss=0.000583]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:03, 52.02batch/s]

Training batches on cuda:   7%|▋         | 13/180 [00:00<00:02, 56.13batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:02, 58.83batch/s]

Training batches on cuda:  15%|█▌        | 27/180 [00:00<00:02, 60.83batch/s]

Training batches on cuda:  19%|█▉        | 35/180 [00:00<00:02, 63.50batch/s]

Training batches on cuda:  24%|██▍       | 43/180 [00:00<00:02, 65.57batch/s]

Training batches on cuda:  28%|██▊       | 51/180 [00:00<00:01, 67.24batch/s]

Training batches on cuda:  32%|███▏      | 58/180 [00:00<00:01, 67.22batch/s]

Training batches on cuda:  37%|███▋      | 66/180 [00:00<00:01, 68.71batch/s]

Training batches on cuda:  41%|████      | 74/180 [00:01<00:01, 71.35batch/s]

Training batches on cuda:  46%|████▌     | 82/180 [00:01<00:01, 72.79batch/s]

Training batches on cuda:  50%|█████     | 90/180 [00:01<00:01, 71.42batch/s]

Training batches on cuda:  54%|█████▍    | 98/180 [00:01<00:01, 70.98batch/s]

Training batches on cuda:  59%|█████▉    | 106/180 [00:01<00:01, 69.89batch/s]

Training batches on cuda:  63%|██████▎   | 113/180 [00:01<00:00, 68.39batch/s]

Training batches on cuda:  67%|██████▋   | 121/180 [00:01<00:00, 69.94batch/s]

Training batches on cuda:  72%|███████▏  | 129/180 [00:01<00:00, 71.71batch/s]

Training batches on cuda:  76%|███████▌  | 137/180 [00:01<00:00, 73.03batch/s]

Training batches on cuda:  81%|████████  | 145/180 [00:02<00:00, 74.67batch/s]

Training batches on cuda:  85%|████████▌ | 153/180 [00:02<00:00, 71.92batch/s]

Training batches on cuda:  89%|████████▉ | 161/180 [00:02<00:00, 72.10batch/s]

Training batches on cuda:  94%|█████████▍| 169/180 [00:02<00:00, 73.62batch/s]

Training batches on cuda:  99%|█████████▉| 178/180 [00:02<00:00, 75.55batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:19,  2.81s/epoch, loss=0.000584, prev_loss=0.000583]
Training epochs on cuda:  40%|████      | 4/10 [00:11<00:16,  2.78s/epoch, loss=0.000584, prev_loss=0.000583]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 6/180 [00:00<00:03, 57.66batch/s]

Training batches on cuda:   7%|▋         | 13/180 [00:00<00:02, 59.14batch/s]

Training batches on cuda:  12%|█▏        | 21/180 [00:00<00:02, 62.38batch/s]

Training batches on cuda:  16%|█▌        | 29/180 [00:00<00:02, 66.37batch/s]

Training batches on cuda:  21%|██        | 37/180 [00:00<00:02, 68.85batch/s]

Training batches on cuda:  25%|██▌       | 45/180 [00:00<00:01, 70.14batch/s]

Training batches on cuda:  29%|██▉       | 53/180 [00:00<00:01, 72.25batch/s]

Training batches on cuda:  34%|███▍      | 61/180 [00:00<00:01, 72.98batch/s]

Training batches on cuda:  38%|███▊      | 69/180 [00:00<00:01, 73.64batch/s]

Training batches on cuda:  43%|████▎     | 77/180 [00:01<00:01, 74.48batch/s]

Training batches on cuda:  47%|████▋     | 85/180 [00:01<00:01, 74.80batch/s]

Training batches on cuda:  52%|█████▏    | 93/180 [00:01<00:01, 75.31batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:01<00:01, 75.61batch/s]

Training batches on cuda:  61%|██████    | 109/180 [00:01<00:00, 75.30batch/s]

Training batches on cuda:  65%|██████▌   | 117/180 [00:01<00:00, 75.67batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:01<00:00, 75.83batch/s]

Training batches on cuda:  74%|███████▍  | 133/180 [00:01<00:00, 75.76batch/s]

Training batches on cuda:  78%|███████▊  | 141/180 [00:01<00:00, 75.67batch/s]

Training batches on cuda:  83%|████████▎ | 149/180 [00:02<00:00, 71.33batch/s]

Training batches on cuda:  87%|████████▋ | 157/180 [00:02<00:00, 72.30batch/s]

Training batches on cuda:  92%|█████████▏| 165/180 [00:02<00:00, 73.57batch/s]

Training batches on cuda:  96%|█████████▌| 173/180 [00:02<00:00, 74.21batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:13<00:16,  2.78s/epoch, loss=0.000584, prev_loss=0.000584]
Training epochs on cuda:  50%|█████     | 5/10 [00:13<00:13,  2.75s/epoch, loss=0.000584, prev_loss=0.000584]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:03, 49.15batch/s]

Training batches on cuda:   7%|▋         | 12/180 [00:00<00:03, 52.64batch/s]

Training batches on cuda:  11%|█         | 20/180 [00:00<00:02, 57.89batch/s]

Training batches on cuda:  16%|█▌        | 28/180 [00:00<00:02, 62.33batch/s]

Training batches on cuda:  20%|██        | 36/180 [00:00<00:02, 66.34batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:00<00:01, 68.64batch/s]

Training batches on cuda:  29%|██▉       | 52/180 [00:00<00:01, 69.72batch/s]

Training batches on cuda:  33%|███▎      | 60/180 [00:00<00:01, 71.52batch/s]

Training batches on cuda:  37%|███▋      | 67/180 [00:00<00:01, 70.81batch/s]

Training batches on cuda:  42%|████▏     | 75/180 [00:01<00:01, 72.55batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:01<00:01, 69.71batch/s]

Training batches on cuda:  50%|█████     | 90/180 [00:01<00:01, 68.90batch/s]

Training batches on cuda:  54%|█████▍    | 97/180 [00:01<00:01, 68.12batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:01<00:01, 66.51batch/s]

Training batches on cuda:  62%|██████▏   | 111/180 [00:01<00:01, 66.77batch/s]

Training batches on cuda:  66%|██████▌   | 119/180 [00:01<00:00, 69.30batch/s]

Training batches on cuda:  71%|███████   | 127/180 [00:01<00:00, 69.64batch/s]

Training batches on cuda:  75%|███████▌  | 135/180 [00:01<00:00, 69.73batch/s]

Training batches on cuda:  79%|███████▉  | 143/180 [00:02<00:00, 70.22batch/s]

Training batches on cuda:  84%|████████▍ | 151/180 [00:02<00:00, 70.03batch/s]

Training batches on cuda:  88%|████████▊ | 159/180 [00:02<00:00, 69.69batch/s]

Training batches on cuda:  93%|█████████▎| 167/180 [00:02<00:00, 70.01batch/s]

Training batches on cuda:  97%|█████████▋| 175/180 [00:02<00:00, 71.50batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:16<00:13,  2.75s/epoch, loss=0.000584, prev_loss=0.000584]
Training epochs on cuda:  60%|██████    | 6/10 [00:16<00:11,  2.77s/epoch, loss=0.000584, prev_loss=0.000584]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 5/180 [00:00<00:03, 44.42batch/s]

Training batches on cuda:   7%|▋         | 12/180 [00:00<00:03, 49.30batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:03, 50.30batch/s]

Training batches on cuda:  14%|█▍        | 25/180 [00:00<00:02, 52.95batch/s]

Training batches on cuda:  17%|█▋        | 31/180 [00:00<00:02, 54.75batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:00<00:02, 56.55batch/s]

Training batches on cuda:  25%|██▌       | 45/180 [00:00<00:02, 57.59batch/s]

Training batches on cuda:  29%|██▉       | 52/180 [00:00<00:02, 58.48batch/s]

Training batches on cuda:  33%|███▎      | 59/180 [00:00<00:01, 60.87batch/s]

Training batches on cuda:  37%|███▋      | 67/180 [00:01<00:01, 64.63batch/s]

Training batches on cuda:  42%|████▏     | 75/180 [00:01<00:01, 67.98batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:01<00:01, 70.86batch/s]

Training batches on cuda:  51%|█████     | 91/180 [00:01<00:01, 72.43batch/s]

Training batches on cuda:  55%|█████▌    | 99/180 [00:01<00:01, 65.09batch/s]

Training batches on cuda:  59%|█████▉    | 107/180 [00:01<00:01, 68.06batch/s]

Training batches on cuda:  64%|██████▍   | 115/180 [00:01<00:00, 70.05batch/s]

Training batches on cuda:  68%|██████▊   | 123/180 [00:01<00:00, 71.46batch/s]

Training batches on cuda:  73%|███████▎  | 131/180 [00:01<00:00, 72.60batch/s]

Training batches on cuda:  77%|███████▋  | 139/180 [00:02<00:00, 73.40batch/s]

Training batches on cuda:  82%|████████▏ | 147/180 [00:02<00:00, 74.08batch/s]

Training batches on cuda:  86%|████████▌ | 155/180 [00:02<00:00, 74.05batch/s]

Training batches on cuda:  91%|█████████ | 163/180 [00:02<00:00, 74.49batch/s]

Training batches on cuda:  95%|█████████▌| 171/180 [00:02<00:00, 72.88batch/s]

Training batches on cuda:  99%|█████████▉| 179/180 [00:02<00:00, 71.02batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:19<00:11,  2.77s/epoch, loss=0.000584, prev_loss=0.000584]
Training epochs on cuda:  70%|███████   | 7/10 [00:19<00:08,  2.79s/epoch, loss=0.000584, prev_loss=0.000584]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/180 [00:00<00:04, 35.52batch/s]

Training batches on cuda:   6%|▌         | 11/180 [00:00<00:04, 40.99batch/s]

Training batches on cuda:  10%|█         | 18/180 [00:00<00:03, 46.49batch/s]

Training batches on cuda:  14%|█▍        | 25/180 [00:00<00:03, 51.49batch/s]

Training batches on cuda:  18%|█▊        | 33/180 [00:00<00:02, 56.75batch/s]

Training batches on cuda:  23%|██▎       | 41/180 [00:00<00:02, 61.04batch/s]

Training batches on cuda:  27%|██▋       | 48/180 [00:00<00:02, 61.80batch/s]

Training batches on cuda:  31%|███       | 55/180 [00:00<00:01, 63.80batch/s]

Training batches on cuda:  34%|███▍      | 62/180 [00:00<00:01, 65.48batch/s]

Training batches on cuda:  38%|███▊      | 69/180 [00:01<00:01, 66.61batch/s]

Training batches on cuda:  42%|████▏     | 76/180 [00:01<00:01, 67.48batch/s]

Training batches on cuda:  47%|████▋     | 84/180 [00:01<00:01, 69.19batch/s]

Training batches on cuda:  51%|█████     | 92/180 [00:01<00:01, 71.65batch/s]

Training batches on cuda:  56%|█████▌    | 100/180 [00:01<00:01, 73.47batch/s]

Training batches on cuda:  60%|██████    | 108/180 [00:01<00:00, 73.65batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:01<00:00, 74.90batch/s]

Training batches on cuda:  69%|██████▉   | 124/180 [00:01<00:00, 75.71batch/s]

Training batches on cuda:  73%|███████▎  | 132/180 [00:01<00:00, 76.20batch/s]

Training batches on cuda:  78%|███████▊  | 140/180 [00:01<00:00, 76.06batch/s]

Training batches on cuda:  82%|████████▏ | 148/180 [00:02<00:00, 76.32batch/s]

Training batches on cuda:  87%|████████▋ | 156/180 [00:02<00:00, 74.88batch/s]

Training batches on cuda:  91%|█████████ | 164/180 [00:02<00:00, 74.88batch/s]

Training batches on cuda:  96%|█████████▌| 172/180 [00:02<00:00, 75.18batch/s]

Training batches on cuda: 100%|██████████| 180/180 [00:02<00:00, 76.02batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:22<00:08,  2.79s/epoch, loss=0.000584, prev_loss=0.000584]
Training epochs on cuda:  80%|████████  | 8/10 [00:22<00:05,  2.79s/epoch, loss=0.000584, prev_loss=0.000584]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/180 [00:00<00:05, 29.54batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:04, 34.71batch/s]

Training batches on cuda:   9%|▉         | 16/180 [00:00<00:04, 39.47batch/s]

Training batches on cuda:  13%|█▎        | 23/180 [00:00<00:03, 45.03batch/s]

Training batches on cuda:  17%|█▋        | 30/180 [00:00<00:03, 49.80batch/s]

Training batches on cuda:  21%|██        | 37/180 [00:00<00:02, 53.94batch/s]

Training batches on cuda:  24%|██▍       | 44/180 [00:00<00:02, 57.39batch/s]

Training batches on cuda:  28%|██▊       | 51/180 [00:00<00:02, 59.79batch/s]

Training batches on cuda:  32%|███▏      | 58/180 [00:00<00:02, 55.90batch/s]

Training batches on cuda:  36%|███▌      | 65/180 [00:01<00:01, 58.70batch/s]

Training batches on cuda:  40%|████      | 72/180 [00:01<00:01, 60.16batch/s]

Training batches on cuda:  44%|████▍     | 79/180 [00:01<00:01, 58.66batch/s]

Training batches on cuda:  48%|████▊     | 86/180 [00:01<00:01, 59.94batch/s]

Training batches on cuda:  52%|█████▏    | 93/180 [00:01<00:01, 61.29batch/s]

Training batches on cuda:  56%|█████▌    | 101/180 [00:01<00:01, 63.78batch/s]

Training batches on cuda:  60%|██████    | 108/180 [00:01<00:01, 65.27batch/s]

Training batches on cuda:  64%|██████▍   | 116/180 [00:01<00:00, 67.74batch/s]

Training batches on cuda:  69%|██████▉   | 125/180 [00:01<00:00, 70.94batch/s]

Training batches on cuda:  74%|███████▍  | 133/180 [00:02<00:00, 73.40batch/s]

Training batches on cuda:  79%|███████▉  | 142/180 [00:02<00:00, 75.89batch/s]

Training batches on cuda:  84%|████████▍ | 151/180 [00:02<00:00, 77.54batch/s]

Training batches on cuda:  88%|████████▊ | 159/180 [00:02<00:00, 72.63batch/s]

Training batches on cuda:  93%|█████████▎| 167/180 [00:02<00:00, 72.81batch/s]

Training batches on cuda:  97%|█████████▋| 175/180 [00:02<00:00, 74.41batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:25<00:05,  2.79s/epoch, loss=0.000584, prev_loss=0.000584]
Training epochs on cuda:  90%|█████████ | 9/10 [00:25<00:02,  2.83s/epoch, loss=0.000584, prev_loss=0.000584]

Training batches on cuda:   0%|          | 0/180 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 4/180 [00:00<00:04, 35.70batch/s]

Training batches on cuda:   6%|▌         | 10/180 [00:00<00:04, 39.38batch/s]

Training batches on cuda:   9%|▉         | 17/180 [00:00<00:03, 44.16batch/s]

Training batches on cuda:  13%|█▎        | 24/180 [00:00<00:03, 49.26batch/s]

Training batches on cuda:  17%|█▋        | 31/180 [00:00<00:02, 53.61batch/s]

Training batches on cuda:  21%|██        | 38/180 [00:00<00:02, 57.05batch/s]

Training batches on cuda:  25%|██▌       | 45/180 [00:00<00:02, 60.09batch/s]

Training batches on cuda:  29%|██▉       | 52/180 [00:00<00:02, 61.95batch/s]

Training batches on cuda:  33%|███▎      | 60/180 [00:00<00:01, 64.17batch/s]

Training batches on cuda:  38%|███▊      | 68/180 [00:01<00:01, 67.76batch/s]

Training batches on cuda:  42%|████▏     | 75/180 [00:01<00:01, 63.78batch/s]

Training batches on cuda:  46%|████▌     | 83/180 [00:01<00:01, 65.46batch/s]

Training batches on cuda:  50%|█████     | 90/180 [00:01<00:01, 66.61batch/s]

Training batches on cuda:  54%|█████▍    | 97/180 [00:01<00:01, 67.46batch/s]

Training batches on cuda:  58%|█████▊    | 104/180 [00:01<00:01, 68.06batch/s]

Training batches on cuda:  62%|██████▏   | 112/180 [00:01<00:00, 69.29batch/s]

Training batches on cuda:  67%|██████▋   | 120/180 [00:01<00:00, 69.38batch/s]

Training batches on cuda:  71%|███████   | 128/180 [00:01<00:00, 70.78batch/s]

Training batches on cuda:  76%|███████▌  | 136/180 [00:02<00:00, 72.87batch/s]

Training batches on cuda:  80%|████████  | 144/180 [00:02<00:00, 73.24batch/s]

Training batches on cuda:  84%|████████▍ | 152/180 [00:02<00:00, 74.81batch/s]

Training batches on cuda:  89%|████████▉ | 160/180 [00:02<00:00, 75.64batch/s]

Training batches on cuda:  93%|█████████▎| 168/180 [00:02<00:00, 76.32batch/s]

Training batches on cuda:  98%|█████████▊| 176/180 [00:02<00:00, 76.99batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:28<00:02,  2.83s/epoch, loss=0.000584, prev_loss=0.000584]2021-09-16:19:32:14,361 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:32:15,311 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:32:30,243 INFO     [evaluator.py:657] Evaluation took 14.93s seconds
2021-09-16:19:32:30,359 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00,  7.64s/epoch, loss=0.000584, prev_loss=0.000584]
Training epochs on cuda: 100%|██████████| 10/10 [00:44<00:00,  4.41s/epoch, loss=0.000584, prev_loss=0.000584]
2021-09-16:19:32:30,360 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmplf8lgr11'
2021-09-16:19:32:30,452 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmplf8lgr11' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.51ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.57ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.59ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.61ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.62ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.62ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.63ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.64ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.64ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.64ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.64ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:32:45,278 INFO     [evaluator.py:657] Evaluation took 14.80s seconds
[I 2021-09-16 19:32:45,282] Trial 89 finished with value: -0.004751561374011892 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.11240423080563183, 'optimizer.lr': 0.059829180421207745, 'negative_sampler.num_negs_per_pos': 11, 'training.batch_size': 1732}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:32:45,305 WARNING  [api.py:823] No random seed is specified. Setting to 544543861.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 15.99batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:06, 19.24batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 22.13batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 25.20batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 27.74batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 29.83batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 31.17batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 32.36batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 31.88batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 33.28batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 33.13batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 32.61batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 32.62batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 31.95batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 32.66batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 33.06batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 33.48batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 33.87batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 34.23batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 34.18batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 34.18batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 34.39batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 34.30batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 34.51batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 34.50batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 34.65batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 34.66batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 34.62batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 34.66batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 34.76batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 35.03batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 34.53batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000412, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:36,  4.03s/epoch, loss=0.000412, prev_loss=nan]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 17.89batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 20.70batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 23.40batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 25.76batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 27.83batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 29.97batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 30.62batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 31.85batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 32.63batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 33.20batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 33.67batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 33.91batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 34.27batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 34.57batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 34.67batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 34.61batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 34.68batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 34.47batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 34.51batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 34.59batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 34.71batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 34.71batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 34.95batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 34.94batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 35.39batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 34.54batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 34.19batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 34.52batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 34.71batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 34.77batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 34.54batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 34.33batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:36,  4.03s/epoch, loss=0.000403, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:32,  4.02s/epoch, loss=0.000403, prev_loss=nan]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:06, 18.21batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 21.09batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 24.08batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 26.63batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 28.50batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 30.09batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 31.41batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 32.23batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 33.10batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 30.36batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 31.27batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 31.59batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 31.48batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 31.42batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 31.70batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:02, 31.98batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:02<00:01, 32.22batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 33.30batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 34.30batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 35.21batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 35.85batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 36.03batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 35.74batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 35.45batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 35.26batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 35.20batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 34.60batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 34.13batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 34.12batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 34.06batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 34.02batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 33.96batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:32,  4.02s/epoch, loss=0.000353, prev_loss=0.000403]
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:28,  4.02s/epoch, loss=0.000353, prev_loss=0.000403]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/128 [00:00<00:13,  9.25batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:10, 11.83batch/s]

Training batches on cuda:   7%|▋         | 9/128 [00:00<00:08, 14.77batch/s]

Training batches on cuda:  10%|█         | 13/128 [00:00<00:06, 17.88batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:05, 20.90batch/s]

Training batches on cuda:  16%|█▋        | 21/128 [00:00<00:04, 23.79batch/s]

Training batches on cuda:  20%|█▉        | 25/128 [00:00<00:03, 26.34batch/s]

Training batches on cuda:  23%|██▎       | 29/128 [00:00<00:03, 28.47batch/s]

Training batches on cuda:  26%|██▌       | 33/128 [00:01<00:03, 30.19batch/s]

Training batches on cuda:  29%|██▉       | 37/128 [00:01<00:02, 31.57batch/s]

Training batches on cuda:  32%|███▏      | 41/128 [00:01<00:02, 32.56batch/s]

Training batches on cuda:  35%|███▌      | 45/128 [00:01<00:02, 33.37batch/s]

Training batches on cuda:  38%|███▊      | 49/128 [00:01<00:02, 33.75batch/s]

Training batches on cuda:  41%|████▏     | 53/128 [00:01<00:02, 34.23batch/s]

Training batches on cuda:  45%|████▍     | 57/128 [00:01<00:02, 34.34batch/s]

Training batches on cuda:  48%|████▊     | 61/128 [00:01<00:01, 34.71batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:01<00:01, 34.81batch/s]

Training batches on cuda:  54%|█████▍    | 69/128 [00:02<00:01, 34.82batch/s]

Training batches on cuda:  57%|█████▋    | 73/128 [00:02<00:01, 31.82batch/s]

Training batches on cuda:  60%|██████    | 77/128 [00:02<00:01, 32.60batch/s]

Training batches on cuda:  63%|██████▎   | 81/128 [00:02<00:01, 33.28batch/s]

Training batches on cuda:  66%|██████▋   | 85/128 [00:02<00:01, 33.90batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:02<00:01, 34.05batch/s]

Training batches on cuda:  73%|███████▎  | 93/128 [00:02<00:01, 34.31batch/s]

Training batches on cuda:  76%|███████▌  | 97/128 [00:02<00:00, 34.39batch/s]

Training batches on cuda:  79%|███████▉  | 101/128 [00:03<00:00, 34.56batch/s]

Training batches on cuda:  82%|████████▏ | 105/128 [00:03<00:00, 34.77batch/s]

Training batches on cuda:  85%|████████▌ | 109/128 [00:03<00:00, 34.91batch/s]

Training batches on cuda:  88%|████████▊ | 113/128 [00:03<00:00, 34.92batch/s]

Training batches on cuda:  91%|█████████▏| 117/128 [00:03<00:00, 34.94batch/s]

Training batches on cuda:  95%|█████████▍| 121/128 [00:03<00:00, 35.07batch/s]

Training batches on cuda:  98%|█████████▊| 125/128 [00:03<00:00, 35.00batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:16<00:28,  4.02s/epoch, loss=0.000251, prev_loss=0.000353]
Training epochs on cuda:  40%|████      | 4/10 [00:16<00:24,  4.01s/epoch, loss=0.000251, prev_loss=0.000353]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 17.58batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 20.54batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 23.37batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 25.92batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 28.02batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 29.86batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 31.15batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 32.17batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 33.09batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 33.58batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 34.05batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 34.18batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 34.41batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 34.70batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 34.75batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 34.96batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 35.07batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 34.94batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 35.00batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 35.00batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 35.16batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 35.02batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 34.95batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 34.91batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 34.60batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:02<00:00, 34.80batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 34.84batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 31.40batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 32.47batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 33.17batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 33.61batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 34.10batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:24,  4.01s/epoch, loss=0.000177, prev_loss=0.000251]
Training epochs on cuda:  50%|█████     | 5/10 [00:20<00:20,  4.00s/epoch, loss=0.000177, prev_loss=0.000251]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 17.87batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 20.91batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 23.94batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 26.50batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 28.68batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 30.45batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 31.76batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 32.66batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 31.93batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 32.90batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 34.07batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 34.85batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 35.60batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 35.69batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:01, 35.33batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 35.40batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 35.13batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 35.31batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 34.51batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 34.03batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 33.65batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 33.00batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 32.68batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:01, 32.66batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 32.51batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 32.33batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 32.37batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 32.48batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 32.48batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 32.40batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 32.27batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 32.06batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:24<00:20,  4.00s/epoch, loss=0.000155, prev_loss=0.000177]
Training epochs on cuda:  60%|██████    | 6/10 [00:24<00:16,  4.03s/epoch, loss=0.000155, prev_loss=0.000177]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 17.97batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 21.14batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 24.44batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 27.10batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 29.51batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 31.21batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 32.95batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:02, 33.99batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:00<00:02, 34.66batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 35.13batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 35.67batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 36.23batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 36.62batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 35.97batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:01, 35.75batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 35.35batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 35.21batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:01<00:01, 35.06batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 34.72batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 34.74batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 34.79batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 35.02batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 35.17batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 35.05batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 33.90batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:02<00:00, 34.25batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 34.30batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 34.34batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 34.30batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 34.44batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 34.58batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 34.23batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:28<00:16,  4.03s/epoch, loss=0.000139, prev_loss=0.000155]
Training epochs on cuda:  70%|███████   | 7/10 [00:28<00:11,  3.99s/epoch, loss=0.000139, prev_loss=0.000155]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:06, 19.98batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 22.72batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 25.37batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 27.60batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 29.29batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 30.85batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 31.77batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 32.59batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 33.06batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 32.40batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 32.83batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 33.07batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 33.52batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 33.64batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 33.46batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 33.79batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 33.26batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 33.04batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 33.86batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 34.91batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 35.43batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 35.82batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 36.79batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 36.64batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 36.62batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:02<00:00, 35.88batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 35.66batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 35.19batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 35.36batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 35.01batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 34.59batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 34.66batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:31<00:11,  3.99s/epoch, loss=0.000128, prev_loss=0.000139]
Training epochs on cuda:  80%|████████  | 8/10 [00:31<00:07,  3.98s/epoch, loss=0.000128, prev_loss=0.000139]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 16.61batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:06, 19.65batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 22.58batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 25.38batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 27.78batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 29.86batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 31.28batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 32.32batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 32.98batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 33.26batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 33.79batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 32.79batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 33.56batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 34.11batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 34.38batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 34.57batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 34.55batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 34.35batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 34.21batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 34.39batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 34.38batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 33.28batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 33.18batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:01, 33.52batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 34.34batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 33.92batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 34.05batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 32.50batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 32.30batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 33.11batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 33.37batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 33.42batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:35<00:07,  3.98s/epoch, loss=0.000125, prev_loss=0.000128]
Training epochs on cuda:  90%|█████████ | 9/10 [00:35<00:03,  3.99s/epoch, loss=0.000125, prev_loss=0.000128]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:08, 15.17batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:06, 18.23batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 20.77batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 23.57batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:04, 26.04batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 28.02batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 29.90batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 31.23batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 32.35batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 33.14batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 32.74batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 32.79batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 33.20batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 33.32batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 32.59batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:02, 32.51batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:02<00:01, 32.30batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 31.88batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 31.44batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 31.66batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 31.46batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 32.42batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 33.07batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:01, 33.67batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:03<00:00, 33.87batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 33.42batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 31.27batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 30.78batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 28.38batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 29.99batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 31.05batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 31.99batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:40<00:03,  3.99s/epoch, loss=0.000124, prev_loss=0.000125]2021-09-16:19:33:26,37 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:33:27,72 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:33:41,991 INFO     [evaluator.py:657] Evaluation took 14.92s seconds
2021-09-16:19:33:42,107 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00,  8.88s/epoch, loss=0.000124, prev_loss=0.000125]
Training epochs on cuda: 100%|██████████| 10/10 [00:56<00:00,  5.63s/epoch, loss=0.000124, prev_loss=0.000125]
2021-09-16:19:33:42,107 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpoy41auci'
2021-09-16:19:33:42,189 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpoy41auci' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.41ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.48ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.53ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.65ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.66ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:33:56,956 INFO     [evaluator.py:657] Evaluation took 14.75s seconds
[I 2021-09-16 19:33:56,960] Trial 90 finished with value: 0.9717278019411042 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.014862630259842899, 'optimizer.lr': 0.04430153423164086, 'negative_sampler.num_negs_per_pos': 19, 'training.batch_size': 2432}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:33:56,984 WARNING  [api.py:823] No random seed is specified. Setting to 3552722196.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 25.38batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 28.44batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 30.97batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:03, 32.89batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:03, 34.74batch/s]

Training batches on cuda:  17%|█▋        | 24/140 [00:00<00:03, 34.89batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:03, 35.04batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:03, 35.31batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:00<00:02, 36.39batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:01<00:02, 36.79batch/s]

Training batches on cuda:  31%|███▏      | 44/140 [00:01<00:02, 36.96batch/s]

Training batches on cuda:  34%|███▍      | 48/140 [00:01<00:02, 37.26batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:01<00:02, 37.23batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:01<00:02, 36.97batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:01<00:02, 37.91batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:01, 38.64batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 38.77batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:01<00:01, 38.94batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 38.48batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 37.81batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 37.66batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 36.41batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 35.33batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:02<00:01, 36.24batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:02<00:01, 36.93batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:02<00:00, 37.46batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:02<00:00, 37.73batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 38.26batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 38.45batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 38.74batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 38.72batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 38.84batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:03<00:00, 39.07batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:03<00:00, 38.98batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.00045, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:35,  3.92s/epoch, loss=0.00045, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:07, 18.38batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 21.76batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 25.07batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 28.08batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:03, 30.55batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 32.54batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 34.20batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 35.28batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:02, 36.04batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:02, 36.75batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 37.18batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 37.43batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 37.32batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 37.85batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 38.30batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 38.64batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:01, 38.79batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 38.19batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:01<00:01, 37.42batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 37.43batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 36.56batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 36.57batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 36.21batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 35.91batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:02<00:01, 35.83batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:02<00:01, 36.19batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:02<00:00, 36.16batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:02<00:00, 36.31batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 36.54batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 36.62batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 36.58batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 36.08batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 36.32batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:03<00:00, 37.01batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:03<00:00, 36.44batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:07<00:35,  3.92s/epoch, loss=0.00044, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:31,  3.95s/epoch, loss=0.00044, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 26.06batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 28.76batch/s]

Training batches on cuda:   9%|▊         | 12/140 [00:00<00:04, 31.42batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:03, 31.98batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:03, 33.60batch/s]

Training batches on cuda:  17%|█▋        | 24/140 [00:00<00:03, 34.95batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:03, 35.88batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:02, 36.44batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:00<00:02, 37.05batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:01<00:02, 37.40batch/s]

Training batches on cuda:  31%|███▏      | 44/140 [00:01<00:02, 37.67batch/s]

Training batches on cuda:  34%|███▍      | 48/140 [00:01<00:02, 37.98batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:01<00:02, 38.10batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:01<00:02, 38.17batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:01<00:02, 38.42batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:02, 37.78batch/s]

Training batches on cuda:  49%|████▊     | 68/140 [00:01<00:01, 38.08batch/s]

Training batches on cuda:  51%|█████▏    | 72/140 [00:01<00:01, 37.07batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:02<00:01, 37.14batch/s]

Training batches on cuda:  57%|█████▋    | 80/140 [00:02<00:01, 36.64batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:02<00:01, 36.70batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:02<00:01, 36.89batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:02<00:01, 37.66batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:02<00:01, 36.94batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:02<00:01, 33.26batch/s]

Training batches on cuda:  74%|███████▍  | 104/140 [00:02<00:01, 34.64batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:02<00:00, 35.83batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:03<00:00, 36.13batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:03<00:00, 35.81batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:03<00:00, 36.77batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:03<00:00, 37.38batch/s]

Training batches on cuda:  91%|█████████▏| 128/140 [00:03<00:00, 36.55batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:03<00:00, 37.22batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:03<00:00, 37.17batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:31,  3.95s/epoch, loss=0.000381, prev_loss=0.00044]
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:27,  3.98s/epoch, loss=0.000381, prev_loss=0.00044]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:07, 17.77batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 21.09batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 24.40batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 27.34batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 30.11batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 32.02batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 33.42batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 34.81batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:00<00:02, 35.71batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:02, 35.72batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 36.19batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 36.51batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 37.03batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 37.61batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 37.90batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 38.07batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:01, 38.18batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:01<00:01, 38.51batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:01, 38.35batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 38.26batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 37.98batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 38.00batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 37.72batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 37.39batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:02<00:01, 37.25batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:02<00:01, 37.26batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:02<00:00, 37.88batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:02<00:00, 38.30batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 38.91batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 39.21batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:03<00:00, 39.58batch/s]

Training batches on cuda:  92%|█████████▏| 129/140 [00:03<00:00, 39.87batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:03<00:00, 40.01batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:03<00:00, 40.06batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:27,  3.98s/epoch, loss=0.000261, prev_loss=0.000381]
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:23,  3.95s/epoch, loss=0.000261, prev_loss=0.000381]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 24.44batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 27.54batch/s]

Training batches on cuda:   9%|▊         | 12/140 [00:00<00:04, 30.58batch/s]

Training batches on cuda:  12%|█▏        | 17/140 [00:00<00:03, 32.97batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 34.90batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 36.59batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:02, 37.64batch/s]

Training batches on cuda:  26%|██▋       | 37/140 [00:00<00:02, 38.62batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 39.27batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 39.65batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:01<00:02, 39.79batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:01<00:02, 39.82batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:01<00:02, 39.85batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:01, 39.86batch/s]

Training batches on cuda:  49%|████▊     | 68/140 [00:01<00:01, 39.88batch/s]

Training batches on cuda:  52%|█████▏    | 73/140 [00:01<00:01, 40.02batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:01<00:01, 40.24batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 40.31batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:02<00:01, 40.24batch/s]

Training batches on cuda:  66%|██████▋   | 93/140 [00:02<00:01, 40.26batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:02<00:01, 40.16batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:02<00:01, 36.43batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:02<00:00, 37.40batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:02<00:00, 38.36batch/s]

Training batches on cuda:  84%|████████▎ | 117/140 [00:02<00:00, 38.97batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 39.60batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 39.92batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:03<00:00, 40.10batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:03<00:00, 40.09batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:19<00:23,  3.95s/epoch, loss=0.000195, prev_loss=0.000261]
Training epochs on cuda:  50%|█████     | 5/10 [00:19<00:19,  3.88s/epoch, loss=0.000195, prev_loss=0.000261]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 26.58batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 29.43batch/s]

Training batches on cuda:   9%|▊         | 12/140 [00:00<00:03, 32.06batch/s]

Training batches on cuda:  12%|█▏        | 17/140 [00:00<00:03, 34.10batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 35.83batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 37.04batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:02, 38.02batch/s]

Training batches on cuda:  26%|██▋       | 37/140 [00:00<00:02, 38.80batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 39.41batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 39.80batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:01<00:02, 39.95batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:01<00:02, 39.89batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:01<00:01, 40.06batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:01, 40.23batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:01<00:01, 40.47batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:01, 40.50batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:02<00:01, 40.60batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 40.68batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 40.65batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:02<00:01, 40.55batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:02<00:00, 40.52batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:02<00:00, 40.62batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:02<00:00, 40.54batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:02<00:00, 40.55batch/s]

Training batches on cuda:  86%|████████▋ | 121/140 [00:03<00:00, 40.48batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 40.38batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 40.48batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:03<00:00, 40.38batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:23<00:19,  3.88s/epoch, loss=0.000157, prev_loss=0.000195]
Training epochs on cuda:  60%|██████    | 6/10 [00:23<00:15,  3.83s/epoch, loss=0.000157, prev_loss=0.000195]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 24.55batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:05, 26.58batch/s]

Training batches on cuda:   9%|▊         | 12/140 [00:00<00:04, 29.59batch/s]

Training batches on cuda:  12%|█▏        | 17/140 [00:00<00:03, 32.21batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 34.35batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 35.80batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:02, 37.04batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:00<00:02, 38.13batch/s]

Training batches on cuda:  29%|██▉       | 41/140 [00:01<00:02, 38.73batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 39.30batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 39.65batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:01<00:02, 39.76batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:01<00:01, 39.89batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:01, 40.14batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:01<00:01, 40.30batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:01, 40.26batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:02<00:01, 40.32batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 40.36batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 40.29batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:02<00:01, 40.27batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:02<00:00, 40.29batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:02<00:00, 40.05batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:02<00:00, 39.96batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:02<00:00, 39.77batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:03<00:00, 40.06batch/s]

Training batches on cuda:  89%|████████▉ | 125/140 [00:03<00:00, 40.27batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 40.23batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:03<00:00, 40.26batch/s]

Training batches on cuda: 100%|██████████| 140/140 [00:03<00:00, 41.62batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:27<00:15,  3.83s/epoch, loss=0.000142, prev_loss=0.000157]
Training epochs on cuda:  70%|███████   | 7/10 [00:27<00:11,  3.82s/epoch, loss=0.000142, prev_loss=0.000157]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 27.02batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 29.76batch/s]

Training batches on cuda:   9%|▊         | 12/140 [00:00<00:03, 32.24batch/s]

Training batches on cuda:  12%|█▏        | 17/140 [00:00<00:03, 34.48batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 36.06batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 37.16batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:02, 38.01batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:00<00:02, 38.66batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:01<00:02, 38.83batch/s]

Training batches on cuda:  31%|███▏      | 44/140 [00:01<00:02, 39.13batch/s]

Training batches on cuda:  35%|███▌      | 49/140 [00:01<00:02, 39.57batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 39.65batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 35.99batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 37.11batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:01, 37.90batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:01<00:01, 38.49batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:01<00:01, 39.06batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:02<00:01, 39.49batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 39.68batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 39.93batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:02<00:01, 40.15batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:02<00:00, 40.17batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:02<00:00, 40.07batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:02<00:00, 39.96batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:02<00:00, 39.83batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 39.77batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 39.67batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 39.70batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 39.68batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:03<00:00, 40.06batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:30<00:11,  3.82s/epoch, loss=0.000135, prev_loss=0.000142]
Training epochs on cuda:  80%|████████  | 8/10 [00:30<00:07,  3.79s/epoch, loss=0.000135, prev_loss=0.000142]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 24.63batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 27.81batch/s]

Training batches on cuda:   9%|▊         | 12/140 [00:00<00:04, 30.66batch/s]

Training batches on cuda:  11%|█▏        | 16/140 [00:00<00:03, 31.98batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:03, 33.07batch/s]

Training batches on cuda:  17%|█▋        | 24/140 [00:00<00:03, 33.83batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:03, 35.06batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:00<00:03, 35.51batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:01<00:02, 35.94batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:01<00:02, 36.49batch/s]

Training batches on cuda:  31%|███▏      | 44/140 [00:01<00:02, 36.75batch/s]

Training batches on cuda:  34%|███▍      | 48/140 [00:01<00:02, 37.25batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:01<00:02, 37.25batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:01<00:02, 36.56batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:01<00:02, 36.23batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:02, 35.99batch/s]

Training batches on cuda:  49%|████▊     | 68/140 [00:01<00:01, 36.68batch/s]

Training batches on cuda:  51%|█████▏    | 72/140 [00:01<00:01, 37.28batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:02<00:01, 37.68batch/s]

Training batches on cuda:  57%|█████▋    | 80/140 [00:02<00:01, 37.73batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:02<00:01, 38.20batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:02<00:01, 38.25batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:02<00:01, 38.15batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:02<00:01, 38.26batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:02<00:01, 38.24batch/s]

Training batches on cuda:  74%|███████▍  | 104/140 [00:02<00:00, 38.44batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:02<00:00, 38.37batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:03<00:00, 38.50batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:03<00:00, 38.62batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:03<00:00, 38.51batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:03<00:00, 38.75batch/s]

Training batches on cuda:  91%|█████████▏| 128/140 [00:03<00:00, 38.46batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:03<00:00, 38.26batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:03<00:00, 38.37batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:34<00:07,  3.79s/epoch, loss=0.000131, prev_loss=0.000135]
Training epochs on cuda:  90%|█████████ | 9/10 [00:34<00:03,  3.84s/epoch, loss=0.000131, prev_loss=0.000135]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 25.02batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 27.85batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 30.43batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:03, 32.40batch/s]

Training batches on cuda:  14%|█▎        | 19/140 [00:00<00:03, 34.18batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:03, 35.58batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 36.54batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:02, 37.26batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:00<00:02, 37.84batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:01<00:02, 37.97batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:01<00:02, 38.19batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 38.23batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 38.32batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:01<00:02, 38.36batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:02, 38.46batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 38.31batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:01, 38.23batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:01<00:01, 38.33batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:01<00:01, 38.41batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:02<00:01, 38.36batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 38.54batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:02<00:01, 38.58batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 38.44batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:02<00:01, 38.56batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:02<00:01, 38.72batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:02<00:00, 38.60batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:02<00:00, 38.63batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:02<00:00, 38.55batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 38.85batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 38.66batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 38.63batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 38.56batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 38.46batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:03<00:00, 38.49batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:03<00:00, 38.52batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:38<00:03,  3.84s/epoch, loss=0.000129, prev_loss=0.000131]2021-09-16:19:34:36,148 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:34:37,132 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:34:52,81 INFO     [evaluator.py:657] Evaluation took 14.95s seconds
2021-09-16:19:34:52,205 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00,  8.66s/epoch, loss=0.000129, prev_loss=0.000131]
Training epochs on cuda: 100%|██████████| 10/10 [00:54<00:00,  5.47s/epoch, loss=0.000129, prev_loss=0.000131]
2021-09-16:19:34:52,206 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp9dw9zlch'
2021-09-16:19:34:52,281 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp9dw9zlch' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.32ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.40ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.47ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.52ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.63ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:35:07,70 INFO     [evaluator.py:657] Evaluation took 14.76s seconds
[I 2021-09-16 19:35:07,075] Trial 91 finished with value: 0.9743218415375482 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.014285166102205982, 'optimizer.lr': 0.04246570384044034, 'negative_sampler.num_negs_per_pos': 19, 'training.batch_size': 2232}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:35:07,97 WARNING  [api.py:823] No random seed is specified. Setting to 1524322484.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/128 [00:00<00:05, 24.16batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:04, 26.38batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:04, 28.66batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:03, 30.44batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:03, 31.74batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:03, 32.91batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:00<00:03, 33.15batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:00<00:02, 32.98batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:02, 33.24batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 33.36batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 33.56batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 33.16batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:02, 33.68batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:02, 33.44batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:02, 34.09batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:01<00:01, 34.98batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:01<00:01, 35.27batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:02<00:01, 35.66batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:02<00:01, 35.82batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 34.80batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 35.22batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 35.43batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:01, 35.57batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:02<00:00, 35.71batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:02<00:00, 35.84batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:02<00:00, 34.97batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:03<00:00, 34.34batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:03<00:00, 33.91batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 33.90batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 33.75batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 33.73batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 33.54batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000416, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:35,  3.98s/epoch, loss=0.000416, prev_loss=nan]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:08, 15.40batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:07, 17.46batch/s]

Training batches on cuda:   6%|▋         | 8/128 [00:00<00:06, 19.48batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:05, 21.27batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:05, 22.60batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:04, 23.57batch/s]

Training batches on cuda:  16%|█▌        | 20/128 [00:00<00:04, 24.82batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:04, 25.62batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:01<00:03, 27.17batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:01<00:03, 28.86batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:03, 30.18batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 31.24batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 32.16batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 32.85batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:02, 33.23batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:02, 33.54batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:02, 33.57batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:02<00:01, 33.65batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:02<00:01, 33.68batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:02<00:01, 33.80batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:02<00:01, 33.16batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 33.09batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 32.86batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 33.32batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:01, 33.60batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:03<00:00, 33.68batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:03<00:00, 34.13batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:03<00:00, 34.48batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:03<00:00, 34.59batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:03<00:00, 34.45batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 34.10batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 33.51batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 33.37batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:04<00:00, 33.24batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:35,  3.98s/epoch, loss=0.000413, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:32,  4.06s/epoch, loss=0.000413, prev_loss=nan]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:06, 18.79batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 21.61batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 24.17batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 26.38batch/s]

Training batches on cuda:  13%|█▎        | 17/128 [00:00<00:04, 25.46batch/s]

Training batches on cuda:  16%|█▋        | 21/128 [00:00<00:03, 27.17batch/s]

Training batches on cuda:  20%|█▉        | 25/128 [00:00<00:03, 28.70batch/s]

Training batches on cuda:  23%|██▎       | 29/128 [00:00<00:03, 29.93batch/s]

Training batches on cuda:  26%|██▌       | 33/128 [00:01<00:03, 30.92batch/s]

Training batches on cuda:  29%|██▉       | 37/128 [00:01<00:02, 31.58batch/s]

Training batches on cuda:  32%|███▏      | 41/128 [00:01<00:02, 31.73batch/s]

Training batches on cuda:  35%|███▌      | 45/128 [00:01<00:02, 32.06batch/s]

Training batches on cuda:  38%|███▊      | 49/128 [00:01<00:02, 32.17batch/s]

Training batches on cuda:  41%|████▏     | 53/128 [00:01<00:02, 32.04batch/s]

Training batches on cuda:  45%|████▍     | 57/128 [00:01<00:02, 32.09batch/s]

Training batches on cuda:  48%|████▊     | 61/128 [00:01<00:02, 32.39batch/s]

Training batches on cuda:  51%|█████     | 65/128 [00:02<00:01, 32.25batch/s]

Training batches on cuda:  54%|█████▍    | 69/128 [00:02<00:01, 32.26batch/s]

Training batches on cuda:  57%|█████▋    | 73/128 [00:02<00:01, 32.22batch/s]

Training batches on cuda:  60%|██████    | 77/128 [00:02<00:01, 32.25batch/s]

Training batches on cuda:  63%|██████▎   | 81/128 [00:02<00:01, 32.47batch/s]

Training batches on cuda:  66%|██████▋   | 85/128 [00:02<00:01, 31.88batch/s]

Training batches on cuda:  70%|██████▉   | 89/128 [00:02<00:01, 31.77batch/s]

Training batches on cuda:  73%|███████▎  | 93/128 [00:02<00:01, 31.57batch/s]

Training batches on cuda:  76%|███████▌  | 97/128 [00:03<00:00, 31.42batch/s]

Training batches on cuda:  79%|███████▉  | 101/128 [00:03<00:00, 31.53batch/s]

Training batches on cuda:  82%|████████▏ | 105/128 [00:03<00:00, 31.63batch/s]

Training batches on cuda:  85%|████████▌ | 109/128 [00:03<00:00, 31.72batch/s]

Training batches on cuda:  88%|████████▊ | 113/128 [00:03<00:00, 31.64batch/s]

Training batches on cuda:  91%|█████████▏| 117/128 [00:03<00:00, 31.66batch/s]

Training batches on cuda:  95%|█████████▍| 121/128 [00:03<00:00, 31.56batch/s]

Training batches on cuda:  98%|█████████▊| 125/128 [00:03<00:00, 31.62batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:12<00:32,  4.06s/epoch, loss=0.000413, prev_loss=0.000413]
Training epochs on cuda:  30%|███       | 3/10 [00:12<00:28,  4.12s/epoch, loss=0.000413, prev_loss=0.000413]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/128 [00:00<00:04, 25.28batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:04, 27.28batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:04, 28.99batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:03, 30.38batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:03, 31.33batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:03, 32.05batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:00<00:03, 32.65batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:00<00:02, 33.07batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:02, 33.35batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 33.57batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 33.90batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 33.72batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:02, 34.04batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:02, 34.06batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:02, 33.84batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:01<00:01, 33.74batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:02<00:01, 33.59batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:02<00:01, 33.53batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:02<00:01, 33.76batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 33.64batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 33.68batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 33.84batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:01, 33.93batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:02<00:00, 33.82batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:02<00:00, 33.78batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:03<00:00, 33.95batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:03<00:00, 33.97batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:03<00:00, 33.81batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 33.69batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 34.01batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 34.11batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 33.99batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:16<00:28,  4.12s/epoch, loss=0.000413, prev_loss=0.000413]
Training epochs on cuda:  40%|████      | 4/10 [00:16<00:24,  4.10s/epoch, loss=0.000413, prev_loss=0.000413]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/128 [00:00<00:05, 22.75batch/s]

Training batches on cuda:   5%|▌         | 7/128 [00:00<00:04, 24.98batch/s]

Training batches on cuda:   9%|▊         | 11/128 [00:00<00:04, 27.47batch/s]

Training batches on cuda:  12%|█▏        | 15/128 [00:00<00:03, 29.00batch/s]

Training batches on cuda:  15%|█▍        | 19/128 [00:00<00:03, 30.39batch/s]

Training batches on cuda:  18%|█▊        | 23/128 [00:00<00:03, 31.06batch/s]

Training batches on cuda:  21%|██        | 27/128 [00:00<00:03, 31.89batch/s]

Training batches on cuda:  24%|██▍       | 31/128 [00:00<00:02, 32.37batch/s]

Training batches on cuda:  27%|██▋       | 35/128 [00:01<00:02, 32.20batch/s]

Training batches on cuda:  30%|███       | 39/128 [00:01<00:02, 32.84batch/s]

Training batches on cuda:  34%|███▎      | 43/128 [00:01<00:02, 32.92batch/s]

Training batches on cuda:  37%|███▋      | 47/128 [00:01<00:02, 32.70batch/s]

Training batches on cuda:  40%|███▉      | 51/128 [00:01<00:02, 32.74batch/s]

Training batches on cuda:  43%|████▎     | 55/128 [00:01<00:02, 32.89batch/s]

Training batches on cuda:  46%|████▌     | 59/128 [00:01<00:02, 33.39batch/s]

Training batches on cuda:  49%|████▉     | 63/128 [00:01<00:01, 33.63batch/s]

Training batches on cuda:  52%|█████▏    | 67/128 [00:02<00:01, 33.55batch/s]

Training batches on cuda:  55%|█████▌    | 71/128 [00:02<00:01, 33.79batch/s]

Training batches on cuda:  59%|█████▊    | 75/128 [00:02<00:01, 33.92batch/s]

Training batches on cuda:  62%|██████▏   | 79/128 [00:02<00:01, 30.54batch/s]

Training batches on cuda:  65%|██████▍   | 83/128 [00:02<00:01, 31.30batch/s]

Training batches on cuda:  68%|██████▊   | 87/128 [00:02<00:01, 32.07batch/s]

Training batches on cuda:  71%|███████   | 91/128 [00:02<00:01, 32.52batch/s]

Training batches on cuda:  74%|███████▍  | 95/128 [00:02<00:01, 32.78batch/s]

Training batches on cuda:  77%|███████▋  | 99/128 [00:03<00:00, 32.91batch/s]

Training batches on cuda:  80%|████████  | 103/128 [00:03<00:00, 33.14batch/s]

Training batches on cuda:  84%|████████▎ | 107/128 [00:03<00:00, 33.25batch/s]

Training batches on cuda:  87%|████████▋ | 111/128 [00:03<00:00, 33.31batch/s]

Training batches on cuda:  90%|████████▉ | 115/128 [00:03<00:00, 32.86batch/s]

Training batches on cuda:  93%|█████████▎| 119/128 [00:03<00:00, 33.05batch/s]

Training batches on cuda:  96%|█████████▌| 123/128 [00:03<00:00, 33.29batch/s]

Training batches on cuda:  99%|█████████▉| 127/128 [00:03<00:00, 33.40batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:24,  4.10s/epoch, loss=0.000409, prev_loss=0.000413]
Training epochs on cuda:  50%|█████     | 5/10 [00:20<00:20,  4.10s/epoch, loss=0.000409, prev_loss=0.000413]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:06, 19.88batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 22.57batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 25.11batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 27.17batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 28.52batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 30.15batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 30.43batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 30.33batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:03, 30.77batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 31.21batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 31.45batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 32.05batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 32.42batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 31.45batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 31.36batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:02, 32.24batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:02<00:01, 32.56batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 32.03batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 31.25batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 31.17batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 31.20batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 32.36batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 33.44batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:01, 33.66batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:03<00:00, 33.67batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 33.74batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 33.88batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 33.80batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 33.82batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 33.79batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 33.75batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 33.71batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:24<00:20,  4.10s/epoch, loss=0.000387, prev_loss=0.000409]
Training epochs on cuda:  60%|██████    | 6/10 [00:24<00:16,  4.11s/epoch, loss=0.000387, prev_loss=0.000409]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/128 [00:00<00:14,  8.77batch/s]

Training batches on cuda:   4%|▍         | 5/128 [00:00<00:10, 11.22batch/s]

Training batches on cuda:   6%|▋         | 8/128 [00:00<00:08, 13.80batch/s]

Training batches on cuda:   9%|▉         | 12/128 [00:00<00:06, 16.82batch/s]

Training batches on cuda:  12%|█▎        | 16/128 [00:00<00:05, 19.86batch/s]

Training batches on cuda:  16%|█▌        | 20/128 [00:00<00:04, 22.56batch/s]

Training batches on cuda:  19%|█▉        | 24/128 [00:00<00:04, 25.15batch/s]

Training batches on cuda:  22%|██▏       | 28/128 [00:00<00:03, 27.29batch/s]

Training batches on cuda:  25%|██▌       | 32/128 [00:01<00:03, 29.14batch/s]

Training batches on cuda:  28%|██▊       | 36/128 [00:01<00:03, 30.38batch/s]

Training batches on cuda:  31%|███▏      | 40/128 [00:01<00:02, 31.32batch/s]

Training batches on cuda:  34%|███▍      | 44/128 [00:01<00:02, 31.97batch/s]

Training batches on cuda:  38%|███▊      | 48/128 [00:01<00:02, 32.50batch/s]

Training batches on cuda:  41%|████      | 52/128 [00:01<00:02, 32.78batch/s]

Training batches on cuda:  44%|████▍     | 56/128 [00:01<00:02, 32.89batch/s]

Training batches on cuda:  47%|████▋     | 60/128 [00:01<00:02, 32.71batch/s]

Training batches on cuda:  50%|█████     | 64/128 [00:02<00:01, 32.95batch/s]

Training batches on cuda:  53%|█████▎    | 68/128 [00:02<00:01, 33.01batch/s]

Training batches on cuda:  56%|█████▋    | 72/128 [00:02<00:01, 32.75batch/s]

Training batches on cuda:  59%|█████▉    | 76/128 [00:02<00:01, 33.11batch/s]

Training batches on cuda:  62%|██████▎   | 80/128 [00:02<00:01, 32.99batch/s]

Training batches on cuda:  66%|██████▌   | 84/128 [00:02<00:01, 32.90batch/s]

Training batches on cuda:  69%|██████▉   | 88/128 [00:02<00:01, 33.22batch/s]

Training batches on cuda:  72%|███████▏  | 92/128 [00:02<00:01, 33.13batch/s]

Training batches on cuda:  75%|███████▌  | 96/128 [00:02<00:00, 33.23batch/s]

Training batches on cuda:  78%|███████▊  | 100/128 [00:03<00:00, 33.47batch/s]

Training batches on cuda:  81%|████████▏ | 104/128 [00:03<00:00, 33.32batch/s]

Training batches on cuda:  84%|████████▍ | 108/128 [00:03<00:00, 33.20batch/s]

Training batches on cuda:  88%|████████▊ | 112/128 [00:03<00:00, 33.34batch/s]

Training batches on cuda:  91%|█████████ | 116/128 [00:03<00:00, 33.35batch/s]

Training batches on cuda:  94%|█████████▍| 120/128 [00:03<00:00, 33.41batch/s]

Training batches on cuda:  97%|█████████▋| 124/128 [00:03<00:00, 33.30batch/s]

Training batches on cuda: 100%|██████████| 128/128 [00:03<00:00, 33.80batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:28<00:16,  4.11s/epoch, loss=0.00034, prev_loss=0.000387] 
Training epochs on cuda:  70%|███████   | 7/10 [00:28<00:12,  4.12s/epoch, loss=0.00034, prev_loss=0.000387]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:06, 18.96batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 21.63batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:04, 24.26batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 26.31batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 28.23batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 29.68batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 30.72batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 31.56batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 32.14batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 31.88batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 32.18batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 31.92batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 31.98batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 32.33batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 32.27batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:02, 32.36batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:02<00:01, 31.93batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 31.67batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 31.39batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 32.02batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 32.38batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 32.83batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 33.03batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:01, 32.99batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:03<00:00, 32.76batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 32.47batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 31.46batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 32.26batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 32.58batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 32.81batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 32.45batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 32.87batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:33<00:12,  4.12s/epoch, loss=0.000271, prev_loss=0.00034]
Training epochs on cuda:  80%|████████  | 8/10 [00:33<00:08,  4.14s/epoch, loss=0.000271, prev_loss=0.00034]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 17.87batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:05, 20.59batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 23.40batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 25.79batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 27.69batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 29.11batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 30.24batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 29.75batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:03, 30.99batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 31.84batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 31.95batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 32.52batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 31.79batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 32.09batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 32.80batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 33.20batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:02<00:01, 32.74batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 33.31batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 33.55batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 33.79batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 33.91batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 33.84batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 33.95batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:01, 33.91batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:03<00:00, 33.54batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 33.56batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 33.86batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 33.84batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 33.80batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 33.88batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 33.89batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 33.67batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:37<00:08,  4.14s/epoch, loss=0.000246, prev_loss=0.000271]
Training epochs on cuda:  90%|█████████ | 9/10 [00:37<00:04,  4.13s/epoch, loss=0.000246, prev_loss=0.000271]

Training batches on cuda:   0%|          | 0/128 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/128 [00:00<00:07, 17.32batch/s]

Training batches on cuda:   5%|▍         | 6/128 [00:00<00:06, 20.22batch/s]

Training batches on cuda:   8%|▊         | 10/128 [00:00<00:05, 23.06batch/s]

Training batches on cuda:  11%|█         | 14/128 [00:00<00:04, 25.55batch/s]

Training batches on cuda:  14%|█▍        | 18/128 [00:00<00:03, 27.64batch/s]

Training batches on cuda:  17%|█▋        | 22/128 [00:00<00:03, 29.28batch/s]

Training batches on cuda:  20%|██        | 26/128 [00:00<00:03, 30.60batch/s]

Training batches on cuda:  23%|██▎       | 30/128 [00:00<00:03, 31.66batch/s]

Training batches on cuda:  27%|██▋       | 34/128 [00:01<00:02, 32.34batch/s]

Training batches on cuda:  30%|██▉       | 38/128 [00:01<00:02, 32.70batch/s]

Training batches on cuda:  33%|███▎      | 42/128 [00:01<00:02, 33.28batch/s]

Training batches on cuda:  36%|███▌      | 46/128 [00:01<00:02, 33.61batch/s]

Training batches on cuda:  39%|███▉      | 50/128 [00:01<00:02, 33.83batch/s]

Training batches on cuda:  42%|████▏     | 54/128 [00:01<00:02, 34.01batch/s]

Training batches on cuda:  45%|████▌     | 58/128 [00:01<00:02, 34.18batch/s]

Training batches on cuda:  48%|████▊     | 62/128 [00:01<00:01, 33.31batch/s]

Training batches on cuda:  52%|█████▏    | 66/128 [00:01<00:01, 33.46batch/s]

Training batches on cuda:  55%|█████▍    | 70/128 [00:02<00:01, 33.83batch/s]

Training batches on cuda:  58%|█████▊    | 74/128 [00:02<00:01, 33.83batch/s]

Training batches on cuda:  61%|██████    | 78/128 [00:02<00:01, 33.42batch/s]

Training batches on cuda:  64%|██████▍   | 82/128 [00:02<00:01, 34.33batch/s]

Training batches on cuda:  67%|██████▋   | 86/128 [00:02<00:01, 34.99batch/s]

Training batches on cuda:  70%|███████   | 90/128 [00:02<00:01, 35.28batch/s]

Training batches on cuda:  73%|███████▎  | 94/128 [00:02<00:00, 35.76batch/s]

Training batches on cuda:  77%|███████▋  | 98/128 [00:02<00:00, 36.16batch/s]

Training batches on cuda:  80%|███████▉  | 102/128 [00:03<00:00, 36.21batch/s]

Training batches on cuda:  83%|████████▎ | 106/128 [00:03<00:00, 36.61batch/s]

Training batches on cuda:  86%|████████▌ | 110/128 [00:03<00:00, 36.59batch/s]

Training batches on cuda:  89%|████████▉ | 114/128 [00:03<00:00, 36.45batch/s]

Training batches on cuda:  92%|█████████▏| 118/128 [00:03<00:00, 36.18batch/s]

Training batches on cuda:  95%|█████████▌| 122/128 [00:03<00:00, 36.31batch/s]

Training batches on cuda:  98%|█████████▊| 126/128 [00:03<00:00, 36.10batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:41<00:04,  4.13s/epoch, loss=0.000248, prev_loss=0.000246]2021-09-16:19:35:48,757 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:35:49,708 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:36:04,618 INFO     [evaluator.py:657] Evaluation took 14.91s seconds
2021-09-16:19:36:04,794 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00,  8.89s/epoch, loss=0.000248, prev_loss=0.000246]
Training epochs on cuda: 100%|██████████| 10/10 [00:57<00:00,  5.72s/epoch, loss=0.000248, prev_loss=0.000246]
2021-09-16:19:36:04,795 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmph5x2xxsj'
2021-09-16:19:36:04,848 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmph5x2xxsj' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.51ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.64ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:36:19,611 INFO     [evaluator.py:657] Evaluation took 14.75s seconds
[I 2021-09-16 19:36:19,615] Trial 92 finished with value: 0.8978219245914283 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.06150408491034006, 'optimizer.lr': 0.04334055260992982, 'negative_sampler.num_negs_per_pos': 20, 'training.batch_size': 2432}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:36:19,638 WARNING  [api.py:823] No random seed is specified. Setting to 1846699535.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:07, 18.85batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 21.41batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 23.90batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 25.96batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 27.57batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:04, 28.93batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 29.81batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 30.40batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 30.93batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 31.46batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:03, 32.09batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 32.08batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 32.06batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 32.37batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 32.52batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 31.14batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:02<00:02, 31.72batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 32.06batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:02, 31.93batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 31.68batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 31.91batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 32.41batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 32.66batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 33.04batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:03<00:01, 32.10batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 31.61batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 31.07batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 30.97batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 30.93batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 30.80batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 30.43batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:04<00:00, 30.12batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:04<00:00, 29.81batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:04<00:00, 29.86batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:04<00:00, 29.78batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:04<00:00, 29.62batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000448, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:42,  4.70s/epoch, loss=0.000448, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:06, 19.72batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:05, 22.38batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 25.00batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 27.03batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 28.68batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 29.91batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 30.82batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 31.39batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 31.93batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 32.04batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:03, 32.41batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 32.82batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 33.01batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 32.90batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 32.77batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 32.83batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:02<00:02, 32.94batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 32.93batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:02, 30.37batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 31.19batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 31.89batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 32.07batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 32.41batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 32.85batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:03<00:01, 33.01batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 33.01batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 33.05batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 33.29batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 32.95batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 33.13batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 33.25batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 33.31batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 33.35batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:04<00:00, 33.32batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:04<00:00, 33.32batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:09<00:42,  4.70s/epoch, loss=0.000425, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:37,  4.66s/epoch, loss=0.000425, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/140 [00:00<00:14,  9.89batch/s]

Training batches on cuda:   4%|▎         | 5/140 [00:00<00:10, 12.45batch/s]

Training batches on cuda:   6%|▋         | 9/140 [00:00<00:08, 15.34batch/s]

Training batches on cuda:   9%|▉         | 13/140 [00:00<00:06, 18.39batch/s]

Training batches on cuda:  12%|█▏        | 17/140 [00:00<00:05, 20.95batch/s]

Training batches on cuda:  15%|█▌        | 21/140 [00:00<00:05, 23.50batch/s]

Training batches on cuda:  18%|█▊        | 25/140 [00:00<00:04, 25.70batch/s]

Training batches on cuda:  21%|██        | 29/140 [00:00<00:04, 27.57batch/s]

Training batches on cuda:  24%|██▎       | 33/140 [00:01<00:03, 29.07batch/s]

Training batches on cuda:  26%|██▋       | 37/140 [00:01<00:03, 29.30batch/s]

Training batches on cuda:  29%|██▉       | 41/140 [00:01<00:03, 30.52batch/s]

Training batches on cuda:  32%|███▏      | 45/140 [00:01<00:03, 31.50batch/s]

Training batches on cuda:  35%|███▌      | 49/140 [00:01<00:02, 32.22batch/s]

Training batches on cuda:  38%|███▊      | 53/140 [00:01<00:02, 32.48batch/s]

Training batches on cuda:  41%|████      | 57/140 [00:01<00:02, 31.92batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:01<00:02, 30.56batch/s]

Training batches on cuda:  46%|████▋     | 65/140 [00:02<00:02, 31.33batch/s]

Training batches on cuda:  49%|████▉     | 69/140 [00:02<00:02, 31.64batch/s]

Training batches on cuda:  52%|█████▏    | 73/140 [00:02<00:02, 32.21batch/s]

Training batches on cuda:  55%|█████▌    | 77/140 [00:02<00:01, 32.55batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:02<00:01, 32.90batch/s]

Training batches on cuda:  61%|██████    | 85/140 [00:02<00:01, 33.35batch/s]

Training batches on cuda:  64%|██████▎   | 89/140 [00:02<00:01, 32.32batch/s]

Training batches on cuda:  66%|██████▋   | 93/140 [00:02<00:01, 32.45batch/s]

Training batches on cuda:  69%|██████▉   | 97/140 [00:03<00:01, 32.81batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:03<00:01, 33.19batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:03<00:01, 33.08batch/s]

Training batches on cuda:  78%|███████▊  | 109/140 [00:03<00:00, 33.18batch/s]

Training batches on cuda:  81%|████████  | 113/140 [00:03<00:00, 33.36batch/s]

Training batches on cuda:  84%|████████▎ | 117/140 [00:03<00:00, 33.36batch/s]

Training batches on cuda:  86%|████████▋ | 121/140 [00:03<00:00, 33.64batch/s]

Training batches on cuda:  89%|████████▉ | 125/140 [00:03<00:00, 33.77batch/s]

Training batches on cuda:  92%|█████████▏| 129/140 [00:04<00:00, 33.69batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:04<00:00, 33.51batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:04<00:00, 33.53batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:37,  4.66s/epoch, loss=0.000309, prev_loss=0.000425]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:32,  4.62s/epoch, loss=0.000309, prev_loss=0.000425]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 24.02batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:05, 26.12batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 27.90batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:04, 29.42batch/s]

Training batches on cuda:  14%|█▎        | 19/140 [00:00<00:03, 30.64batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:03, 31.44batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 32.10batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:03, 31.89batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:01<00:03, 32.53batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:01<00:03, 32.90batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:01<00:02, 33.23batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 33.42batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 33.53batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:01<00:02, 33.61batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:02, 33.62batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 33.72batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:02<00:02, 33.91batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:02<00:02, 33.77batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:02<00:01, 33.75batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:02<00:01, 33.84batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 33.82batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:02<00:01, 33.95batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 33.92batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:02<00:01, 33.89batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:02<00:01, 34.03batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:03<00:01, 33.97batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:03<00:00, 34.03batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:03<00:00, 34.08batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 33.84batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 33.81batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 33.88batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 33.86batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 33.95batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:04<00:00, 34.13batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:04<00:00, 33.90batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:32,  4.62s/epoch, loss=0.000206, prev_loss=0.000309]
Training epochs on cuda:  40%|████      | 4/10 [00:18<00:27,  4.57s/epoch, loss=0.000206, prev_loss=0.000309]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:08, 15.58batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:07, 18.47batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:06, 21.30batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:05, 23.92batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 26.01batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:04, 27.89batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 29.09batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 30.37batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 31.30batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 32.15batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 32.68batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 33.06batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 33.13batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 33.45batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 33.49batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 33.67batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:02<00:02, 33.84batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 32.32batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:02, 32.41batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 32.67batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 32.66batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 32.98batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 33.18batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 32.94batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:03<00:01, 33.05batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 33.41batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 33.47batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 33.31batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 33.30batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 33.00batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 32.15batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 30.16batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:04<00:00, 30.83batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:04<00:00, 30.91batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:04<00:00, 31.07batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:27,  4.57s/epoch, loss=0.000161, prev_loss=0.000206]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:22,  4.57s/epoch, loss=0.000161, prev_loss=0.000206]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:06, 19.93batch/s]

Training batches on cuda:   4%|▎         | 5/140 [00:00<00:06, 22.07batch/s]

Training batches on cuda:   6%|▋         | 9/140 [00:00<00:05, 24.16batch/s]

Training batches on cuda:   9%|▉         | 13/140 [00:00<00:04, 25.89batch/s]

Training batches on cuda:  12%|█▏        | 17/140 [00:00<00:04, 27.13batch/s]

Training batches on cuda:  14%|█▍        | 20/140 [00:00<00:04, 27.80batch/s]

Training batches on cuda:  17%|█▋        | 24/140 [00:00<00:03, 29.27batch/s]

Training batches on cuda:  20%|██        | 28/140 [00:00<00:03, 30.53batch/s]

Training batches on cuda:  23%|██▎       | 32/140 [00:01<00:03, 31.54batch/s]

Training batches on cuda:  26%|██▌       | 36/140 [00:01<00:03, 32.15batch/s]

Training batches on cuda:  29%|██▊       | 40/140 [00:01<00:03, 32.88batch/s]

Training batches on cuda:  31%|███▏      | 44/140 [00:01<00:02, 33.74batch/s]

Training batches on cuda:  34%|███▍      | 48/140 [00:01<00:02, 34.20batch/s]

Training batches on cuda:  37%|███▋      | 52/140 [00:01<00:02, 34.19batch/s]

Training batches on cuda:  40%|████      | 56/140 [00:01<00:02, 33.72batch/s]

Training batches on cuda:  43%|████▎     | 60/140 [00:01<00:02, 33.48batch/s]

Training batches on cuda:  46%|████▌     | 64/140 [00:01<00:02, 33.24batch/s]

Training batches on cuda:  49%|████▊     | 68/140 [00:02<00:02, 33.17batch/s]

Training batches on cuda:  51%|█████▏    | 72/140 [00:02<00:02, 33.33batch/s]

Training batches on cuda:  54%|█████▍    | 76/140 [00:02<00:01, 33.03batch/s]

Training batches on cuda:  57%|█████▋    | 80/140 [00:02<00:01, 32.54batch/s]

Training batches on cuda:  60%|██████    | 84/140 [00:02<00:01, 32.61batch/s]

Training batches on cuda:  63%|██████▎   | 88/140 [00:02<00:01, 32.59batch/s]

Training batches on cuda:  66%|██████▌   | 92/140 [00:02<00:01, 32.78batch/s]

Training batches on cuda:  69%|██████▊   | 96/140 [00:02<00:01, 32.57batch/s]

Training batches on cuda:  71%|███████▏  | 100/140 [00:03<00:01, 32.53batch/s]

Training batches on cuda:  74%|███████▍  | 104/140 [00:03<00:01, 33.02batch/s]

Training batches on cuda:  77%|███████▋  | 108/140 [00:03<00:00, 33.30batch/s]

Training batches on cuda:  80%|████████  | 112/140 [00:03<00:00, 33.44batch/s]

Training batches on cuda:  83%|████████▎ | 116/140 [00:03<00:00, 33.51batch/s]

Training batches on cuda:  86%|████████▌ | 120/140 [00:03<00:00, 33.59batch/s]

Training batches on cuda:  89%|████████▊ | 124/140 [00:03<00:00, 33.54batch/s]

Training batches on cuda:  91%|█████████▏| 128/140 [00:03<00:00, 34.04batch/s]

Training batches on cuda:  94%|█████████▍| 132/140 [00:04<00:00, 34.47batch/s]

Training batches on cuda:  97%|█████████▋| 136/140 [00:04<00:00, 34.81batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:27<00:22,  4.57s/epoch, loss=0.00013, prev_loss=0.000161] 
Training epochs on cuda:  60%|██████    | 6/10 [00:27<00:18,  4.54s/epoch, loss=0.00013, prev_loss=0.000161]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:08, 16.37batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 19.20batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 21.78batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:05, 23.78batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 25.89batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:04, 27.73batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 29.09batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 29.96batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 30.61batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 30.83batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:03, 31.43batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 32.10batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 32.37batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 32.45batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 32.40batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 32.88batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:02<00:02, 33.06batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 32.80batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:02, 32.61batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 32.64batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 32.33batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 32.72batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 33.02batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 33.14batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:03<00:01, 33.19batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 33.28batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 33.27batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 33.41batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 33.08batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 34.23batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 34.77batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 34.44batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:04<00:00, 34.19batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:04<00:00, 33.51batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:04<00:00, 33.44batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:31<00:18,  4.54s/epoch, loss=0.000116, prev_loss=0.00013]
Training epochs on cuda:  70%|███████   | 7/10 [00:31<00:13,  4.55s/epoch, loss=0.000116, prev_loss=0.00013]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:07, 18.57batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 21.42batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 23.97batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 26.23batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 28.13batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:04, 29.45batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 30.69batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 31.62batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 32.24batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 32.73batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 33.12batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 33.21batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 33.40batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 33.36batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 33.49batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 33.60batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:02<00:02, 33.70batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 33.48batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:01, 33.54batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 33.57batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 33.55batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 33.64batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 33.66batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 33.63batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:02<00:01, 33.84batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 33.90batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 33.80batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 33.80batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 33.92batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 34.08batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 34.06batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 33.91batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 33.67batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:04<00:00, 33.61batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:04<00:00, 33.56batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:36<00:13,  4.55s/epoch, loss=0.000113, prev_loss=0.000116]
Training epochs on cuda:  80%|████████  | 8/10 [00:36<00:09,  4.51s/epoch, loss=0.000113, prev_loss=0.000116]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:06, 19.99batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:05, 22.54batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 24.73batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 26.50batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 27.60batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:04, 29.10batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 29.35batch/s]

Training batches on cuda:  21%|██        | 29/140 [00:00<00:04, 27.64batch/s]

Training batches on cuda:  24%|██▎       | 33/140 [00:01<00:03, 29.05batch/s]

Training batches on cuda:  26%|██▋       | 37/140 [00:01<00:03, 30.21batch/s]

Training batches on cuda:  29%|██▉       | 41/140 [00:01<00:03, 30.81batch/s]

Training batches on cuda:  32%|███▏      | 45/140 [00:01<00:03, 31.49batch/s]

Training batches on cuda:  35%|███▌      | 49/140 [00:01<00:02, 31.81batch/s]

Training batches on cuda:  38%|███▊      | 53/140 [00:01<00:02, 31.37batch/s]

Training batches on cuda:  41%|████      | 57/140 [00:01<00:02, 31.91batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:01<00:02, 32.44batch/s]

Training batches on cuda:  46%|████▋     | 65/140 [00:02<00:02, 32.79batch/s]

Training batches on cuda:  49%|████▉     | 69/140 [00:02<00:02, 32.89batch/s]

Training batches on cuda:  52%|█████▏    | 73/140 [00:02<00:02, 32.04batch/s]

Training batches on cuda:  55%|█████▌    | 77/140 [00:02<00:01, 32.66batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:02<00:01, 33.03batch/s]

Training batches on cuda:  61%|██████    | 85/140 [00:02<00:01, 33.06batch/s]

Training batches on cuda:  64%|██████▎   | 89/140 [00:02<00:01, 33.03batch/s]

Training batches on cuda:  66%|██████▋   | 93/140 [00:02<00:01, 33.01batch/s]

Training batches on cuda:  69%|██████▉   | 97/140 [00:03<00:01, 33.22batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:03<00:01, 32.47batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:03<00:01, 32.79batch/s]

Training batches on cuda:  78%|███████▊  | 109/140 [00:03<00:00, 32.77batch/s]

Training batches on cuda:  81%|████████  | 113/140 [00:03<00:00, 33.03batch/s]

Training batches on cuda:  84%|████████▎ | 117/140 [00:03<00:00, 33.32batch/s]

Training batches on cuda:  86%|████████▋ | 121/140 [00:03<00:00, 33.25batch/s]

Training batches on cuda:  89%|████████▉ | 125/140 [00:03<00:00, 32.83batch/s]

Training batches on cuda:  92%|█████████▏| 129/140 [00:04<00:00, 33.18batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:04<00:00, 33.37batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:04<00:00, 33.40batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:40<00:09,  4.51s/epoch, loss=0.000113, prev_loss=0.000113]
Training epochs on cuda:  90%|█████████ | 9/10 [00:40<00:04,  4.55s/epoch, loss=0.000113, prev_loss=0.000113]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:06, 19.76batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:05, 22.39batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 24.80batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 26.77batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 28.26batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 29.71batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 30.35batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 31.25batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 31.99batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 32.40batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 32.80batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 32.92batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 32.83batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 32.81batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 32.81batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 32.67batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:02<00:02, 32.57batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 32.85batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:02, 32.74batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 32.67batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 32.95batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 32.72batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 33.02batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 33.28batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:03<00:01, 33.31batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 33.49batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 33.66batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 33.65batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 33.45batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 33.27batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 33.40batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 33.47batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 33.64batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:04<00:00, 33.50batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:04<00:00, 33.62batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:45<00:04,  4.55s/epoch, loss=0.000113, prev_loss=0.000113]2021-09-16:19:37:05,615 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:37:06,693 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:37:21,632 INFO     [evaluator.py:657] Evaluation took 14.94s seconds
2021-09-16:19:37:21,796 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  9.38s/epoch, loss=0.000113, prev_loss=0.000113]
Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  6.16s/epoch, loss=0.000113, prev_loss=0.000113]
2021-09-16:19:37:21,796 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmppdllbqqa'
2021-09-16:19:37:21,874 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmppdllbqqa' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.39ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.46ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.57ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.62ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.66ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.66ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.66ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:37:36,656 INFO     [evaluator.py:657] Evaluation took 14.76s seconds
[I 2021-09-16 19:37:36,661] Trial 93 finished with value: 0.9766430417521119 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.010658359780171782, 'optimizer.lr': 0.04052229067469055, 'negative_sampler.num_negs_per_pos': 25, 'training.batch_size': 2232}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:37:36,684 WARNING  [api.py:823] No random seed is specified. Setting to 718706600.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 26.78batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 28.85batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 30.73batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:03, 32.20batch/s]

Training batches on cuda:  14%|█▎        | 19/140 [00:00<00:03, 33.24batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:03, 33.51batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 33.72batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:03, 33.70batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:01<00:03, 33.70batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:01<00:02, 33.75batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:01<00:02, 33.73batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 33.85batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 33.84batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:01<00:02, 33.87batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:02, 33.86batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 33.97batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:02, 34.05batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:02<00:02, 34.12batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:02<00:01, 34.19batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:02<00:01, 34.09batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 32.76batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:02<00:01, 33.03batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 33.20batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:02<00:01, 33.42batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:02<00:01, 33.57batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:03<00:01, 33.69batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:03<00:00, 33.77batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:03<00:00, 33.89batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 33.79batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 33.88batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 34.01batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 33.99batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 33.78batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:03<00:00, 33.83batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:04<00:00, 33.93batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000486, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:39,  4.34s/epoch, loss=0.000486, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 23.73batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:05, 25.85batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 27.89batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:04, 29.48batch/s]

Training batches on cuda:  14%|█▎        | 19/140 [00:00<00:03, 30.39batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:03, 31.36batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 32.12batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:03, 32.53batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:01<00:03, 32.93batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:01<00:03, 33.29batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:01<00:02, 33.36batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 33.56batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 33.72batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:01<00:02, 33.78batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:02, 30.88batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 31.46batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:02<00:02, 32.11batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:02<00:02, 32.52batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:02<00:01, 32.84batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:02<00:01, 32.63batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 32.76batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:02<00:01, 33.03batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 33.04batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:02<00:01, 33.12batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:03<00:01, 33.13batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:03<00:01, 33.31batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:03<00:01, 32.38batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:03<00:00, 32.88batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 32.92batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 33.11batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 33.31batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 33.46batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 33.44batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:04<00:00, 33.11batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:04<00:00, 33.04batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:08<00:39,  4.34s/epoch, loss=0.000462, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:08<00:34,  4.37s/epoch, loss=0.000462, prev_loss=nan]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:07, 19.15batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 21.89batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 24.55batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 26.72batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 28.61batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 30.10batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 31.12batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 31.90batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 32.59batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 32.98batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 33.24batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 33.83batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 32.98batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 32.54batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 32.55batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 31.99batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:02<00:02, 32.25batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 32.71batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:02, 32.94batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:02, 30.19batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 31.20batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 31.89batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 32.35batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 32.82batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:03<00:01, 33.13batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 33.31batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 33.50batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 33.57batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 33.63batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 33.79batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 33.74batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 33.80batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 33.80batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:04<00:00, 33.90batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:04<00:00, 33.57batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:34,  4.37s/epoch, loss=0.000462, prev_loss=0.000462]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:30,  4.42s/epoch, loss=0.000462, prev_loss=0.000462]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 23.79batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:05, 26.24batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 28.36batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:04, 30.27batch/s]

Training batches on cuda:  14%|█▎        | 19/140 [00:00<00:03, 31.36batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:03, 32.60batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 33.22batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:03, 33.52batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:01<00:03, 33.43batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:01<00:03, 33.58batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:01<00:02, 33.83batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 33.72batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 33.83batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:01<00:02, 33.48batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:02, 33.68batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 33.72batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:02<00:02, 32.71batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:02<00:02, 32.99batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:02<00:01, 32.84batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:02<00:01, 33.02batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 33.13batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:02<00:01, 33.01batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 32.95batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:02<00:01, 30.39batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:03<00:01, 31.09batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:03<00:01, 31.72batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:03<00:01, 32.06batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:03<00:00, 32.44batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 32.89batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 32.72batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 32.61batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 33.04batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 33.06batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:04<00:00, 33.02batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:04<00:00, 32.95batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:17<00:30,  4.42s/epoch, loss=0.000462, prev_loss=0.000462]
Training epochs on cuda:  40%|████      | 4/10 [00:17<00:26,  4.44s/epoch, loss=0.000462, prev_loss=0.000462]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 23.84batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:05, 25.93batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 27.73batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:04, 29.13batch/s]

Training batches on cuda:  14%|█▎        | 19/140 [00:00<00:04, 30.21batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:03, 30.81batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 31.53batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:03, 32.03batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:01<00:03, 32.46batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:01<00:03, 31.86batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:01<00:03, 32.17batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 32.49batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 32.82batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:01<00:02, 33.19batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:02, 33.22batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 33.01batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:02<00:02, 32.72batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:02<00:02, 33.04batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:02<00:01, 33.28batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:02<00:01, 33.64batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 34.39batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:02<00:01, 34.33batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 32.92batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:02<00:01, 32.57batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:03<00:01, 32.21batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:03<00:01, 31.63batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:03<00:01, 31.48batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:03<00:01, 28.70batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 29.47batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 30.20batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 30.57batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 30.97batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:04<00:00, 31.02batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:04<00:00, 31.17batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:04<00:00, 31.27batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:26,  4.44s/epoch, loss=0.000462, prev_loss=0.000462]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:22,  4.51s/epoch, loss=0.000462, prev_loss=0.000462]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 23.97batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:05, 26.40batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 28.56batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:04, 30.40batch/s]

Training batches on cuda:  14%|█▎        | 19/140 [00:00<00:03, 31.82batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:03, 32.88batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 33.78batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:03, 34.24batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:01<00:03, 34.66batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:01<00:02, 34.62batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:01<00:02, 34.33batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 34.31batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 34.08batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:01<00:02, 34.03batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:02, 34.01batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 33.46batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:02, 34.25batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:02<00:01, 34.58batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:02<00:01, 33.85batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:02<00:01, 33.92batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 33.93batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:02<00:01, 33.62batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 33.82batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:02<00:01, 33.81batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:02<00:01, 33.86batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:03<00:01, 33.84batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:03<00:00, 33.98batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:03<00:00, 33.94batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 33.97batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 34.00batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 34.18batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 31.24batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 31.83batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:04<00:00, 32.36batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:04<00:00, 32.81batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:26<00:22,  4.51s/epoch, loss=0.000462, prev_loss=0.000462]
Training epochs on cuda:  60%|██████    | 6/10 [00:26<00:17,  4.46s/epoch, loss=0.000462, prev_loss=0.000462]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:07, 17.81batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 20.55batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 23.28batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 25.80batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 27.62batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:04, 29.25batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 30.40batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 31.26batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 31.98batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 32.53batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:03, 32.60batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 32.91batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 33.15batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 32.91batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 33.04batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 33.07batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:02<00:02, 32.58batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 32.65batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:02, 32.64batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 31.55batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 32.28batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 31.93batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 32.22batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 32.53batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:03<00:01, 32.87batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 33.21batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 33.49batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 33.60batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 33.43batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 33.49batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 33.51batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 33.40batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 33.43batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:04<00:00, 32.78batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:04<00:00, 33.59batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:31<00:17,  4.46s/epoch, loss=0.000462, prev_loss=0.000462]
Training epochs on cuda:  70%|███████   | 7/10 [00:31<00:13,  4.48s/epoch, loss=0.000462, prev_loss=0.000462]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/140 [00:00<00:05, 25.75batch/s]

Training batches on cuda:   5%|▌         | 7/140 [00:00<00:04, 27.94batch/s]

Training batches on cuda:   8%|▊         | 11/140 [00:00<00:04, 29.88batch/s]

Training batches on cuda:  11%|█         | 15/140 [00:00<00:03, 31.48batch/s]

Training batches on cuda:  14%|█▎        | 19/140 [00:00<00:03, 32.69batch/s]

Training batches on cuda:  16%|█▋        | 23/140 [00:00<00:03, 33.90batch/s]

Training batches on cuda:  19%|█▉        | 27/140 [00:00<00:03, 34.51batch/s]

Training batches on cuda:  22%|██▏       | 31/140 [00:00<00:03, 34.77batch/s]

Training batches on cuda:  25%|██▌       | 35/140 [00:01<00:02, 35.32batch/s]

Training batches on cuda:  28%|██▊       | 39/140 [00:01<00:02, 35.43batch/s]

Training batches on cuda:  31%|███       | 43/140 [00:01<00:02, 35.76batch/s]

Training batches on cuda:  34%|███▎      | 47/140 [00:01<00:02, 36.10batch/s]

Training batches on cuda:  36%|███▋      | 51/140 [00:01<00:02, 36.23batch/s]

Training batches on cuda:  39%|███▉      | 55/140 [00:01<00:02, 36.13batch/s]

Training batches on cuda:  42%|████▏     | 59/140 [00:01<00:02, 36.09batch/s]

Training batches on cuda:  45%|████▌     | 63/140 [00:01<00:02, 36.07batch/s]

Training batches on cuda:  48%|████▊     | 67/140 [00:01<00:02, 35.78batch/s]

Training batches on cuda:  51%|█████     | 71/140 [00:02<00:01, 35.75batch/s]

Training batches on cuda:  54%|█████▎    | 75/140 [00:02<00:01, 35.44batch/s]

Training batches on cuda:  56%|█████▋    | 79/140 [00:02<00:01, 34.84batch/s]

Training batches on cuda:  59%|█████▉    | 83/140 [00:02<00:01, 34.46batch/s]

Training batches on cuda:  62%|██████▏   | 87/140 [00:02<00:01, 34.18batch/s]

Training batches on cuda:  65%|██████▌   | 91/140 [00:02<00:01, 34.15batch/s]

Training batches on cuda:  68%|██████▊   | 95/140 [00:02<00:01, 34.08batch/s]

Training batches on cuda:  71%|███████   | 99/140 [00:02<00:01, 33.91batch/s]

Training batches on cuda:  74%|███████▎  | 103/140 [00:02<00:01, 33.85batch/s]

Training batches on cuda:  76%|███████▋  | 107/140 [00:03<00:00, 33.99batch/s]

Training batches on cuda:  79%|███████▉  | 111/140 [00:03<00:00, 33.98batch/s]

Training batches on cuda:  82%|████████▏ | 115/140 [00:03<00:00, 33.90batch/s]

Training batches on cuda:  85%|████████▌ | 119/140 [00:03<00:00, 33.75batch/s]

Training batches on cuda:  88%|████████▊ | 123/140 [00:03<00:00, 33.77batch/s]

Training batches on cuda:  91%|█████████ | 127/140 [00:03<00:00, 33.78batch/s]

Training batches on cuda:  94%|█████████▎| 131/140 [00:03<00:00, 33.81batch/s]

Training batches on cuda:  96%|█████████▋| 135/140 [00:03<00:00, 33.91batch/s]

Training batches on cuda:  99%|█████████▉| 139/140 [00:04<00:00, 33.89batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:13,  4.48s/epoch, loss=0.000462, prev_loss=0.000462]
Training epochs on cuda:  80%|████████  | 8/10 [00:35<00:08,  4.43s/epoch, loss=0.000462, prev_loss=0.000462]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:08, 17.00batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 19.96batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 22.63batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:05, 25.16batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 27.22batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:04, 28.81batch/s]

Training batches on cuda:  18%|█▊        | 25/140 [00:00<00:04, 26.72batch/s]

Training batches on cuda:  21%|██        | 29/140 [00:00<00:03, 29.09batch/s]

Training batches on cuda:  24%|██▎       | 33/140 [00:01<00:03, 30.59batch/s]

Training batches on cuda:  26%|██▋       | 37/140 [00:01<00:03, 31.58batch/s]

Training batches on cuda:  29%|██▉       | 41/140 [00:01<00:03, 32.13batch/s]

Training batches on cuda:  32%|███▏      | 45/140 [00:01<00:02, 32.66batch/s]

Training batches on cuda:  35%|███▌      | 49/140 [00:01<00:02, 33.05batch/s]

Training batches on cuda:  38%|███▊      | 53/140 [00:01<00:02, 33.31batch/s]

Training batches on cuda:  41%|████      | 57/140 [00:01<00:02, 33.56batch/s]

Training batches on cuda:  44%|████▎     | 61/140 [00:01<00:02, 33.65batch/s]

Training batches on cuda:  46%|████▋     | 65/140 [00:02<00:02, 33.50batch/s]

Training batches on cuda:  49%|████▉     | 69/140 [00:02<00:02, 33.66batch/s]

Training batches on cuda:  52%|█████▏    | 73/140 [00:02<00:01, 34.00batch/s]

Training batches on cuda:  55%|█████▌    | 77/140 [00:02<00:01, 33.50batch/s]

Training batches on cuda:  58%|█████▊    | 81/140 [00:02<00:01, 33.76batch/s]

Training batches on cuda:  61%|██████    | 85/140 [00:02<00:01, 33.87batch/s]

Training batches on cuda:  64%|██████▎   | 89/140 [00:02<00:01, 33.93batch/s]

Training batches on cuda:  66%|██████▋   | 93/140 [00:02<00:01, 33.85batch/s]

Training batches on cuda:  69%|██████▉   | 97/140 [00:02<00:01, 33.99batch/s]

Training batches on cuda:  72%|███████▏  | 101/140 [00:03<00:01, 33.81batch/s]

Training batches on cuda:  75%|███████▌  | 105/140 [00:03<00:01, 33.89batch/s]

Training batches on cuda:  78%|███████▊  | 109/140 [00:03<00:00, 33.92batch/s]

Training batches on cuda:  81%|████████  | 113/140 [00:03<00:00, 34.10batch/s]

Training batches on cuda:  84%|████████▎ | 117/140 [00:03<00:00, 34.05batch/s]

Training batches on cuda:  86%|████████▋ | 121/140 [00:03<00:00, 34.07batch/s]

Training batches on cuda:  89%|████████▉ | 125/140 [00:03<00:00, 33.87batch/s]

Training batches on cuda:  92%|█████████▏| 129/140 [00:03<00:00, 33.85batch/s]

Training batches on cuda:  95%|█████████▌| 133/140 [00:04<00:00, 33.77batch/s]

Training batches on cuda:  98%|█████████▊| 137/140 [00:04<00:00, 33.89batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:40<00:08,  4.43s/epoch, loss=0.000463, prev_loss=0.000462]
Training epochs on cuda:  90%|█████████ | 9/10 [00:40<00:04,  4.43s/epoch, loss=0.000463, prev_loss=0.000462]

Training batches on cuda:   0%|          | 0/140 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/140 [00:00<00:07, 18.71batch/s]

Training batches on cuda:   4%|▍         | 6/140 [00:00<00:06, 21.52batch/s]

Training batches on cuda:   7%|▋         | 10/140 [00:00<00:05, 23.98batch/s]

Training batches on cuda:  10%|█         | 14/140 [00:00<00:04, 26.30batch/s]

Training batches on cuda:  13%|█▎        | 18/140 [00:00<00:04, 28.06batch/s]

Training batches on cuda:  16%|█▌        | 22/140 [00:00<00:03, 29.71batch/s]

Training batches on cuda:  19%|█▊        | 26/140 [00:00<00:03, 30.90batch/s]

Training batches on cuda:  21%|██▏       | 30/140 [00:00<00:03, 31.82batch/s]

Training batches on cuda:  24%|██▍       | 34/140 [00:01<00:03, 32.68batch/s]

Training batches on cuda:  27%|██▋       | 38/140 [00:01<00:03, 33.14batch/s]

Training batches on cuda:  30%|███       | 42/140 [00:01<00:02, 33.34batch/s]

Training batches on cuda:  33%|███▎      | 46/140 [00:01<00:02, 33.65batch/s]

Training batches on cuda:  36%|███▌      | 50/140 [00:01<00:02, 33.75batch/s]

Training batches on cuda:  39%|███▊      | 54/140 [00:01<00:02, 33.92batch/s]

Training batches on cuda:  41%|████▏     | 58/140 [00:01<00:02, 34.15batch/s]

Training batches on cuda:  44%|████▍     | 62/140 [00:01<00:02, 33.52batch/s]

Training batches on cuda:  47%|████▋     | 66/140 [00:01<00:02, 33.75batch/s]

Training batches on cuda:  50%|█████     | 70/140 [00:02<00:02, 33.88batch/s]

Training batches on cuda:  53%|█████▎    | 74/140 [00:02<00:01, 33.98batch/s]

Training batches on cuda:  56%|█████▌    | 78/140 [00:02<00:01, 34.00batch/s]

Training batches on cuda:  59%|█████▊    | 82/140 [00:02<00:01, 34.05batch/s]

Training batches on cuda:  61%|██████▏   | 86/140 [00:02<00:01, 33.96batch/s]

Training batches on cuda:  64%|██████▍   | 90/140 [00:02<00:01, 34.03batch/s]

Training batches on cuda:  67%|██████▋   | 94/140 [00:02<00:01, 33.98batch/s]

Training batches on cuda:  70%|███████   | 98/140 [00:02<00:01, 34.00batch/s]

Training batches on cuda:  73%|███████▎  | 102/140 [00:03<00:01, 33.72batch/s]

Training batches on cuda:  76%|███████▌  | 106/140 [00:03<00:01, 33.77batch/s]

Training batches on cuda:  79%|███████▊  | 110/140 [00:03<00:00, 33.91batch/s]

Training batches on cuda:  81%|████████▏ | 114/140 [00:03<00:00, 33.89batch/s]

Training batches on cuda:  84%|████████▍ | 118/140 [00:03<00:00, 33.89batch/s]

Training batches on cuda:  87%|████████▋ | 122/140 [00:03<00:00, 33.98batch/s]

Training batches on cuda:  90%|█████████ | 126/140 [00:03<00:00, 33.76batch/s]

Training batches on cuda:  93%|█████████▎| 130/140 [00:03<00:00, 33.50batch/s]

Training batches on cuda:  96%|█████████▌| 134/140 [00:04<00:00, 33.59batch/s]

Training batches on cuda:  99%|█████████▊| 138/140 [00:04<00:00, 33.49batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:44<00:04,  4.43s/epoch, loss=0.000462, prev_loss=0.000463]2021-09-16:19:38:21,748 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:38:22,769 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:38:37,685 INFO     [evaluator.py:657] Evaluation took 14.92s seconds
2021-09-16:19:38:37,821 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00,  9.27s/epoch, loss=0.000462, prev_loss=0.000463]
Training epochs on cuda: 100%|██████████| 10/10 [01:00<00:00,  6.07s/epoch, loss=0.000462, prev_loss=0.000463]
2021-09-16:19:38:37,821 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp1d9pn36v'
2021-09-16:19:38:37,897 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp1d9pn36v' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.52ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.56ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.59ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.62ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:05<00:08, 2.65ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.62ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.63ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:38:52,698 INFO     [evaluator.py:657] Evaluation took 14.77s seconds
[I 2021-09-16 19:38:52,703] Trial 94 finished with value: 0.005896781684476937 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.6181547374310519, 'optimizer.lr': 0.04145928226257, 'negative_sampler.num_negs_per_pos': 25, 'training.batch_size': 2232}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:38:52,725 WARNING  [api.py:823] No random seed is specified. Setting to 755083005.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:04, 27.86batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:04, 30.50batch/s]

Training batches on cuda:   9%|▉         | 12/134 [00:00<00:03, 33.25batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:00<00:03, 35.62batch/s]

Training batches on cuda:  16%|█▋        | 22/134 [00:00<00:03, 37.22batch/s]

Training batches on cuda:  19%|█▉        | 26/134 [00:00<00:02, 37.93batch/s]

Training batches on cuda:  22%|██▏       | 30/134 [00:00<00:02, 38.41batch/s]

Training batches on cuda:  25%|██▌       | 34/134 [00:00<00:02, 38.54batch/s]

Training batches on cuda:  28%|██▊       | 38/134 [00:00<00:02, 38.36batch/s]

Training batches on cuda:  31%|███▏      | 42/134 [00:01<00:02, 37.95batch/s]

Training batches on cuda:  34%|███▍      | 46/134 [00:01<00:02, 37.58batch/s]

Training batches on cuda:  37%|███▋      | 50/134 [00:01<00:02, 37.62batch/s]

Training batches on cuda:  40%|████      | 54/134 [00:01<00:02, 37.50batch/s]

Training batches on cuda:  43%|████▎     | 58/134 [00:01<00:02, 37.32batch/s]

Training batches on cuda:  46%|████▋     | 62/134 [00:01<00:01, 37.75batch/s]

Training batches on cuda:  49%|████▉     | 66/134 [00:01<00:01, 37.76batch/s]

Training batches on cuda:  52%|█████▏    | 70/134 [00:01<00:01, 37.85batch/s]

Training batches on cuda:  55%|█████▌    | 74/134 [00:01<00:01, 38.06batch/s]

Training batches on cuda:  58%|█████▊    | 78/134 [00:02<00:01, 37.98batch/s]

Training batches on cuda:  61%|██████    | 82/134 [00:02<00:01, 38.12batch/s]

Training batches on cuda:  64%|██████▍   | 86/134 [00:02<00:01, 38.27batch/s]

Training batches on cuda:  67%|██████▋   | 90/134 [00:02<00:01, 38.10batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:02<00:01, 38.18batch/s]

Training batches on cuda:  73%|███████▎  | 98/134 [00:02<00:00, 38.11batch/s]

Training batches on cuda:  76%|███████▌  | 102/134 [00:02<00:00, 38.14batch/s]

Training batches on cuda:  79%|███████▉  | 106/134 [00:02<00:00, 38.14batch/s]

Training batches on cuda:  82%|████████▏ | 110/134 [00:02<00:00, 38.20batch/s]

Training batches on cuda:  85%|████████▌ | 114/134 [00:02<00:00, 38.02batch/s]

Training batches on cuda:  88%|████████▊ | 118/134 [00:03<00:00, 37.52batch/s]

Training batches on cuda:  91%|█████████ | 122/134 [00:03<00:00, 37.90batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:03<00:00, 37.95batch/s]

Training batches on cuda:  97%|█████████▋| 130/134 [00:03<00:00, 38.07batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000434, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:33,  3.72s/epoch, loss=0.000434, prev_loss=nan]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:05, 24.77batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:04, 27.26batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:00<00:04, 29.69batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:00<00:03, 31.86batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:00<00:03, 33.53batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:00<00:03, 34.77batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:00<00:02, 35.78batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:00<00:02, 36.44batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:00<00:02, 36.79batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:01<00:02, 37.11batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:01<00:02, 35.99batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:01<00:02, 36.46batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:01<00:02, 36.74batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:01<00:02, 37.43batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:01<00:01, 37.67batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:01<00:01, 37.68batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:01<00:01, 37.90batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:01<00:01, 37.84batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:02<00:01, 37.91batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:02<00:01, 37.92batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:02<00:01, 35.00batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:02<00:01, 35.76batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:02<00:01, 36.35batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:02<00:01, 36.68batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:02<00:00, 37.11batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:02<00:00, 37.36batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:02<00:00, 37.15batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:03<00:00, 37.37batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:03<00:00, 37.22batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:03<00:00, 37.33batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:03<00:00, 37.13batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:03<00:00, 37.29batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:03<00:00, 36.89batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:07<00:33,  3.72s/epoch, loss=0.000432, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:30,  3.76s/epoch, loss=0.000432, prev_loss=nan]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:05, 25.92batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:04, 28.49batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:00<00:03, 30.81batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:00<00:03, 32.59batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:00<00:03, 34.11batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:00<00:03, 35.25batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:00<00:02, 35.96batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:00<00:02, 36.29batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:00<00:02, 36.45batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:01<00:02, 36.44batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:01<00:02, 36.61batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:01<00:02, 37.16batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:01<00:02, 37.49batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:01<00:02, 36.68batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:01<00:02, 37.04batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:01<00:01, 37.20batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:01<00:01, 37.32batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:01<00:01, 36.72batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:02<00:01, 37.08batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:02<00:01, 37.02batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:02<00:01, 36.99batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:02<00:01, 37.34batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:02<00:01, 36.94batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:02<00:01, 36.71batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:02<00:00, 36.53batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:02<00:00, 36.81batch/s]

Training batches on cuda:  80%|███████▉  | 107/134 [00:02<00:00, 36.50batch/s]

Training batches on cuda:  83%|████████▎ | 111/134 [00:03<00:00, 36.95batch/s]

Training batches on cuda:  86%|████████▌ | 115/134 [00:03<00:00, 37.04batch/s]

Training batches on cuda:  89%|████████▉ | 119/134 [00:03<00:00, 37.37batch/s]

Training batches on cuda:  92%|█████████▏| 123/134 [00:03<00:00, 37.19batch/s]

Training batches on cuda:  95%|█████████▍| 127/134 [00:03<00:00, 35.27batch/s]

Training batches on cuda:  98%|█████████▊| 131/134 [00:03<00:00, 35.85batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:11<00:30,  3.76s/epoch, loss=0.000431, prev_loss=0.000432]
Training epochs on cuda:  30%|███       | 3/10 [00:11<00:26,  3.79s/epoch, loss=0.000431, prev_loss=0.000432]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/134 [00:00<00:07, 17.33batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:06, 20.34batch/s]

Training batches on cuda:   7%|▋         | 10/134 [00:00<00:05, 23.42batch/s]

Training batches on cuda:  10%|█         | 14/134 [00:00<00:04, 26.19batch/s]

Training batches on cuda:  13%|█▎        | 18/134 [00:00<00:04, 27.50batch/s]

Training batches on cuda:  16%|█▋        | 22/134 [00:00<00:03, 29.47batch/s]

Training batches on cuda:  19%|█▉        | 26/134 [00:00<00:03, 31.10batch/s]

Training batches on cuda:  22%|██▏       | 30/134 [00:00<00:03, 32.61batch/s]

Training batches on cuda:  25%|██▌       | 34/134 [00:01<00:03, 32.73batch/s]

Training batches on cuda:  28%|██▊       | 38/134 [00:01<00:02, 33.35batch/s]

Training batches on cuda:  31%|███▏      | 42/134 [00:01<00:02, 34.18batch/s]

Training batches on cuda:  34%|███▍      | 46/134 [00:01<00:02, 34.40batch/s]

Training batches on cuda:  37%|███▋      | 50/134 [00:01<00:02, 34.94batch/s]

Training batches on cuda:  40%|████      | 54/134 [00:01<00:02, 35.30batch/s]

Training batches on cuda:  43%|████▎     | 58/134 [00:01<00:02, 34.76batch/s]

Training batches on cuda:  46%|████▋     | 62/134 [00:01<00:02, 35.40batch/s]

Training batches on cuda:  49%|████▉     | 66/134 [00:01<00:01, 36.10batch/s]

Training batches on cuda:  52%|█████▏    | 70/134 [00:02<00:01, 36.45batch/s]

Training batches on cuda:  55%|█████▌    | 74/134 [00:02<00:01, 36.78batch/s]

Training batches on cuda:  58%|█████▊    | 78/134 [00:02<00:01, 37.22batch/s]

Training batches on cuda:  61%|██████    | 82/134 [00:02<00:01, 37.16batch/s]

Training batches on cuda:  64%|██████▍   | 86/134 [00:02<00:01, 37.02batch/s]

Training batches on cuda:  67%|██████▋   | 90/134 [00:02<00:01, 37.35batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:02<00:01, 37.45batch/s]

Training batches on cuda:  73%|███████▎  | 98/134 [00:02<00:00, 37.75batch/s]

Training batches on cuda:  76%|███████▌  | 102/134 [00:02<00:00, 36.92batch/s]

Training batches on cuda:  79%|███████▉  | 106/134 [00:03<00:00, 36.95batch/s]

Training batches on cuda:  82%|████████▏ | 110/134 [00:03<00:00, 37.18batch/s]

Training batches on cuda:  85%|████████▌ | 114/134 [00:03<00:00, 37.37batch/s]

Training batches on cuda:  88%|████████▊ | 118/134 [00:03<00:00, 37.46batch/s]

Training batches on cuda:  91%|█████████ | 122/134 [00:03<00:00, 37.53batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:03<00:00, 37.61batch/s]

Training batches on cuda:  97%|█████████▋| 130/134 [00:03<00:00, 36.93batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:26,  3.79s/epoch, loss=0.000427, prev_loss=0.000431]
Training epochs on cuda:  40%|████      | 4/10 [00:15<00:23,  3.86s/epoch, loss=0.000427, prev_loss=0.000431]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:15,  8.43batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:11, 10.93batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:09, 13.82batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:00<00:07, 17.01batch/s]

Training batches on cuda:  12%|█▏        | 16/134 [00:00<00:06, 18.43batch/s]

Training batches on cuda:  15%|█▍        | 20/134 [00:00<00:05, 21.22batch/s]

Training batches on cuda:  18%|█▊        | 24/134 [00:00<00:04, 24.53batch/s]

Training batches on cuda:  21%|██        | 28/134 [00:00<00:03, 27.50batch/s]

Training batches on cuda:  24%|██▍       | 32/134 [00:01<00:03, 29.91batch/s]

Training batches on cuda:  27%|██▋       | 36/134 [00:01<00:03, 31.86batch/s]

Training batches on cuda:  30%|██▉       | 40/134 [00:01<00:02, 33.29batch/s]

Training batches on cuda:  33%|███▎      | 44/134 [00:01<00:02, 34.74batch/s]

Training batches on cuda:  36%|███▌      | 48/134 [00:01<00:02, 35.61batch/s]

Training batches on cuda:  39%|███▉      | 52/134 [00:01<00:02, 36.12batch/s]

Training batches on cuda:  42%|████▏     | 56/134 [00:01<00:02, 36.88batch/s]

Training batches on cuda:  45%|████▍     | 60/134 [00:01<00:01, 37.08batch/s]

Training batches on cuda:  48%|████▊     | 64/134 [00:01<00:01, 35.93batch/s]

Training batches on cuda:  51%|█████     | 68/134 [00:01<00:01, 36.02batch/s]

Training batches on cuda:  54%|█████▎    | 72/134 [00:02<00:01, 36.17batch/s]

Training batches on cuda:  57%|█████▋    | 76/134 [00:02<00:01, 36.79batch/s]

Training batches on cuda:  60%|█████▉    | 80/134 [00:02<00:01, 36.65batch/s]

Training batches on cuda:  63%|██████▎   | 84/134 [00:02<00:01, 37.15batch/s]

Training batches on cuda:  66%|██████▌   | 88/134 [00:02<00:01, 37.36batch/s]

Training batches on cuda:  69%|██████▊   | 92/134 [00:02<00:01, 37.57batch/s]

Training batches on cuda:  72%|███████▏  | 96/134 [00:02<00:01, 37.62batch/s]

Training batches on cuda:  75%|███████▍  | 100/134 [00:02<00:00, 37.74batch/s]

Training batches on cuda:  78%|███████▊  | 104/134 [00:02<00:00, 37.85batch/s]

Training batches on cuda:  81%|████████  | 108/134 [00:03<00:00, 37.84batch/s]

Training batches on cuda:  84%|████████▎ | 112/134 [00:03<00:00, 37.75batch/s]

Training batches on cuda:  87%|████████▋ | 116/134 [00:03<00:00, 37.97batch/s]

Training batches on cuda:  90%|████████▉ | 120/134 [00:03<00:00, 38.03batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:03<00:00, 38.80batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:03<00:00, 39.00batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:03<00:00, 38.21batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:19<00:23,  3.86s/epoch, loss=0.000415, prev_loss=0.000427]
Training epochs on cuda:  50%|█████     | 5/10 [00:19<00:19,  3.88s/epoch, loss=0.000415, prev_loss=0.000427]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:14,  9.01batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:11, 11.62batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:08, 14.63batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:00<00:06, 17.89batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:00<00:05, 21.22batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:00<00:04, 24.25batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:00<00:04, 27.17batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:00<00:03, 29.66batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:00<00:03, 31.53batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:01<00:02, 33.13batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:01<00:02, 34.25batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:01<00:02, 35.24batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:01<00:02, 36.05batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:01<00:02, 35.65batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:01<00:02, 35.55batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:01<00:02, 35.58batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:01<00:01, 35.19batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:01<00:01, 35.06batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:02<00:01, 34.74batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:02<00:01, 34.80batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:02<00:01, 34.87batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:02<00:01, 34.87batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:02<00:01, 34.90batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:02<00:01, 34.88batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:02<00:01, 35.55batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:02<00:00, 36.04batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:02<00:00, 36.46batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:03<00:00, 36.92batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:03<00:00, 36.34batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:03<00:00, 36.62batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:03<00:00, 37.03batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:03<00:00, 36.52batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:03<00:00, 36.69batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:03<00:00, 36.91batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:23<00:19,  3.88s/epoch, loss=0.000407, prev_loss=0.000415]
Training epochs on cuda:  60%|██████    | 6/10 [00:23<00:15,  3.92s/epoch, loss=0.000407, prev_loss=0.000415]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/134 [00:00<00:15,  8.83batch/s]

Training batches on cuda:   4%|▎         | 5/134 [00:00<00:11, 11.40batch/s]

Training batches on cuda:   7%|▋         | 9/134 [00:00<00:08, 14.40batch/s]

Training batches on cuda:  10%|▉         | 13/134 [00:00<00:06, 17.71batch/s]

Training batches on cuda:  13%|█▎        | 17/134 [00:00<00:05, 21.08batch/s]

Training batches on cuda:  16%|█▌        | 21/134 [00:00<00:04, 24.32batch/s]

Training batches on cuda:  19%|█▊        | 25/134 [00:00<00:03, 27.27batch/s]

Training batches on cuda:  22%|██▏       | 29/134 [00:00<00:03, 29.76batch/s]

Training batches on cuda:  25%|██▍       | 33/134 [00:00<00:03, 31.91batch/s]

Training batches on cuda:  28%|██▊       | 37/134 [00:01<00:02, 33.43batch/s]

Training batches on cuda:  31%|███       | 41/134 [00:01<00:02, 34.77batch/s]

Training batches on cuda:  34%|███▎      | 45/134 [00:01<00:02, 35.83batch/s]

Training batches on cuda:  37%|███▋      | 49/134 [00:01<00:02, 36.12batch/s]

Training batches on cuda:  40%|███▉      | 53/134 [00:01<00:02, 36.83batch/s]

Training batches on cuda:  43%|████▎     | 57/134 [00:01<00:02, 37.31batch/s]

Training batches on cuda:  46%|████▌     | 61/134 [00:01<00:01, 37.62batch/s]

Training batches on cuda:  49%|████▊     | 65/134 [00:01<00:01, 37.73batch/s]

Training batches on cuda:  51%|█████▏    | 69/134 [00:01<00:01, 37.87batch/s]

Training batches on cuda:  54%|█████▍    | 73/134 [00:02<00:01, 37.83batch/s]

Training batches on cuda:  57%|█████▋    | 77/134 [00:02<00:01, 38.05batch/s]

Training batches on cuda:  60%|██████    | 81/134 [00:02<00:01, 38.16batch/s]

Training batches on cuda:  63%|██████▎   | 85/134 [00:02<00:01, 38.20batch/s]

Training batches on cuda:  66%|██████▋   | 89/134 [00:02<00:01, 38.06batch/s]

Training batches on cuda:  69%|██████▉   | 93/134 [00:02<00:01, 38.15batch/s]

Training batches on cuda:  72%|███████▏  | 97/134 [00:02<00:00, 38.21batch/s]

Training batches on cuda:  75%|███████▌  | 101/134 [00:02<00:00, 38.25batch/s]

Training batches on cuda:  78%|███████▊  | 105/134 [00:02<00:00, 38.28batch/s]

Training batches on cuda:  81%|████████▏ | 109/134 [00:02<00:00, 38.19batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:03<00:00, 38.30batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:03<00:00, 37.93batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:03<00:00, 37.88batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:03<00:00, 37.91batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:03<00:00, 37.95batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:03<00:00, 37.93batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:27<00:15,  3.92s/epoch, loss=0.000396, prev_loss=0.000407]
Training epochs on cuda:  70%|███████   | 7/10 [00:27<00:11,  3.90s/epoch, loss=0.000396, prev_loss=0.000407]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 3/134 [00:00<00:04, 26.26batch/s]

Training batches on cuda:   5%|▌         | 7/134 [00:00<00:04, 28.74batch/s]

Training batches on cuda:   8%|▊         | 11/134 [00:00<00:03, 30.87batch/s]

Training batches on cuda:  11%|█         | 15/134 [00:00<00:03, 32.72batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:00<00:03, 34.21batch/s]

Training batches on cuda:  17%|█▋        | 23/134 [00:00<00:03, 35.03batch/s]

Training batches on cuda:  20%|██        | 27/134 [00:00<00:02, 35.95batch/s]

Training batches on cuda:  23%|██▎       | 31/134 [00:00<00:02, 36.11batch/s]

Training batches on cuda:  26%|██▌       | 35/134 [00:00<00:02, 36.32batch/s]

Training batches on cuda:  29%|██▉       | 39/134 [00:01<00:02, 36.55batch/s]

Training batches on cuda:  32%|███▏      | 43/134 [00:01<00:02, 36.90batch/s]

Training batches on cuda:  35%|███▌      | 47/134 [00:01<00:02, 37.17batch/s]

Training batches on cuda:  38%|███▊      | 51/134 [00:01<00:02, 37.50batch/s]

Training batches on cuda:  41%|████      | 55/134 [00:01<00:02, 37.10batch/s]

Training batches on cuda:  44%|████▍     | 59/134 [00:01<00:02, 37.38batch/s]

Training batches on cuda:  47%|████▋     | 63/134 [00:01<00:01, 37.50batch/s]

Training batches on cuda:  50%|█████     | 67/134 [00:01<00:01, 36.95batch/s]

Training batches on cuda:  53%|█████▎    | 71/134 [00:01<00:01, 36.64batch/s]

Training batches on cuda:  56%|█████▌    | 75/134 [00:02<00:01, 36.99batch/s]

Training batches on cuda:  59%|█████▉    | 79/134 [00:02<00:01, 36.86batch/s]

Training batches on cuda:  62%|██████▏   | 83/134 [00:02<00:01, 37.19batch/s]

Training batches on cuda:  65%|██████▍   | 87/134 [00:02<00:01, 37.39batch/s]

Training batches on cuda:  68%|██████▊   | 91/134 [00:02<00:01, 37.55batch/s]

Training batches on cuda:  71%|███████   | 95/134 [00:02<00:01, 37.97batch/s]

Training batches on cuda:  74%|███████▍  | 99/134 [00:02<00:00, 38.39batch/s]

Training batches on cuda:  77%|███████▋  | 103/134 [00:02<00:00, 38.69batch/s]

Training batches on cuda:  81%|████████  | 108/134 [00:02<00:00, 39.12batch/s]

Training batches on cuda:  84%|████████▍ | 113/134 [00:03<00:00, 39.65batch/s]

Training batches on cuda:  87%|████████▋ | 117/134 [00:03<00:00, 38.60batch/s]

Training batches on cuda:  90%|█████████ | 121/134 [00:03<00:00, 38.16batch/s]

Training batches on cuda:  93%|█████████▎| 125/134 [00:03<00:00, 37.84batch/s]

Training batches on cuda:  96%|█████████▋| 129/134 [00:03<00:00, 37.10batch/s]

Training batches on cuda:  99%|█████████▉| 133/134 [00:03<00:00, 37.23batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:31<00:11,  3.90s/epoch, loss=0.000313, prev_loss=0.000396]
Training epochs on cuda:  80%|████████  | 8/10 [00:31<00:07,  3.88s/epoch, loss=0.000313, prev_loss=0.000396]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/134 [00:00<00:08, 16.08batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:06, 19.26batch/s]

Training batches on cuda:   7%|▋         | 10/134 [00:00<00:05, 22.54batch/s]

Training batches on cuda:  10%|█         | 14/134 [00:00<00:04, 25.93batch/s]

Training batches on cuda:  14%|█▍        | 19/134 [00:00<00:03, 29.06batch/s]

Training batches on cuda:  18%|█▊        | 24/134 [00:00<00:03, 31.92batch/s]

Training batches on cuda:  21%|██        | 28/134 [00:00<00:03, 33.93batch/s]

Training batches on cuda:  24%|██▍       | 32/134 [00:00<00:02, 35.05batch/s]

Training batches on cuda:  27%|██▋       | 36/134 [00:00<00:02, 35.82batch/s]

Training batches on cuda:  30%|██▉       | 40/134 [00:01<00:02, 36.26batch/s]

Training batches on cuda:  33%|███▎      | 44/134 [00:01<00:02, 36.84batch/s]

Training batches on cuda:  36%|███▌      | 48/134 [00:01<00:02, 37.24batch/s]

Training batches on cuda:  39%|███▉      | 52/134 [00:01<00:02, 37.48batch/s]

Training batches on cuda:  42%|████▏     | 56/134 [00:01<00:02, 37.26batch/s]

Training batches on cuda:  45%|████▍     | 60/134 [00:01<00:01, 37.42batch/s]

Training batches on cuda:  48%|████▊     | 64/134 [00:01<00:01, 37.78batch/s]

Training batches on cuda:  51%|█████     | 68/134 [00:01<00:01, 37.81batch/s]

Training batches on cuda:  54%|█████▎    | 72/134 [00:01<00:01, 37.79batch/s]

Training batches on cuda:  57%|█████▋    | 76/134 [00:02<00:01, 37.92batch/s]

Training batches on cuda:  60%|█████▉    | 80/134 [00:02<00:01, 36.67batch/s]

Training batches on cuda:  63%|██████▎   | 84/134 [00:02<00:01, 37.12batch/s]

Training batches on cuda:  66%|██████▌   | 88/134 [00:02<00:01, 37.33batch/s]

Training batches on cuda:  69%|██████▊   | 92/134 [00:02<00:01, 37.65batch/s]

Training batches on cuda:  72%|███████▏  | 96/134 [00:02<00:01, 37.67batch/s]

Training batches on cuda:  75%|███████▍  | 100/134 [00:02<00:00, 37.77batch/s]

Training batches on cuda:  78%|███████▊  | 104/134 [00:02<00:00, 38.14batch/s]

Training batches on cuda:  81%|████████  | 108/134 [00:02<00:00, 38.03batch/s]

Training batches on cuda:  84%|████████▎ | 112/134 [00:03<00:00, 38.02batch/s]

Training batches on cuda:  87%|████████▋ | 116/134 [00:03<00:00, 37.86batch/s]

Training batches on cuda:  90%|████████▉ | 120/134 [00:03<00:00, 37.42batch/s]

Training batches on cuda:  93%|█████████▎| 124/134 [00:03<00:00, 37.47batch/s]

Training batches on cuda:  96%|█████████▌| 128/134 [00:03<00:00, 37.72batch/s]

Training batches on cuda:  99%|█████████▊| 132/134 [00:03<00:00, 37.56batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:34<00:07,  3.88s/epoch, loss=0.000253, prev_loss=0.000313]
Training epochs on cuda:  90%|█████████ | 9/10 [00:34<00:03,  3.88s/epoch, loss=0.000253, prev_loss=0.000313]

Training batches on cuda:   0%|          | 0/134 [00:00<?, ?batch/s]

Training batches on cuda:   1%|▏         | 2/134 [00:00<00:06, 19.17batch/s]

Training batches on cuda:   4%|▍         | 6/134 [00:00<00:05, 22.16batch/s]

Training batches on cuda:   7%|▋         | 10/134 [00:00<00:04, 24.95batch/s]

Training batches on cuda:  10%|█         | 14/134 [00:00<00:04, 27.30batch/s]

Training batches on cuda:  13%|█▎        | 18/134 [00:00<00:03, 29.17batch/s]

Training batches on cuda:  16%|█▋        | 22/134 [00:00<00:03, 30.88batch/s]

Training batches on cuda:  19%|█▉        | 26/134 [00:00<00:03, 32.23batch/s]

Training batches on cuda:  22%|██▏       | 30/134 [00:00<00:03, 33.52batch/s]

Training batches on cuda:  25%|██▌       | 34/134 [00:01<00:02, 34.41batch/s]

Training batches on cuda:  28%|██▊       | 38/134 [00:01<00:02, 33.22batch/s]

Training batches on cuda:  31%|███▏      | 42/134 [00:01<00:02, 30.91batch/s]

Training batches on cuda:  34%|███▍      | 46/134 [00:01<00:02, 31.99batch/s]

Training batches on cuda:  37%|███▋      | 50/134 [00:01<00:02, 32.90batch/s]

Training batches on cuda:  40%|████      | 54/134 [00:01<00:02, 33.40batch/s]

Training batches on cuda:  43%|████▎     | 58/134 [00:01<00:02, 33.84batch/s]

Training batches on cuda:  46%|████▋     | 62/134 [00:01<00:02, 33.70batch/s]

Training batches on cuda:  49%|████▉     | 66/134 [00:01<00:01, 34.90batch/s]

Training batches on cuda:  52%|█████▏    | 70/134 [00:02<00:01, 35.63batch/s]

Training batches on cuda:  55%|█████▌    | 74/134 [00:02<00:01, 36.17batch/s]

Training batches on cuda:  58%|█████▊    | 78/134 [00:02<00:01, 36.76batch/s]

Training batches on cuda:  61%|██████    | 82/134 [00:02<00:01, 37.15batch/s]

Training batches on cuda:  64%|██████▍   | 86/134 [00:02<00:01, 37.49batch/s]

Training batches on cuda:  67%|██████▋   | 90/134 [00:02<00:01, 37.63batch/s]

Training batches on cuda:  70%|███████   | 94/134 [00:02<00:01, 37.68batch/s]

Training batches on cuda:  73%|███████▎  | 98/134 [00:02<00:00, 38.02batch/s]

Training batches on cuda:  76%|███████▌  | 102/134 [00:02<00:00, 38.09batch/s]

Training batches on cuda:  79%|███████▉  | 106/134 [00:03<00:00, 37.56batch/s]

Training batches on cuda:  82%|████████▏ | 110/134 [00:03<00:00, 37.86batch/s]

Training batches on cuda:  85%|████████▌ | 114/134 [00:03<00:00, 37.99batch/s]

Training batches on cuda:  88%|████████▊ | 118/134 [00:03<00:00, 37.72batch/s]

Training batches on cuda:  91%|█████████ | 122/134 [00:03<00:00, 37.62batch/s]

Training batches on cuda:  94%|█████████▍| 126/134 [00:03<00:00, 37.61batch/s]

Training batches on cuda:  97%|█████████▋| 130/134 [00:03<00:00, 37.06batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:38<00:03,  3.88s/epoch, loss=0.000232, prev_loss=0.000253]2021-09-16:19:39:32,177 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:39:33,250 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:39:48,142 INFO     [evaluator.py:657] Evaluation took 14.89s seconds
2021-09-16:19:39:48,272 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00,  8.74s/epoch, loss=0.000232, prev_loss=0.000253]
Training epochs on cuda: 100%|██████████| 10/10 [00:55<00:00,  5.50s/epoch, loss=0.000232, prev_loss=0.000253]
2021-09-16:19:39:48,273 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpz1v182gv'
2021-09-16:19:39:48,343 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpz1v182gv' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:16, 2.37ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.62ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  12%|█▏        | 4.61k/38.8k [00:01<00:13, 2.59ktriple/s]
Evaluating on cuda:  13%|█▎        | 5.12k/38.8k [00:01<00:12, 2.61ktriple/s]
Evaluating on cuda:  15%|█▍        | 5.63k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  16%|█▌        | 6.14k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  17%|█▋        | 6.66k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  18%|█▊        | 7.17k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|█▉        | 7.68k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  21%|██        | 8.19k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.70k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▎       | 9.22k/38.8k [00:03<00:11, 2.66ktriple/s]
Evaluating on cuda:  25%|██▌       | 9.73k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  26%|██▋       | 10.2k/38.8k [00:03<00:10, 2.66ktriple/s]
Evaluating on cuda:  28%|██▊       | 10.8k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  29%|██▉       | 11.3k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  30%|███       | 11.8k/38.8k [00:04<00:10, 2.66ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.3k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  33%|███▎      | 12.8k/38.8k [00:04<00:09, 2.66ktriple/s]
Evaluating on cuda:  34%|███▍      | 13.3k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  36%|███▌      | 13.8k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  37%|███▋      | 14.3k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.8k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  40%|███▉      | 15.4k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  41%|████      | 15.9k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.4k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  44%|████▎     | 16.9k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  45%|████▍     | 17.4k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  46%|████▌     | 17.9k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.4k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 18.9k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  50%|█████     | 19.5k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████▏    | 20.0k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.5k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  54%|█████▍    | 21.0k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▌    | 21.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.0k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  58%|█████▊    | 22.5k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▉    | 23.0k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████    | 23.6k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  62%|██████▏   | 24.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▍   | 25.1k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  66%|██████▌   | 25.6k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 26.1k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▊   | 26.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  70%|██████▉   | 27.1k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.6k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.2k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.7k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  75%|███████▌  | 29.2k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▋  | 29.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.2k/38.8k [00:11<00:03, 2.64ktriple/s]
Evaluating on cuda:  79%|███████▉  | 30.7k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|████████  | 31.2k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 31.7k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  83%|████████▎ | 32.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▍ | 32.8k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▌ | 33.3k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  87%|████████▋ | 33.8k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.3k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|████████▉ | 34.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  91%|█████████ | 35.3k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.8k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▎| 36.4k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  95%|█████████▍| 36.9k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▋| 37.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  98%|█████████▊| 37.9k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  99%|█████████▉| 38.4k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:40:03,117 INFO     [evaluator.py:657] Evaluation took 14.75s seconds
[I 2021-09-16 19:40:03,122] Trial 95 finished with value: 0.9127969396643392 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.04340795027034219, 'optimizer.lr': 0.03738207513753791, 'negative_sampler.num_negs_per_pos': 18, 'training.batch_size': 2332}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:40:03,144 WARNING  [api.py:823] No random seed is specified. Setting to 1093664823.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:07, 15.57batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:06, 17.54batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:05, 19.30batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:05, 20.94batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:04, 22.30batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 23.32batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 24.01batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:04, 24.60batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 24.97batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 25.00batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 25.01batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 23.12batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 23.89batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 24.55batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 25.06batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:02, 25.36batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:02<00:02, 25.59batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 25.51batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 25.70batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 25.86batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 25.88batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 25.98batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 25.95batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:02, 25.87batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:01, 25.93batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:03<00:01, 25.95batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 25.64batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 25.61batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 25.74batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 25.82batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 25.63batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 25.55batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:00, 25.24batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:04<00:00, 25.26batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 25.14batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 25.25batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 25.57batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 25.44batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 25.60batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 25.17batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 25.47batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:05<?, ?epoch/s, loss=0.0004, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:05<00:46,  5.11s/epoch, loss=0.0004, prev_loss=nan]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:08, 14.47batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:07, 16.59batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:06, 18.64batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:05, 20.38batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:05, 21.73batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 22.86batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 23.73batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:04, 24.31batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 24.84batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 25.23batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 25.46batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 24.44batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 24.54batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 24.52batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 24.54batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:03, 24.89batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:02<00:02, 24.96batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 25.33batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 25.44batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 25.62batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 25.73batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 25.61batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 25.43batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:02, 25.39batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:01, 25.37batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:03<00:01, 25.22batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 25.20batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 25.41batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 25.53batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 25.58batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 25.25batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 25.18batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:00, 25.24batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:04<00:00, 25.30batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 25.41batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 25.57batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 25.68batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 25.70batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 25.79batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 25.75batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 25.54batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:10<00:46,  5.11s/epoch, loss=0.000397, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:10<00:40,  5.11s/epoch, loss=0.000397, prev_loss=nan]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:07, 16.01batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:06, 17.82batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:05, 19.52batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:05, 21.09batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:04, 22.45batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 23.40batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 24.08batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:04, 24.64batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 25.05batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 25.29batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 25.60batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 23.33batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 24.07batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 24.63batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 24.98batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:02, 25.35batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:02<00:02, 25.57batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 25.55batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 25.68batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 25.76batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 25.75batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 25.80batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 25.86batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:02, 25.90batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:01, 25.88batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:03<00:01, 25.92batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 25.82batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 25.86batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 25.89batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 25.86batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 25.89batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 26.04batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:00, 25.81batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:03<00:00, 25.85batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 25.99batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 25.80batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 25.82batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 25.84batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 25.88batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 25.93batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 26.05batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:15<00:40,  5.11s/epoch, loss=0.000397, prev_loss=0.000397]
Training epochs on cuda:  30%|███       | 3/10 [00:15<00:35,  5.09s/epoch, loss=0.000397, prev_loss=0.000397]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:06, 18.23batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:05, 19.83batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:05, 21.33batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:04, 22.58batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:04, 23.50batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 24.25batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 24.83batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:03, 25.14batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 25.37batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 25.51batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 25.67batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 23.48batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 24.18batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 24.64batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 24.72batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:03, 24.88batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:02<00:02, 24.92batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 25.06batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 25.05batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 25.04batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 25.09batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 25.16batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 25.21batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:02, 25.49batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:01, 25.56batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:03<00:01, 25.47batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 25.40batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 25.32batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 25.15batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 25.08batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 25.30batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 25.25batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:00, 25.53batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:04<00:00, 24.84batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 24.75batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 25.21batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 25.47batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 25.58batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 25.73batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 25.85batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 25.85batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:20<00:35,  5.09s/epoch, loss=0.000392, prev_loss=0.000397]
Training epochs on cuda:  40%|████      | 4/10 [00:20<00:30,  5.09s/epoch, loss=0.000392, prev_loss=0.000397]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:07, 16.61batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:06, 18.49batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:05, 20.13batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:05, 21.61batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:04, 22.85batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 23.74batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 24.40batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:04, 24.85batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 25.19batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 25.42batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 25.74batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 24.15batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 24.66batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 25.07batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 24.98batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:02, 25.40batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:01<00:02, 25.97batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 26.50batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 26.73batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 26.90batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 27.08batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 27.21batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 26.99batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:01, 26.79batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:01, 26.49batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:02<00:01, 26.39batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 26.31batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 26.17batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 26.08batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 25.72batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 25.46batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 25.40batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:00, 25.21batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:03<00:00, 25.17batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 25.21batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 25.43batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 25.64batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 25.58batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 25.50batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 25.50batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 25.48batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:25<00:30,  5.09s/epoch, loss=0.00039, prev_loss=0.000392] 
Training epochs on cuda:  50%|█████     | 5/10 [00:25<00:25,  5.06s/epoch, loss=0.00039, prev_loss=0.000392]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:08, 14.74batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:07, 16.84batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:06, 18.92batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:05, 20.62batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:04, 22.02batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 23.07batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 23.71batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:04, 24.36batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 24.84batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 25.30batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 25.42batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 23.38batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 24.11batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 24.69batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 25.02batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:03, 25.27batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:02<00:02, 25.50batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 25.65batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 25.73batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 25.95batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 25.80batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 25.98batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 25.96batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:01, 26.01batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:01, 26.03batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:03<00:01, 25.66batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 25.67batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 25.45batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 24.93batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 24.62batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 24.87batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 25.11batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:00, 25.28batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:04<00:00, 25.37batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 25.51batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 25.65batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 25.25batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 25.36batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 25.37batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 25.53batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 25.66batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:30<00:25,  5.06s/epoch, loss=0.00039, prev_loss=0.00039] 
Training epochs on cuda:  60%|██████    | 6/10 [00:30<00:20,  5.06s/epoch, loss=0.00039, prev_loss=0.00039]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:07, 16.81batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:06, 18.68batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:05, 20.22batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:05, 21.65batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:04, 22.85batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 23.62batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 23.90batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:04, 24.29batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 24.69batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 24.91batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 24.85batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 24.79batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 25.03batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 25.22batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 25.39batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:03, 25.25batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:02<00:02, 25.10batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 25.45batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 25.66batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 25.68batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 25.72batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 25.87batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 25.65batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:02, 25.57batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:01, 25.60batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:03<00:01, 25.61batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 25.68batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 25.58batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 25.56batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 25.38batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 25.18batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 25.49batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:00, 25.63batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:03<00:00, 25.89batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 25.91batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 25.84batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 26.13batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 26.12batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 26.04batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 26.01batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 26.09batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:35<00:20,  5.06s/epoch, loss=0.000374, prev_loss=0.00039]
Training epochs on cuda:  70%|███████   | 7/10 [00:35<00:15,  5.06s/epoch, loss=0.000374, prev_loss=0.00039]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:07, 17.26batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:06, 19.05batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:05, 20.78batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:05, 22.14batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:04, 23.19batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 23.99batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 24.58batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:03, 25.09batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 25.32batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 25.07batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 24.81batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 22.74batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 23.56batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 23.73batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 24.32batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:03, 24.82batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:02<00:02, 25.03batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 25.32batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 25.54batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 25.39batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 25.09batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 25.42batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 25.44batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:02, 25.41batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:01, 25.59batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:03<00:01, 25.62batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 25.46batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 25.63batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 25.60batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 25.45batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 25.22batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 25.31batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:00, 25.32batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:04<00:00, 25.15batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 25.18batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 25.36batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 25.62batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 25.67batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 25.53batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 25.33batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 25.47batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:40<00:15,  5.06s/epoch, loss=0.000354, prev_loss=0.000374]
Training epochs on cuda:  80%|████████  | 8/10 [00:40<00:10,  5.08s/epoch, loss=0.000354, prev_loss=0.000374]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/123 [00:00<00:15,  8.01batch/s]

Training batches on cuda:   3%|▎         | 4/123 [00:00<00:11, 10.07batch/s]

Training batches on cuda:   6%|▌         | 7/123 [00:00<00:09, 12.34batch/s]

Training batches on cuda:   8%|▊         | 10/123 [00:00<00:07, 14.66batch/s]

Training batches on cuda:  11%|█         | 13/123 [00:00<00:06, 16.88batch/s]

Training batches on cuda:  13%|█▎        | 16/123 [00:00<00:05, 18.80batch/s]

Training batches on cuda:  15%|█▌        | 19/123 [00:00<00:05, 20.53batch/s]

Training batches on cuda:  18%|█▊        | 22/123 [00:00<00:04, 21.30batch/s]

Training batches on cuda:  20%|██        | 25/123 [00:01<00:04, 21.87batch/s]

Training batches on cuda:  23%|██▎       | 28/123 [00:01<00:04, 23.13batch/s]

Training batches on cuda:  25%|██▌       | 31/123 [00:01<00:03, 23.47batch/s]

Training batches on cuda:  28%|██▊       | 34/123 [00:01<00:03, 24.52batch/s]

Training batches on cuda:  30%|███       | 37/123 [00:01<00:03, 24.93batch/s]

Training batches on cuda:  33%|███▎      | 40/123 [00:01<00:03, 24.20batch/s]

Training batches on cuda:  35%|███▍      | 43/123 [00:01<00:03, 24.68batch/s]

Training batches on cuda:  37%|███▋      | 46/123 [00:01<00:03, 24.87batch/s]

Training batches on cuda:  40%|███▉      | 49/123 [00:02<00:02, 24.95batch/s]

Training batches on cuda:  42%|████▏     | 52/123 [00:02<00:02, 24.75batch/s]

Training batches on cuda:  45%|████▍     | 55/123 [00:02<00:02, 24.78batch/s]

Training batches on cuda:  47%|████▋     | 58/123 [00:02<00:02, 24.94batch/s]

Training batches on cuda:  50%|████▉     | 61/123 [00:02<00:02, 25.28batch/s]

Training batches on cuda:  52%|█████▏    | 64/123 [00:02<00:02, 25.59batch/s]

Training batches on cuda:  54%|█████▍    | 67/123 [00:02<00:02, 25.80batch/s]

Training batches on cuda:  57%|█████▋    | 70/123 [00:02<00:02, 25.88batch/s]

Training batches on cuda:  59%|█████▉    | 73/123 [00:02<00:01, 25.95batch/s]

Training batches on cuda:  62%|██████▏   | 76/123 [00:03<00:01, 26.03batch/s]

Training batches on cuda:  64%|██████▍   | 79/123 [00:03<00:01, 25.98batch/s]

Training batches on cuda:  67%|██████▋   | 82/123 [00:03<00:01, 25.99batch/s]

Training batches on cuda:  69%|██████▉   | 85/123 [00:03<00:01, 25.97batch/s]

Training batches on cuda:  72%|███████▏  | 88/123 [00:03<00:01, 25.93batch/s]

Training batches on cuda:  74%|███████▍  | 91/123 [00:03<00:01, 25.66batch/s]

Training batches on cuda:  76%|███████▋  | 94/123 [00:03<00:01, 25.50batch/s]

Training batches on cuda:  79%|███████▉  | 97/123 [00:03<00:01, 25.44batch/s]

Training batches on cuda:  81%|████████▏ | 100/123 [00:04<00:00, 25.27batch/s]

Training batches on cuda:  84%|████████▎ | 103/123 [00:04<00:00, 25.45batch/s]

Training batches on cuda:  86%|████████▌ | 106/123 [00:04<00:00, 25.16batch/s]

Training batches on cuda:  89%|████████▊ | 109/123 [00:04<00:00, 25.08batch/s]

Training batches on cuda:  91%|█████████ | 112/123 [00:04<00:00, 25.27batch/s]

Training batches on cuda:  93%|█████████▎| 115/123 [00:04<00:00, 24.65batch/s]

Training batches on cuda:  96%|█████████▌| 118/123 [00:04<00:00, 23.72batch/s]

Training batches on cuda:  98%|█████████▊| 121/123 [00:04<00:00, 24.66batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:45<00:10,  5.08s/epoch, loss=0.000364, prev_loss=0.000354]
Training epochs on cuda:  90%|█████████ | 9/10 [00:45<00:05,  5.09s/epoch, loss=0.000364, prev_loss=0.000354]

Training batches on cuda:   0%|          | 0/123 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/123 [00:00<00:06, 19.34batch/s]

Training batches on cuda:   4%|▍         | 5/123 [00:00<00:05, 21.07batch/s]

Training batches on cuda:   7%|▋         | 8/123 [00:00<00:05, 22.47batch/s]

Training batches on cuda:   9%|▉         | 11/123 [00:00<00:04, 23.15batch/s]

Training batches on cuda:  11%|█▏        | 14/123 [00:00<00:04, 23.74batch/s]

Training batches on cuda:  14%|█▍        | 17/123 [00:00<00:04, 24.39batch/s]

Training batches on cuda:  16%|█▋        | 20/123 [00:00<00:04, 24.89batch/s]

Training batches on cuda:  19%|█▊        | 23/123 [00:00<00:03, 25.28batch/s]

Training batches on cuda:  21%|██        | 26/123 [00:01<00:03, 25.39batch/s]

Training batches on cuda:  24%|██▎       | 29/123 [00:01<00:03, 25.59batch/s]

Training batches on cuda:  26%|██▌       | 32/123 [00:01<00:03, 23.51batch/s]

Training batches on cuda:  28%|██▊       | 35/123 [00:01<00:03, 23.86batch/s]

Training batches on cuda:  31%|███       | 38/123 [00:01<00:03, 24.34batch/s]

Training batches on cuda:  33%|███▎      | 41/123 [00:01<00:03, 24.48batch/s]

Training batches on cuda:  36%|███▌      | 44/123 [00:01<00:03, 24.69batch/s]

Training batches on cuda:  38%|███▊      | 47/123 [00:01<00:03, 25.16batch/s]

Training batches on cuda:  41%|████      | 50/123 [00:02<00:02, 25.45batch/s]

Training batches on cuda:  43%|████▎     | 53/123 [00:02<00:02, 25.52batch/s]

Training batches on cuda:  46%|████▌     | 56/123 [00:02<00:02, 24.94batch/s]

Training batches on cuda:  48%|████▊     | 59/123 [00:02<00:02, 24.90batch/s]

Training batches on cuda:  50%|█████     | 62/123 [00:02<00:02, 24.52batch/s]

Training batches on cuda:  53%|█████▎    | 65/123 [00:02<00:02, 24.43batch/s]

Training batches on cuda:  55%|█████▌    | 68/123 [00:02<00:02, 24.31batch/s]

Training batches on cuda:  58%|█████▊    | 71/123 [00:02<00:02, 24.20batch/s]

Training batches on cuda:  60%|██████    | 74/123 [00:02<00:02, 24.22batch/s]

Training batches on cuda:  63%|██████▎   | 77/123 [00:03<00:01, 24.27batch/s]

Training batches on cuda:  65%|██████▌   | 80/123 [00:03<00:01, 24.16batch/s]

Training batches on cuda:  67%|██████▋   | 83/123 [00:03<00:01, 24.36batch/s]

Training batches on cuda:  70%|██████▉   | 86/123 [00:03<00:01, 24.85batch/s]

Training batches on cuda:  72%|███████▏  | 89/123 [00:03<00:01, 23.63batch/s]

Training batches on cuda:  75%|███████▍  | 92/123 [00:03<00:01, 23.96batch/s]

Training batches on cuda:  77%|███████▋  | 95/123 [00:03<00:01, 24.10batch/s]

Training batches on cuda:  80%|███████▉  | 98/123 [00:03<00:01, 24.15batch/s]

Training batches on cuda:  82%|████████▏ | 101/123 [00:04<00:00, 24.18batch/s]

Training batches on cuda:  85%|████████▍ | 104/123 [00:04<00:00, 24.23batch/s]

Training batches on cuda:  87%|████████▋ | 107/123 [00:04<00:00, 24.18batch/s]

Training batches on cuda:  89%|████████▉ | 110/123 [00:04<00:00, 24.15batch/s]

Training batches on cuda:  92%|█████████▏| 113/123 [00:04<00:00, 24.23batch/s]

Training batches on cuda:  94%|█████████▍| 116/123 [00:04<00:00, 24.29batch/s]

Training batches on cuda:  97%|█████████▋| 119/123 [00:04<00:00, 24.22batch/s]

Training batches on cuda:  99%|█████████▉| 122/123 [00:04<00:00, 23.90batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:50<00:05,  5.09s/epoch, loss=0.000363, prev_loss=0.000364]2021-09-16:19:40:54,660 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:40:55,731 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:41:10,664 INFO     [evaluator.py:657] Evaluation took 14.93s seconds
2021-09-16:19:41:10,778 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00,  9.97s/epoch, loss=0.000363, prev_loss=0.000364]
Training epochs on cuda: 100%|██████████| 10/10 [01:07<00:00,  6.71s/epoch, loss=0.000363, prev_loss=0.000364]
2021-09-16:19:41:10,779 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmp65cy9v2u'
2021-09-16:19:41:10,873 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmp65cy9v2u' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:17, 2.15ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:16, 2.28ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:15, 2.38ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:15, 2.45ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:14, 2.51ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:14, 2.55ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.58ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.60ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.61ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.62ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.64ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.63ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.64ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.64ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:11, 2.61ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.62ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.63ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.62ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.63ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.64ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.64ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.63ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.64ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.64ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.64ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.64ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.64ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.64ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.64ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.64ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.64ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.62ktriple/s]
2021-09-16:19:41:25,701 INFO     [evaluator.py:657] Evaluation took 14.81s seconds
[I 2021-09-16 19:41:25,706] Trial 96 finished with value: 0.3211621999221975 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.07799812300090662, 'optimizer.lr': 0.03278199761740297, 'negative_sampler.num_negs_per_pos': 29, 'training.batch_size': 2532}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:41:25,730 WARNING  [api.py:823] No random seed is specified. Setting to 3402386094.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:11,  9.71batch/s]

Training batches on cuda:   4%|▎         | 4/114 [00:00<00:09, 12.04batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:07, 14.54batch/s]

Training batches on cuda:   9%|▉         | 10/114 [00:00<00:06, 17.01batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:00<00:05, 19.35batch/s]

Training batches on cuda:  14%|█▍        | 16/114 [00:00<00:04, 21.46batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:00<00:04, 23.13batch/s]

Training batches on cuda:  19%|█▉        | 22/114 [00:00<00:03, 24.35batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:00<00:03, 25.10batch/s]

Training batches on cuda:  25%|██▍       | 28/114 [00:01<00:03, 25.75batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:01<00:03, 26.14batch/s]

Training batches on cuda:  30%|██▉       | 34/114 [00:01<00:03, 26.49batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:01<00:02, 26.61batch/s]

Training batches on cuda:  35%|███▌      | 40/114 [00:01<00:02, 26.67batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:01<00:02, 26.88batch/s]

Training batches on cuda:  40%|████      | 46/114 [00:01<00:02, 26.95batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:01<00:02, 26.91batch/s]

Training batches on cuda:  46%|████▌     | 52/114 [00:01<00:02, 27.03batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:02<00:02, 27.06batch/s]

Training batches on cuda:  51%|█████     | 58/114 [00:02<00:02, 27.12batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:02<00:01, 27.28batch/s]

Training batches on cuda:  56%|█████▌    | 64/114 [00:02<00:01, 27.30batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:02<00:01, 27.14batch/s]

Training batches on cuda:  61%|██████▏   | 70/114 [00:02<00:01, 27.02batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:02<00:01, 27.04batch/s]

Training batches on cuda:  67%|██████▋   | 76/114 [00:02<00:01, 27.04batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:02<00:01, 27.12batch/s]

Training batches on cuda:  72%|███████▏  | 82/114 [00:03<00:01, 27.04batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:03<00:01, 27.13batch/s]

Training batches on cuda:  77%|███████▋  | 88/114 [00:03<00:00, 27.11batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:03<00:00, 27.02batch/s]

Training batches on cuda:  82%|████████▏ | 94/114 [00:03<00:00, 27.02batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:03<00:00, 26.89batch/s]

Training batches on cuda:  88%|████████▊ | 100/114 [00:03<00:00, 26.48batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:03<00:00, 26.17batch/s]

Training batches on cuda:  93%|█████████▎| 106/114 [00:03<00:00, 26.53batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:04<00:00, 26.26batch/s]

Training batches on cuda:  98%|█████████▊| 112/114 [00:04<00:00, 26.42batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:04<?, ?epoch/s, loss=0.000369, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:04<00:40,  4.46s/epoch, loss=0.000369, prev_loss=nan]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:05, 18.92batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 20.69batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 21.92batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 23.26batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.32batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 25.13batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.74batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 26.15batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 26.35batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.48batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.74batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.90batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.97batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 26.89batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 26.91batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 26.54batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.13batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 27.30batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 27.28batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 26.83batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.58batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.62batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.83batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 26.79batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 26.59batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 25.41batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 25.76batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 25.87batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 25.81batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:01, 24.99batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 24.92batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 24.55batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 24.85batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 25.29batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 25.65batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 25.90batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 25.99batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 26.28batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:09<00:40,  4.46s/epoch, loss=0.000367, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:09<00:35,  4.50s/epoch, loss=0.000367, prev_loss=nan]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   1%|          | 1/114 [00:00<00:11,  9.89batch/s]

Training batches on cuda:   4%|▎         | 4/114 [00:00<00:09, 12.07batch/s]

Training batches on cuda:   6%|▌         | 7/114 [00:00<00:07, 14.44batch/s]

Training batches on cuda:   9%|▉         | 10/114 [00:00<00:06, 16.58batch/s]

Training batches on cuda:  11%|█▏        | 13/114 [00:00<00:05, 18.86batch/s]

Training batches on cuda:  14%|█▍        | 16/114 [00:00<00:04, 20.77batch/s]

Training batches on cuda:  17%|█▋        | 19/114 [00:00<00:04, 20.50batch/s]

Training batches on cuda:  19%|█▉        | 22/114 [00:00<00:04, 22.01batch/s]

Training batches on cuda:  22%|██▏       | 25/114 [00:01<00:03, 23.28batch/s]

Training batches on cuda:  25%|██▍       | 28/114 [00:01<00:03, 24.21batch/s]

Training batches on cuda:  27%|██▋       | 31/114 [00:01<00:03, 24.88batch/s]

Training batches on cuda:  30%|██▉       | 34/114 [00:01<00:03, 25.47batch/s]

Training batches on cuda:  32%|███▏      | 37/114 [00:01<00:02, 25.80batch/s]

Training batches on cuda:  35%|███▌      | 40/114 [00:01<00:02, 25.82batch/s]

Training batches on cuda:  38%|███▊      | 43/114 [00:01<00:02, 25.90batch/s]

Training batches on cuda:  40%|████      | 46/114 [00:01<00:02, 26.17batch/s]

Training batches on cuda:  43%|████▎     | 49/114 [00:01<00:02, 26.28batch/s]

Training batches on cuda:  46%|████▌     | 52/114 [00:02<00:02, 26.46batch/s]

Training batches on cuda:  48%|████▊     | 55/114 [00:02<00:02, 26.54batch/s]

Training batches on cuda:  51%|█████     | 58/114 [00:02<00:02, 27.12batch/s]

Training batches on cuda:  54%|█████▎    | 61/114 [00:02<00:01, 27.46batch/s]

Training batches on cuda:  56%|█████▌    | 64/114 [00:02<00:01, 27.77batch/s]

Training batches on cuda:  59%|█████▉    | 67/114 [00:02<00:01, 27.90batch/s]

Training batches on cuda:  61%|██████▏   | 70/114 [00:02<00:01, 28.16batch/s]

Training batches on cuda:  64%|██████▍   | 73/114 [00:02<00:01, 28.07batch/s]

Training batches on cuda:  67%|██████▋   | 76/114 [00:02<00:01, 27.55batch/s]

Training batches on cuda:  69%|██████▉   | 79/114 [00:03<00:01, 27.35batch/s]

Training batches on cuda:  72%|███████▏  | 82/114 [00:03<00:01, 27.39batch/s]

Training batches on cuda:  75%|███████▍  | 85/114 [00:03<00:01, 27.19batch/s]

Training batches on cuda:  77%|███████▋  | 88/114 [00:03<00:00, 27.04batch/s]

Training batches on cuda:  80%|███████▉  | 91/114 [00:03<00:00, 27.02batch/s]

Training batches on cuda:  82%|████████▏ | 94/114 [00:03<00:00, 27.04batch/s]

Training batches on cuda:  85%|████████▌ | 97/114 [00:03<00:00, 26.97batch/s]

Training batches on cuda:  88%|████████▊ | 100/114 [00:03<00:00, 27.03batch/s]

Training batches on cuda:  90%|█████████ | 103/114 [00:03<00:00, 26.79batch/s]

Training batches on cuda:  93%|█████████▎| 106/114 [00:04<00:00, 26.63batch/s]

Training batches on cuda:  96%|█████████▌| 109/114 [00:04<00:00, 26.66batch/s]

Training batches on cuda:  98%|█████████▊| 112/114 [00:04<00:00, 26.53batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:13<00:35,  4.50s/epoch, loss=0.000365, prev_loss=0.000367]
Training epochs on cuda:  30%|███       | 3/10 [00:13<00:31,  4.53s/epoch, loss=0.000365, prev_loss=0.000367]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 18.31batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 20.10batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 21.81batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 23.18batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.21batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.88batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.43batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.91batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 26.25batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.59batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.36batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.83batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 27.18batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.49batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.30batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 27.11batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.09batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 26.99batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 27.00batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 27.09batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 27.20batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 27.64batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 27.81batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.53batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.28batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.17batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:02<00:01, 27.02batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.10batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.12batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.09batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.16batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 27.14batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 27.28batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 27.36batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 27.29batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:03<00:00, 27.32batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.27batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 27.16batch/s]

                                                                              
Training epochs on cuda:  30%|███       | 3/10 [00:18<00:31,  4.53s/epoch, loss=0.000351, prev_loss=0.000365]
Training epochs on cuda:  40%|████      | 4/10 [00:18<00:27,  4.50s/epoch, loss=0.000351, prev_loss=0.000365]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 16.77batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 18.82batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 20.76batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.43batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 23.60batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.64batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.39batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.78batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 26.11batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.51batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.50batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.76batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.97batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.05batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.10batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 25.64batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 26.10batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 26.40batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 26.60batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 26.69batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.75batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.69batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.72batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 26.90batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 26.88batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 26.93batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 26.89batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.02batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 27.06batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 26.73batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 26.64batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.73batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 26.72batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 26.45batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 26.64batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 26.60batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 26.15batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 26.44batch/s]

                                                                              
Training epochs on cuda:  40%|████      | 4/10 [00:22<00:27,  4.50s/epoch, loss=0.000341, prev_loss=0.000351]
Training epochs on cuda:  50%|█████     | 5/10 [00:22<00:22,  4.51s/epoch, loss=0.000341, prev_loss=0.000351]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 18.24batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 19.90batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 21.50batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.93batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.04batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.90batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.59batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.62batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 25.88batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.12batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.34batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:02, 26.50batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.64batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 27.16batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 27.21batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 27.45batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 27.76batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:01<00:02, 28.11batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 28.10batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:01, 28.28batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:02, 25.38batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.23batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.92batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.29batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 27.23batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 26.96batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 26.94batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 26.96batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 26.95batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 26.94batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 27.02batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.96batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 26.85batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 26.96batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 26.88batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 26.91batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 25.95batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 25.78batch/s]

                                                                              
Training epochs on cuda:  50%|█████     | 5/10 [00:27<00:22,  4.51s/epoch, loss=0.000336, prev_loss=0.000341]
Training epochs on cuda:  60%|██████    | 6/10 [00:27<00:18,  4.51s/epoch, loss=0.000336, prev_loss=0.000341]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:05, 18.80batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 20.50batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 21.75batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.98batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 23.66batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.36batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.17batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.29batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 25.84batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 26.16batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 26.50batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 25.55batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 25.82batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 26.14batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 26.39batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 26.52batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 26.34batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 26.50batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 26.26batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 26.66batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.80batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.90batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.68batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 26.86batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 24.74batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 25.50batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 26.28batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 26.87batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 26.93batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 26.93batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 26.98batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.80batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 26.88batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 26.95batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 26.99batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 27.05batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.08batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 26.96batch/s]

                                                                              
Training epochs on cuda:  60%|██████    | 6/10 [00:31<00:18,  4.51s/epoch, loss=0.000287, prev_loss=0.000336]
Training epochs on cuda:  70%|███████   | 7/10 [00:31<00:13,  4.55s/epoch, loss=0.000287, prev_loss=0.000336]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:07, 14.23batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:06, 16.50batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 18.56batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:05, 20.38batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 21.84batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:04, 23.15batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 24.17batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.22batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 25.63batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 24.83batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 24.47batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 24.89batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:03, 24.65batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 24.52batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 24.59batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 24.94batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:02<00:02, 25.00batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 25.55batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 25.83batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 25.92batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.06batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.30batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.53batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 26.29batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 26.43batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:03<00:01, 26.37batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 26.41batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 24.04batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 24.76batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 25.26batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 25.82batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.11batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 26.31batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 26.49batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:04<00:00, 26.73batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 26.69batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 26.80batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 26.85batch/s]

                                                                              
Training epochs on cuda:  70%|███████   | 7/10 [00:36<00:13,  4.55s/epoch, loss=0.00019, prev_loss=0.000287] 
Training epochs on cuda:  80%|████████  | 8/10 [00:36<00:09,  4.58s/epoch, loss=0.00019, prev_loss=0.000287]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:05, 18.70batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 20.23batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:04, 21.85batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 23.07batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 24.11batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 25.02batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 25.55batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.89batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 25.53batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 25.66batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 25.76batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 25.63batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 25.64batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 25.48batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 25.81batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 25.31batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 26.33batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 27.09batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 27.66batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:01, 28.04batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 27.95batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 27.72batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 27.49batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 27.03batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 26.97batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 27.05batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 27.09batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 27.04batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 26.96batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 27.00batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 26.82batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.73batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 25.96batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 25.34batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 25.27batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 25.28batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 25.11batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 25.25batch/s]

                                                                              
Training epochs on cuda:  80%|████████  | 8/10 [00:41<00:09,  4.58s/epoch, loss=0.00018, prev_loss=0.00019] 
Training epochs on cuda:  90%|█████████ | 9/10 [00:41<00:04,  4.59s/epoch, loss=0.00018, prev_loss=0.00019]

Training batches on cuda:   0%|          | 0/114 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/114 [00:00<00:06, 16.79batch/s]

Training batches on cuda:   4%|▍         | 5/114 [00:00<00:05, 18.82batch/s]

Training batches on cuda:   7%|▋         | 8/114 [00:00<00:05, 20.60batch/s]

Training batches on cuda:  10%|▉         | 11/114 [00:00<00:04, 22.21batch/s]

Training batches on cuda:  12%|█▏        | 14/114 [00:00<00:04, 23.02batch/s]

Training batches on cuda:  15%|█▍        | 17/114 [00:00<00:03, 24.27batch/s]

Training batches on cuda:  18%|█▊        | 20/114 [00:00<00:03, 24.93batch/s]

Training batches on cuda:  20%|██        | 23/114 [00:00<00:03, 25.41batch/s]

Training batches on cuda:  23%|██▎       | 26/114 [00:01<00:03, 25.22batch/s]

Training batches on cuda:  25%|██▌       | 29/114 [00:01<00:03, 25.53batch/s]

Training batches on cuda:  28%|██▊       | 32/114 [00:01<00:03, 25.63batch/s]

Training batches on cuda:  31%|███       | 35/114 [00:01<00:03, 25.70batch/s]

Training batches on cuda:  33%|███▎      | 38/114 [00:01<00:02, 26.13batch/s]

Training batches on cuda:  36%|███▌      | 41/114 [00:01<00:02, 25.82batch/s]

Training batches on cuda:  39%|███▊      | 44/114 [00:01<00:02, 26.30batch/s]

Training batches on cuda:  41%|████      | 47/114 [00:01<00:02, 26.40batch/s]

Training batches on cuda:  44%|████▍     | 50/114 [00:01<00:02, 26.70batch/s]

Training batches on cuda:  46%|████▋     | 53/114 [00:02<00:02, 26.42batch/s]

Training batches on cuda:  49%|████▉     | 56/114 [00:02<00:02, 26.53batch/s]

Training batches on cuda:  52%|█████▏    | 59/114 [00:02<00:02, 26.68batch/s]

Training batches on cuda:  54%|█████▍    | 62/114 [00:02<00:01, 26.69batch/s]

Training batches on cuda:  57%|█████▋    | 65/114 [00:02<00:01, 26.40batch/s]

Training batches on cuda:  60%|█████▉    | 68/114 [00:02<00:01, 26.17batch/s]

Training batches on cuda:  62%|██████▏   | 71/114 [00:02<00:01, 25.73batch/s]

Training batches on cuda:  65%|██████▍   | 74/114 [00:02<00:01, 25.96batch/s]

Training batches on cuda:  68%|██████▊   | 77/114 [00:02<00:01, 26.39batch/s]

Training batches on cuda:  70%|███████   | 80/114 [00:03<00:01, 26.57batch/s]

Training batches on cuda:  73%|███████▎  | 83/114 [00:03<00:01, 26.68batch/s]

Training batches on cuda:  75%|███████▌  | 86/114 [00:03<00:01, 26.72batch/s]

Training batches on cuda:  78%|███████▊  | 89/114 [00:03<00:00, 26.72batch/s]

Training batches on cuda:  81%|████████  | 92/114 [00:03<00:00, 26.84batch/s]

Training batches on cuda:  83%|████████▎ | 95/114 [00:03<00:00, 26.98batch/s]

Training batches on cuda:  86%|████████▌ | 98/114 [00:03<00:00, 26.87batch/s]

Training batches on cuda:  89%|████████▊ | 101/114 [00:03<00:00, 26.78batch/s]

Training batches on cuda:  91%|█████████ | 104/114 [00:03<00:00, 26.87batch/s]

Training batches on cuda:  94%|█████████▍| 107/114 [00:04<00:00, 26.90batch/s]

Training batches on cuda:  96%|█████████▋| 110/114 [00:04<00:00, 27.22batch/s]

Training batches on cuda:  99%|█████████▉| 113/114 [00:04<00:00, 27.67batch/s]

                                                                              
Training epochs on cuda:  90%|█████████ | 9/10 [00:45<00:04,  4.59s/epoch, loss=0.000171, prev_loss=0.00018]2021-09-16:19:42:11,768 INFO     [evaluator.py:306] Starting batch_size search for evaluation now...
2021-09-16:19:42:12,765 INFO     [evaluator.py:352] Concluded batch_size search with batch_size=256.
2021-09-16:19:42:27,675 INFO     [evaluator.py:657] Evaluation took 14.91s seconds
2021-09-16:19:42:27,801 INFO     [training_loop.py:1072] => Saved checkpoint after having finished epoch 10.

Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  9.38s/epoch, loss=0.000171, prev_loss=0.00018]
Training epochs on cuda: 100%|██████████| 10/10 [01:01<00:00,  6.16s/epoch, loss=0.000171, prev_loss=0.00018]
2021-09-16:19:42:27,802 INFO     [training_loop.py:1092] => loading checkpoint '/tmp/tmpbfpse_30'
2021-09-16:19:42:27,883 INFO     [training_loop.py:1135] => loaded checkpoint '/tmp/tmpbfpse_30' stopped after having finished epoch 10

Evaluating on cuda:   0%|          | 0.00/38.8k [00:00<?, ?triple/s]
Evaluating on cuda:   1%|          | 256/38.8k [00:00<00:15, 2.50ktriple/s]
Evaluating on cuda:   2%|▏         | 768/38.8k [00:00<00:14, 2.55ktriple/s]
Evaluating on cuda:   3%|▎         | 1.28k/38.8k [00:00<00:14, 2.58ktriple/s]
Evaluating on cuda:   5%|▍         | 1.79k/38.8k [00:00<00:14, 2.60ktriple/s]
Evaluating on cuda:   6%|▌         | 2.30k/38.8k [00:00<00:13, 2.62ktriple/s]
Evaluating on cuda:   7%|▋         | 2.82k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:   9%|▊         | 3.33k/38.8k [00:01<00:13, 2.63ktriple/s]
Evaluating on cuda:  10%|▉         | 3.84k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  11%|█         | 4.35k/38.8k [00:01<00:13, 2.64ktriple/s]
Evaluating on cuda:  13%|█▎        | 4.86k/38.8k [00:01<00:12, 2.65ktriple/s]
Evaluating on cuda:  14%|█▍        | 5.38k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  15%|█▌        | 5.89k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  16%|█▋        | 6.40k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  18%|█▊        | 6.91k/38.8k [00:02<00:12, 2.65ktriple/s]
Evaluating on cuda:  19%|█▉        | 7.42k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  20%|██        | 7.94k/38.8k [00:02<00:11, 2.65ktriple/s]
Evaluating on cuda:  22%|██▏       | 8.45k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  23%|██▎       | 8.96k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  24%|██▍       | 9.47k/38.8k [00:03<00:11, 2.65ktriple/s]
Evaluating on cuda:  26%|██▌       | 9.98k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  27%|██▋       | 10.5k/38.8k [00:03<00:10, 2.65ktriple/s]
Evaluating on cuda:  28%|██▊       | 11.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  30%|██▉       | 11.5k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  31%|███       | 12.0k/38.8k [00:04<00:10, 2.65ktriple/s]
Evaluating on cuda:  32%|███▏      | 12.5k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  34%|███▎      | 13.1k/38.8k [00:04<00:09, 2.65ktriple/s]
Evaluating on cuda:  35%|███▍      | 13.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  36%|███▋      | 14.1k/38.8k [00:05<00:09, 2.66ktriple/s]
Evaluating on cuda:  38%|███▊      | 14.6k/38.8k [00:05<00:09, 2.65ktriple/s]
Evaluating on cuda:  39%|███▉      | 15.1k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  40%|████      | 15.6k/38.8k [00:05<00:08, 2.66ktriple/s]
Evaluating on cuda:  42%|████▏     | 16.1k/38.8k [00:06<00:08, 2.66ktriple/s]
Evaluating on cuda:  43%|████▎     | 16.6k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  44%|████▍     | 17.2k/38.8k [00:06<00:08, 2.65ktriple/s]
Evaluating on cuda:  45%|████▌     | 17.7k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  47%|████▋     | 18.2k/38.8k [00:06<00:07, 2.65ktriple/s]
Evaluating on cuda:  48%|████▊     | 18.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  49%|████▉     | 19.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  51%|█████     | 19.7k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  52%|█████▏    | 20.2k/38.8k [00:07<00:07, 2.65ktriple/s]
Evaluating on cuda:  53%|█████▎    | 20.7k/38.8k [00:07<00:06, 2.65ktriple/s]
Evaluating on cuda:  55%|█████▍    | 21.2k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  56%|█████▌    | 21.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  57%|█████▋    | 22.3k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  59%|█████▊    | 22.8k/38.8k [00:08<00:06, 2.65ktriple/s]
Evaluating on cuda:  60%|█████▉    | 23.3k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  61%|██████▏   | 23.8k/38.8k [00:08<00:05, 2.65ktriple/s]
Evaluating on cuda:  63%|██████▎   | 24.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  64%|██████▍   | 24.8k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  65%|██████▌   | 25.3k/38.8k [00:09<00:05, 2.65ktriple/s]
Evaluating on cuda:  67%|██████▋   | 25.9k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  68%|██████▊   | 26.4k/38.8k [00:09<00:04, 2.65ktriple/s]
Evaluating on cuda:  69%|██████▉   | 26.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  71%|███████   | 27.4k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  72%|███████▏  | 27.9k/38.8k [00:10<00:04, 2.65ktriple/s]
Evaluating on cuda:  73%|███████▎  | 28.4k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  74%|███████▍  | 28.9k/38.8k [00:10<00:03, 2.65ktriple/s]
Evaluating on cuda:  76%|███████▌  | 29.4k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  77%|███████▋  | 30.0k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  78%|███████▊  | 30.5k/38.8k [00:11<00:03, 2.65ktriple/s]
Evaluating on cuda:  80%|███████▉  | 31.0k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  81%|████████  | 31.5k/38.8k [00:11<00:02, 2.65ktriple/s]
Evaluating on cuda:  82%|████████▏ | 32.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  84%|████████▎ | 32.5k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  85%|████████▌ | 33.0k/38.8k [00:12<00:02, 2.65ktriple/s]
Evaluating on cuda:  86%|████████▋ | 33.5k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  88%|████████▊ | 34.0k/38.8k [00:12<00:01, 2.65ktriple/s]
Evaluating on cuda:  89%|████████▉ | 34.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  90%|█████████ | 35.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  92%|█████████▏| 35.6k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  93%|█████████▎| 36.1k/38.8k [00:13<00:01, 2.65ktriple/s]
Evaluating on cuda:  94%|█████████▍| 36.6k/38.8k [00:13<00:00, 2.65ktriple/s]
Evaluating on cuda:  96%|█████████▌| 37.1k/38.8k [00:14<00:00, 2.65ktriple/s]
Evaluating on cuda:  97%|█████████▋| 37.6k/38.8k [00:14<00:00, 2.50ktriple/s]
Evaluating on cuda:  98%|█████████▊| 38.1k/38.8k [00:14<00:00, 2.54ktriple/s]
Evaluating on cuda: 100%|█████████▉| 38.7k/38.8k [00:14<00:00, 2.57ktriple/s]
Evaluating on cuda: 100%|██████████| 38.8k/38.8k [00:14<00:00, 2.63ktriple/s]
2021-09-16:19:42:42,702 INFO     [evaluator.py:657] Evaluation took 14.79s seconds
[I 2021-09-16 19:42:42,707] Trial 97 finished with value: 0.9347863198178764 and parameters: {'loss.margin': 1, 'loss.margin_activation': 'relu', 'regularizer.weight': 0.03255757746821351, 'optimizer.lr': 0.040784954563845476, 'negative_sampler.num_negs_per_pos': 23, 'training.batch_size': 2732}. Best is trial 76 with value: 0.9770494761570895.
/srv/home/bhattacharyya/anaconda3/lib/python3.7/site-packages/optuna/distributions.py:563: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  low=low, old_high=old_high, high=high, step=step
2021-09-16:19:42:42,730 WARNING  [api.py:823] No random seed is specified. Setting to 2195741116.

Training epochs on cuda:   0%|          | 0/10 [00:00<?, ?epoch/s]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   3%|▎         | 3/106 [00:00<00:03, 26.05batch/s]

Training batches on cuda:   7%|▋         | 7/106 [00:00<00:03, 27.92batch/s]

Training batches on cuda:  10%|█         | 11/106 [00:00<00:03, 29.49batch/s]

Training batches on cuda:  14%|█▍        | 15/106 [00:00<00:03, 30.31batch/s]

Training batches on cuda:  18%|█▊        | 19/106 [00:00<00:02, 30.99batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:00<00:02, 30.29batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:00<00:02, 28.79batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:00<00:02, 29.31batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:02, 30.55batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:01<00:02, 31.62batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:01<00:02, 32.15batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:01<00:01, 31.64batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:01<00:01, 32.05batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:01<00:01, 32.26batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:01<00:01, 32.11batch/s]

Training batches on cuda:  58%|█████▊    | 61/106 [00:01<00:01, 32.11batch/s]

Training batches on cuda:  61%|██████▏   | 65/106 [00:02<00:01, 32.08batch/s]

Training batches on cuda:  65%|██████▌   | 69/106 [00:02<00:01, 32.15batch/s]

Training batches on cuda:  69%|██████▉   | 73/106 [00:02<00:01, 32.03batch/s]

Training batches on cuda:  73%|███████▎  | 77/106 [00:02<00:00, 32.07batch/s]

Training batches on cuda:  76%|███████▋  | 81/106 [00:02<00:00, 32.18batch/s]

Training batches on cuda:  80%|████████  | 85/106 [00:02<00:00, 32.06batch/s]

Training batches on cuda:  84%|████████▍ | 89/106 [00:02<00:00, 32.05batch/s]

Training batches on cuda:  88%|████████▊ | 93/106 [00:02<00:00, 32.10batch/s]

Training batches on cuda:  92%|█████████▏| 97/106 [00:03<00:00, 31.28batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:03<00:00, 31.76batch/s]

Training batches on cuda:  99%|█████████▉| 105/106 [00:03<00:00, 31.98batch/s]

                                                                              
Training epochs on cuda:   0%|          | 0/10 [00:03<?, ?epoch/s, loss=0.000346, prev_loss=nan]
Training epochs on cuda:  10%|█         | 1/10 [00:03<00:31,  3.53s/epoch, loss=0.000346, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:05, 18.40batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:04, 20.96batch/s]

Training batches on cuda:   8%|▊         | 8/106 [00:00<00:04, 20.60batch/s]

Training batches on cuda:  11%|█▏        | 12/106 [00:00<00:04, 23.12batch/s]

Training batches on cuda:  15%|█▌        | 16/106 [00:00<00:03, 25.28batch/s]

Training batches on cuda:  19%|█▉        | 20/106 [00:00<00:03, 27.12batch/s]

Training batches on cuda:  23%|██▎       | 24/106 [00:00<00:02, 28.49batch/s]

Training batches on cuda:  26%|██▋       | 28/106 [00:00<00:02, 29.31batch/s]

Training batches on cuda:  29%|██▉       | 31/106 [00:01<00:02, 28.56batch/s]

Training batches on cuda:  33%|███▎      | 35/106 [00:01<00:02, 29.57batch/s]

Training batches on cuda:  37%|███▋      | 39/106 [00:01<00:02, 30.21batch/s]

Training batches on cuda:  41%|████      | 43/106 [00:01<00:02, 30.64batch/s]

Training batches on cuda:  44%|████▍     | 47/106 [00:01<00:01, 30.99batch/s]

Training batches on cuda:  48%|████▊     | 51/106 [00:01<00:01, 31.40batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:01<00:01, 31.04batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:01<00:01, 31.45batch/s]

Training batches on cuda:  59%|█████▉    | 63/106 [00:02<00:01, 31.82batch/s]

Training batches on cuda:  63%|██████▎   | 67/106 [00:02<00:01, 32.04batch/s]

Training batches on cuda:  67%|██████▋   | 71/106 [00:02<00:01, 31.79batch/s]

Training batches on cuda:  71%|███████   | 75/106 [00:02<00:00, 31.96batch/s]

Training batches on cuda:  75%|███████▍  | 79/106 [00:02<00:00, 32.13batch/s]

Training batches on cuda:  78%|███████▊  | 83/106 [00:02<00:00, 32.04batch/s]

Training batches on cuda:  82%|████████▏ | 87/106 [00:02<00:00, 32.19batch/s]

Training batches on cuda:  86%|████████▌ | 91/106 [00:02<00:00, 32.24batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:03<00:00, 32.20batch/s]

Training batches on cuda:  93%|█████████▎| 99/106 [00:03<00:00, 32.17batch/s]

Training batches on cuda:  97%|█████████▋| 103/106 [00:03<00:00, 32.15batch/s]

                                                                              
Training epochs on cuda:  10%|█         | 1/10 [00:07<00:31,  3.53s/epoch, loss=0.000343, prev_loss=nan]
Training epochs on cuda:  20%|██        | 2/10 [00:07<00:28,  3.58s/epoch, loss=0.000343, prev_loss=nan]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:06, 16.37batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:05, 19.01batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:04, 21.65batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:03, 23.90batch/s]

Training batches on cuda:  16%|█▌        | 17/106 [00:00<00:03, 25.41batch/s]

Training batches on cuda:  20%|█▉        | 21/106 [00:00<00:03, 27.07batch/s]

Training batches on cuda:  24%|██▎       | 25/106 [00:00<00:02, 28.06batch/s]

Training batches on cuda:  27%|██▋       | 29/106 [00:00<00:02, 28.95batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:02, 29.48batch/s]

Training batches on cuda:  35%|███▍      | 37/106 [00:01<00:02, 29.98batch/s]

Training batches on cuda:  39%|███▊      | 41/106 [00:01<00:02, 30.48batch/s]

Training batches on cuda:  42%|████▏     | 45/106 [00:01<00:02, 30.43batch/s]

Training batches on cuda:  46%|████▌     | 49/106 [00:01<00:01, 31.00batch/s]

Training batches on cuda:  50%|█████     | 53/106 [00:01<00:01, 28.50batch/s]

Training batches on cuda:  54%|█████▍    | 57/106 [00:01<00:01, 29.26batch/s]

Training batches on cuda:  57%|█████▋    | 60/106 [00:02<00:01, 29.21batch/s]

Training batches on cuda:  60%|██████    | 64/106 [00:02<00:01, 30.18batch/s]

Training batches on cuda:  64%|██████▍   | 68/106 [00:02<00:01, 30.86batch/s]

Training batches on cuda:  68%|██████▊   | 72/106 [00:02<00:01, 31.18batch/s]

Training batches on cuda:  72%|███████▏  | 76/106 [00:02<00:00, 31.25batch/s]

Training batches on cuda:  75%|███████▌  | 80/106 [00:02<00:00, 30.81batch/s]

Training batches on cuda:  79%|███████▉  | 84/106 [00:02<00:00, 30.20batch/s]

Training batches on cuda:  83%|████████▎ | 88/106 [00:02<00:00, 30.17batch/s]

Training batches on cuda:  87%|████████▋ | 92/106 [00:03<00:00, 29.95batch/s]

Training batches on cuda:  90%|████████▉ | 95/106 [00:03<00:00, 29.77batch/s]

Training batches on cuda:  92%|█████████▏| 98/106 [00:03<00:00, 29.67batch/s]

Training batches on cuda:  95%|█████████▌| 101/106 [00:03<00:00, 29.63batch/s]

Training batches on cuda:  98%|█████████▊| 104/106 [00:03<00:00, 29.62batch/s]

                                                                              
Training epochs on cuda:  20%|██        | 2/10 [00:10<00:28,  3.58s/epoch, loss=0.000343, prev_loss=0.000343]
Training epochs on cuda:  30%|███       | 3/10 [00:10<00:25,  3.63s/epoch, loss=0.000343, prev_loss=0.000343]

Training batches on cuda:   0%|          | 0/106 [00:00<?, ?batch/s]

Training batches on cuda:   2%|▏         | 2/106 [00:00<00:05, 19.59batch/s]

Training batches on cuda:   6%|▌         | 6/106 [00:00<00:04, 22.05batch/s]

Training batches on cuda:   9%|▉         | 10/106 [00:00<00:03, 24.10batch/s]

Training batches on cuda:  13%|█▎        | 14/106 [00:00<00:03, 26.16batch/s]

Training batches on cuda:  17%|█▋        | 18/106 [00:00<00:03, 27.50batch/s]

Training batches on cuda:  21%|██        | 22/106 [00:00<00:02, 28.61batch/s]

Training batches on cuda:  25%|██▍       | 26/106 [00:00<00:02, 29.28batch/s]

Training batches on cuda:  28%|██▊       | 30/106 [00:00<00:02, 29.61batch/s]

Training batches on cuda:  31%|███       | 33/106 [00:01<00:02, 29.66batch/s]

Training batches on cuda:  34%|███▍      | 36/106 [00:01<00:02, 29.73batch/s]

Training batches on cuda:  38%|███▊      | 40/106 [00:01<00:02, 29.94batch/s]

Training batches on cuda:  42%|████▏     | 44/106 [00:01<00:02, 30.00batch/s]

Training batches on cuda:  45%|████▌     | 48/106 [00:01<00:01, 29.92batch/s]

Training batches on cuda:  49%|████▉     | 52/106 [00:01<00:01, 29.97batch/s]

Training batches on cuda:  52%|█████▏    | 55/106 [00:01<00:01, 29.88batch/s]

Training batches on cuda:  56%|█████▌    | 59/106 [00:01<00:01, 29.94batch/s]

Training batches on cuda:  58%|█████▊    | 62/106 [00:02<00:01, 29.79batch/s]

Training batches on cuda:  62%|██████▏   | 66/106 [00:02<00:01, 30.08batch/s]

Training batches on cuda:  66%|██████▌   | 70/106 [00:02<00:01, 30.35batch/s]

Training batches on cuda:  70%|██████▉   | 74/106 [00:02<00:01, 30.51batch/s]

Training batches on cuda:  74%|███████▎  | 78/106 [00:02<00:00, 30.80batch/s]

Training batches on cuda:  77%|███████▋  | 82/106 [00:02<00:00, 31.28batch/s]

Training batches on cuda:  81%|████████  | 86/106 [00:02<00:00, 31.40batch/s]

Training batches on cuda:  85%|████████�