nohup: ignoring input The dataframe X: ent1 rel ent2 0 6 17 1973 1 9 17 183 2 9 19 3275 3 9 19 9511 4 9 19 10002 Changing the edge structure from (synset -> synset) to (word -> word) Original test size = 13188392 Reduced test size = 2892340 Train size = 3787 Count of unseen edges = 425432 Count of seen edges = 3787 2021-10-06 19:55:20.389057: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-10-06 19:55:20.422636: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2099990000 Hz 2021-10-06 19:55:20.424988: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x559ccdf378f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-10-06 19:55:20.425042: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-10-06 19:55:20.434874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2021-10-06 19:55:20.749549: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x559ccc4b4190 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-10-06 19:55:20.749606: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2021-10-06 19:55:20.752280: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 19:55:20.752838: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 19:55:20.755946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 19:55:20.758378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 19:55:20.759029: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 19:55:20.762498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 19:55:20.764273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 19:55:20.769798: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 19:55:20.773063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 19:55:20.773127: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 19:55:20.774954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 19:55:20.774975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 19:55:20.774985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 19:55:20.778172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) 2021-10-06 19:55:25.371906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 19:55:25.372016: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 19:55:25.372046: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 19:55:25.372094: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 19:55:25.372119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 19:55:25.372144: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 19:55:25.372168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 19:55:25.372193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 19:55:25.375226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 19:55:25.375263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 19:55:25.375273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 19:55:25.375281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 19:55:25.378369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 0th iteration!#### 2021-10-06 19:59:05.847080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 19:59:05.847322: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 19:59:05.847356: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 19:59:05.847387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 19:59:05.847413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 19:59:05.847450: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 19:59:05.847477: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 19:59:05.847505: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 19:59:05.850547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 19:59:05.850599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 19:59:05.850610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 19:59:05.850628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 19:59:05.853718: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 1th iteration!#### 2021-10-06 20:02:41.165008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:02:41.165135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:02:41.165160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:02:41.165183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:02:41.165220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:02:41.165248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:02:41.165268: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:02:41.165288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:02:41.167707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:02:41.167744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:02:41.167752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:02:41.167758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:02:41.170171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 2th iteration!#### 2021-10-06 20:06:17.277365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:06:17.277539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:06:17.277576: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:06:17.277612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:06:17.277642: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:06:17.277673: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:06:17.277704: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:06:17.277733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:06:17.280794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:06:17.280850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:06:17.280861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:06:17.280870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:06:17.284001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 3th iteration!#### 2021-10-06 20:10:04.744313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:10:04.744492: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:10:04.744529: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:10:04.744563: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:10:04.744612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:10:04.744642: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:10:04.744671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:10:04.744701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:10:04.747781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:10:04.747830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:10:04.747841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:10:04.747850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:10:04.750973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 4th iteration!#### 2021-10-06 20:13:52.534443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:13:52.534594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:13:52.534623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:13:52.534664: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:13:52.534687: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:13:52.534708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:13:52.534731: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:13:52.534752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:13:52.537297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:13:52.537340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:13:52.537349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:13:52.537356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:13:52.540112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 5th iteration!#### 2021-10-06 20:17:28.551542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:17:28.551698: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:17:28.551730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:17:28.551759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:17:28.551784: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:17:28.551829: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:17:28.551855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:17:28.551882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:17:28.554929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:17:28.554976: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:17:28.554987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:17:28.554994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:17:28.558073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 6th iteration!#### 2021-10-06 20:20:54.768595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:20:54.768759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:20:54.768795: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:20:54.768830: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:20:54.768860: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:20:54.768890: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:20:54.768921: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:20:54.768951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:20:54.772174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:20:54.772229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:20:54.772240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:20:54.772248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:20:54.775344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 7th iteration!#### 2021-10-06 20:24:20.170270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:24:20.170438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:24:20.170469: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:24:20.170495: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:24:20.170517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:24:20.170557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:24:20.170581: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:24:20.170604: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:24:20.173166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:24:20.173206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:24:20.173214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:24:20.173221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:24:20.175840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 8th iteration!#### 2021-10-06 20:27:51.221051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:27:51.221195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:27:51.221223: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:27:51.221249: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:27:51.221270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:27:51.221292: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:27:51.221315: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:27:51.221336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:27:51.224026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:27:51.224070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:27:51.224080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:27:51.224086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:27:51.226762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 9th iteration!#### 2021-10-06 20:31:17.686863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:31:17.687026: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:31:17.687059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:31:17.687088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:31:17.687115: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:31:17.687141: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:31:17.687186: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:31:17.687213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:31:17.690872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:31:17.690923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:31:17.690934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:31:17.690942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:31:17.699848: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 10th iteration!#### 2021-10-06 20:34:49.602149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:34:49.602311: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:34:49.602355: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:34:49.602393: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:34:49.602422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:34:49.602451: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:34:49.602482: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:34:49.602510: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:34:49.605535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:34:49.605586: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:34:49.605596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:34:49.605604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:34:49.608692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 11th iteration!#### 2021-10-06 20:38:16.957935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:38:16.958077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:38:16.958110: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:38:16.958138: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:38:16.958164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:38:16.958190: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:38:16.958230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:38:16.958258: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:38:16.961327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:38:16.961379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:38:16.961390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:38:16.961398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:38:16.964494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 12th iteration!#### 2021-10-06 20:41:43.573668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:41:43.573840: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:41:43.573877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:41:43.573913: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:41:43.573942: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:41:43.573972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:41:43.574001: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:41:43.574030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:41:43.577078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:41:43.577129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:41:43.577140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:41:43.577148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:41:43.580248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 13th iteration!#### 2021-10-06 20:45:11.336841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:45:11.337005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:45:11.337044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:45:11.337074: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:45:11.337101: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:45:11.337127: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:45:11.337173: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:45:11.337201: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:45:11.340319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:45:11.340368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:45:11.340379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:45:11.340387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:45:11.343506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 14th iteration!#### 2021-10-06 20:48:36.711967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:48:36.712091: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:48:36.712116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:48:36.712140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:48:36.712160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:48:36.712180: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:48:36.712201: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:48:36.712221: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:48:36.714619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:48:36.714672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:48:36.714680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:48:36.714687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:48:36.717122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 15th iteration!#### 2021-10-06 20:52:03.093424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:52:03.093551: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:52:03.093577: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:52:03.093609: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:52:03.093629: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:52:03.093649: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:52:03.093669: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:52:03.093708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:52:03.096136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:52:03.096176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:52:03.096184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:52:03.096189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:52:03.098637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 16th iteration!#### 2021-10-06 20:55:38.795871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:55:38.796007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:55:38.796037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:55:38.796066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:55:38.796091: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:55:38.796114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:55:38.796140: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:55:38.796164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:55:38.800416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:55:38.800461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:55:38.800470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:55:38.800477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:55:38.803132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 17th iteration!#### 2021-10-06 20:59:09.587411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 20:59:09.587568: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 20:59:09.587600: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 20:59:09.587630: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 20:59:09.587656: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 20:59:09.587683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 20:59:09.587709: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 20:59:09.587753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 20:59:09.590795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 20:59:09.590842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 20:59:09.590852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 20:59:09.590860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 20:59:09.593958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 18th iteration!#### 2021-10-06 21:02:35.589820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:02:35.589979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:02:35.590011: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:02:35.590042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:02:35.590070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:02:35.590096: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:02:35.590124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:02:35.590150: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:02:35.593413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:02:35.593468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:02:35.593479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:02:35.593488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:02:35.596598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 19th iteration!#### 2021-10-06 21:06:00.670242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:06:00.670423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:06:00.670460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:06:00.670489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:06:00.670514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:06:00.670540: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:06:00.670566: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:06:00.670593: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:06:00.673736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:06:00.673785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:06:00.673796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:06:00.673805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:06:00.676940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 20th iteration!#### 2021-10-06 21:09:26.585641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:09:26.585783: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:09:26.585817: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:09:26.585850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:09:26.585877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:09:26.585904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:09:26.585933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:09:26.585959: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:09:26.589008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:09:26.589062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:09:26.589073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:09:26.589082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:09:26.592425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 21th iteration!#### 2021-10-06 21:13:03.228305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:13:03.228456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:13:03.228492: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:13:03.228522: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:13:03.228548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:13:03.228575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:13:03.228600: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:13:03.228628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:13:03.231738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:13:03.231789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:13:03.231800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:13:03.231808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:13:03.234897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 22th iteration!#### 2021-10-06 21:16:29.686952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:16:29.687084: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:16:29.687111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:16:29.687136: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:16:29.687163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:16:29.687184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:16:29.687206: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:16:29.687227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:16:29.689763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:16:29.689804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:16:29.689813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:16:29.689819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:16:29.692453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 23th iteration!#### 2021-10-06 21:19:57.248472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:19:57.248643: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:19:57.248680: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:19:57.248712: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:19:57.248742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:19:57.248772: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:19:57.248801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:19:57.248832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:19:57.251934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:19:57.251986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:19:57.251996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:19:57.252004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:19:57.255128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 24th iteration!#### 2021-10-06 21:23:22.710151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:23:22.710279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:23:22.710308: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:23:22.710334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:23:22.710397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:23:22.710423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:23:22.710448: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:23:22.710472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:23:22.712878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:23:22.712917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:23:22.712925: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:23:22.712931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:23:22.715408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 25th iteration!#### 2021-10-06 21:26:49.025022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:26:49.025176: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:26:49.025208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:26:49.025238: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:26:49.025265: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:26:49.025292: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:26:49.025319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:26:49.025347: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:26:49.028385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:26:49.028452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:26:49.028463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:26:49.028471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:26:49.031583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 26th iteration!#### 2021-10-06 21:30:15.838781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:30:15.838919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:30:15.838945: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:30:15.838970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:30:15.838991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:30:15.839013: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:30:15.839034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:30:15.839053: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:30:15.841768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:30:15.841812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:30:15.841821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:30:15.841828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:30:15.844426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 27th iteration!#### 2021-10-06 21:33:45.979226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: NVIDIA GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545 pciBusID: 0000:89:00.0 2021-10-06 21:33:45.979403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2021-10-06 21:33:45.979440: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2021-10-06 21:33:45.979473: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2021-10-06 21:33:45.979504: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2021-10-06 21:33:45.979535: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2021-10-06 21:33:45.979565: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2021-10-06 21:33:45.979596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-10-06 21:33:45.982638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-10-06 21:33:45.982709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-10-06 21:33:45.982721: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-10-06 21:33:45.982729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-10-06 21:33:45.985949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10321 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:89:00.0, compute capability: 7.5) ####Completed 28th iteration!#### Results: Test mrr = 0.007646983142698449 Test hits_at_1 score = 0.0035090590620868905 Test hits_at_3 score = 0.005893913236689 Test hits_at_10 score = 0.010661544815636364