Hello Marcelo,
The vmware infrasturcture is connected to a shared storage : ibm SAN DS3400 with fc
We tought it was the antivirus update that harnessed the resources of all the vm at the same time so we disabled the updates for a short time and the latency is still there.
here is a part of the vmkernel.log warnings , (no errors found)
2013-05-03T09:55:44.743Z cpu23:4119)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2248 microseconds to 131266 microseconds.
2013-05-03T09:55:55.196Z cpu5:4101)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2248 microseconds to 91238 microseconds.
2013-05-03T09:56:11.973Z cpu3:4099)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2249 microseconds to 45141 microseconds.
2013-05-03T10:01:24.200Z cpu13:4109)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2252 microseconds to 73511 microseconds.
2013-05-03T10:01:55.405Z cpu5:4101)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2252 microseconds to 51562 microseconds.
2013-05-03T10:03:21.168Z cpu7:4103)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2253 microseconds to 182468 microseconds.
2013-05-03T10:04:06.312Z cpu2:4098)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2253 microseconds to 48618 microseconds.
2013-05-03T10:04:54.635Z cpu18:4114)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2254 microseconds to 110686 microseconds.
2013-05-03T10:05:38.964Z cpu4:4100)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2254 microseconds to 73191 microseconds.
2013-05-03T10:06:53.519Z cpu4:4100)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2255 microseconds to 52667 microseconds.
2013-05-03T10:11:27.894Z cpu22:80083)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2256 microseconds to 85978 microseconds.
2013-05-03T10:20:18.522Z cpu19:4115)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000005ec50faf607 performance has deteriorated. I/O latency increased from average value of 2258 microseconds to 59557 microseconds.
2013-05-03T14:13:53.222Z cpu17:8803)WARNING: UserLinux: 1640: UNIMPLEMENTED! write-back of mmap regions unsupported
2013-05-03T14:13:53.231Z cpu17:8803)WARNING: UserLinux: 1640: UNIMPLEMENTED! write-back of mmap regions unsupported
2013-05-03T14:17:30.529Z cpu21:8803)WARNING: UserLinux: 1640: UNIMPLEMENTED! write-back of mmap regions unsupported
2013-05-03T14:17:30.538Z cpu21:8803)WARNING: UserLinux: 1640: UNIMPLEMENTED! write-back of mmap regions unsupported
2013-05-03T15:38:27.211Z cpu8:4104)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000003c04efce5fc performance has deteriorated. I/O latency increased from average value of 3333 microseconds to 68646 microseconds.
2013-05-03T20:58:03.128Z cpu8:4104)WARNING: ScsiDeviceIO: 1218: Device naa.60080e50002369e4000003c04efce5fc performance has deteriorated. I/O latency increased from average value of 3348 microseconds to 69797 microseconds.
2013-05-04T12:54:34.289Z cpu1:2929504)WARNING: UserObj: 675: Failed to crossdup fd 8, /dev/ptyp0 type CHAR: Busy
2013-05-04T12:54:34.289Z cpu1:2929504)WARNING: UserObj: 675: Failed to crossdup fd 10, /dev/ptyp0 type CHAR: Busy
2013-05-04T12:54:34.289Z cpu5:2929583)WARNING: UserLinux: 1340: unsupported: (void)
Thanks