Timestamping of events (classic)
Overview
Learn how Engine computes the timestamps of the events in its database by combining the time of reception of the packets sent by Collectors with the individual time information of each event stored inside every packet.
Timestamping in Collector
Collector reacts to events of interest by recording them in memory. Later, it sends the collected events to Engine either periodically or when it has accumulated a sufficient quantity of events. To detect activity in the system where it is installed, Collector employs different techniques, such as intercepting system calls, that allow it to precisely determine the moment at which the event takes place.
This timestamping of the events on the device of the employee is done according to the time elapsed since the last system boot. Therefore, Collector uses the relative time to timestamp events on employees’ devices. The fact that the time used by Collector is relative is not important for computing a precise timestamp in Engine, as you will see below.
Once Collector has gathered enough events, along with their corresponding timestamps, it builds a network packet and sends it to Engine. Right before sending it, Collector sets a timestamp in the packet, again using the local time relative to the system boot. Therefore, in every packet Collector sends we have:
The timestamps of each individual event sent in the packet.
One general timestamp for the packet itself.
Engine uses the difference between the time of the packet and the time of each individual event to compute the global timestamp of the events.
Timestamping in Engine
Once Engine receives a packet from Collector, it records the time of reception using the system time. To compute the global time of events, Engine assumes that the transmission time of Collector packets from employee computers to Engine is negligible. In this way, the absolute time at which Collector sent the packet is considered to be the same absolute time of reception of the packet in Engine. Therefore, the time that Collector set in the packet just before sending it is the local time of the employee machine relative to system boot that is equivalent to Engine time of reception of the packet. To get the occurrence time of each event, Engine subtracts the difference between the local times of the packet and of each event from the reception time, as shown in the figure below:
Note that the events received in Engine may not follow a sequential order. The most common case is when you receive two packets in a short interval of time and each packet comes from a different Collector. It is likely that the two packets have events that overlap in time, but Engine processes all the events of the second packet after those of the first. Engine deals well with this situation. Note also that Engine always inserts events in the past with respect to its current time. This is obvious, because Engine cannot receive events that have not happened yet. However, for events that lie too far in the past, Engine will not be able to update the in-memory database, since it would be too costly. Engine rejects events older than 30 minutes in the past with respect to its present time.
For Collectors in a local network, however, this is a very unlikely case and it often indicates a problem in the device that hosts Collector.
The case of TCP connections
Collector treats TCP connections differently from all other events regarding the setting of their timestamp. All other events have their timestamp set as soon as they begin to do some kind of activity. On the other hand, when the employee device opens a TCP connection to a server, Collector waits for the connection to be established to set the timestamp of the TCP connection event.