Temporal Performance & Reliability

Reference Papers
Movie Performance
Benchmarking your System
Understanding Video Timing

NEW! MonkeyLogic User's Forum 

Reference Papers

Data regarding the temporal precision of MonkeyLogic is available in the following publications. The first two detail specific performance characteristics of MonkeyLogic, and the third addresses the general issues encountered and the strategies employed in trying to achieve real-time control in a soft-real-time, interpreted-language operating environment.

	High-Performance Execution of Psychophysical Tasks with Complex Stimuli in MATLAB. Wael F. Asaad, Navaneethan Santhanam, Steve McClellan & David J. Freedman Journal of Neurophysiology, 2013, Vol.109, pp 249-260.
	A flexible software tool for temporally-precise behavioral control in MATLAB. Wael F. Asaad & Emad N. Eskandar Journal of Neuroscience Methods, 2008, Vol. 174, pp. 245-258.
	Achieving behavioral control with millisecond resolution in a high-level programming environment. Wael F. Asaad & Emad N. Eskandar Journal of Neuroscience Methods, 2008, Vol.173, No.2, pp 235-240.

Addendum:

Resolution of certain timing inefficiencies (described in the second reference, in Table 1). Note these are also explained and resolved in the first reference.

1) Calls to toggleobject that ended with updates of the control-screen, to plot or erase symbols reflecting the position of currently active visual stimuli, were noted to incur a large temporal cost (up to 30 ms). This issue has been resolved. The function "exit time" for toggleobject now averages less than 1 ms. The DrawMode:Fast option for toggleobject is now, therefore, unnecessary.

2) The first cycle in any behavioral tracking episode in eyejoytrack was noted to take much longer than subsequent ones due to the plotting of target rings around the relevant objects (second reference, section 5.3), up to nearly 30 ms. This issue has been resolved. Plotting the target rings now takes less than 2 ms, on average, so that the first tracking cycle latency is similar to subsequent cycle latencies when the eye- or joystick trace is updated.

3) The trial exit time (which averaged 10-20 ms) is now subtracted from the inter-trial-interval such that the actual ITI time now better approximates the desired ITI time.

Corrigendum:

In the second reference ("A flexible software tool..."), the example code for picking out the first instance of n consecutive correct trials from an error vector e (page 248) is incorrect. It should read:

f = find([1 logical(e) 1]);
t = min(f(diff(f) > n));

Note that the red "f" in the second line was miscoded as "find" in the published text.

Movie Performance

The ability to present movie stimuli (and translating images & movies) while simultaneously tracking behavior has been added, thanks to Markus Siegel, David Freedman, and Steve McClellan. As one would expect, the time required to update video frames does have a cost on the ability to rapidly sample behavior. Fortunately, this cost is relatively small. For example, shown here are the consecutive cycle latencies (i.e., time between behavioral data samples) while tracking behavior with a movie running (From Asaad, Santhanam, McClellan and Freedman, 2013):

The red lines indicate those cycles in which a video update was performed to advance the frames (and position) of the movies. First, notice that no frames are missed (there is a red line every 10 ms for this case in which the video refresh rate was 100 Hz). Second, note that the latencies on those cycles in which a video update was performed took longer (between 2 and 3 ms) than those cycles without such video updates (~1 ms). This means that there are gaps in the on-line tracking of behavior equal to these times, potentially lasting up to nearly 3 ms. However, there is no temporal slippage; these gaps are simply intervals in which MonkeyLogic is temporarily "blind" to changes in the behavioral signals. The average latency across all cycles was 1.15 ms.

It is not unusual to have skipped frames during the first presentation of a movie in only the first trial, especially when using higher frame rates (this will be indicated by a warning message); however, we have found subsequent movie presentations in the following trials do not produce any skipped frames at rates of at least up to 100 Hz.

For a comparison of what this figure looks like when tracking behavior without concurrently showing movies, refer to Figure 3a of the second reference, above. In that figure, the periodic spikes in latency are less frequent, and due solely to the updating of the control-screen. The first cycle in that figure experiences a prolonged latency (~23 ms) due to updating the control-screen with symbolic representations of the visual stimuli; when presenting movies, this is automatically disabled such that there are no skipped frames (explaining why the longest latency here was under 3 ms).

This test was performed on the same machine used in the references above; much faster (and slower) computers are available, so performance will vary (see benchmarking, below).

Benchmarking your System

There are two ways built-in to MonkeyLogic to help benchmark a particular system (both found in the task panel of the main menu):

• MonkeyLogic Latency Test - This button will become enabled once a Conditions file has been loaded along with its settings. Using these settings, a benchmarking trial will be run to determine how quickly the behavioral signal can be sampled while presenting a static picture and while presenting a movie (the latter must perform video updates that momentarily interrupt behavioral sampling, as shown above). The measured times will vary as a function of the number and type of I/O assignments made (e.g., if no analog inputs are used, cycle latencies will be much shorter), and as a function of the number of analog input boards used (cycle latencies will be longer when two DAQ boards are used; nevertheless, this is preferred for reasons described in the first reference, above). The results will be plotted in a new figure window (as shown below). This figure is useful to determine the actual behavioral-sampling performance of one's system in the context of a real trial at the currently-selected I/O settings.

• Matlab Latency Test - This button simply time-stamps an empty-loop one million times and plots the resulting cycle latencies for each of the three process priorities available for use by MonkeyLogic (i.e., "Normal", "High" and "Highest."). This is useful to determine if there are sporadic high latencies caused by the operating system or background applications. Generally, mean latencies across priority settings will differ by a small amount, whereas maximum latencies will vary by several tenths of a millisecond or more. Shown below is an example of a resulting figure.

Understanding Video Timing

Because video refreshes are periodic events, trying to specify particular time intervals within a behavioral task is subject to quantization according to the video refresh interval. When the overhead of specific function calls is added to the mix, seemingly unexpected timing intervals may result. For example, if you used a video refresh rate of 100 Hz and wanted to display a stimulus for exactly 100 ms with some sort of behavioral tracking in-between, you'll find that the actual interval MonkeyLogic runs is 110 ms (one extra frame is added). The reason is depicted in this diagram:

This diagram explains why specifying a certain timing interval can result in an extra video frame. Here, the user wished to specify a 100 ms time interval, spanning exactly 10 video frames. However, 11 video frames are always displayed. This is caused by very slight delays between the toggleobject and eyejoytrack function calls. Specifically, the delays between the eventmarker logged by toggleobject at time t = 0 ms and eyejoytrack execution at time t = 1 ms is due to overhead in completing and exiting the toggleobject function and overhead entering and starting the eyejoytrack function. At the end of that interval, there is additional overhead exiting the eyejoytrack function and entering the toggleobject function. All of these slight delays are additive, pushing the end of the intended 100 ms interval into the next video frame. The depicted one millisecond delay between functions is used as an example (the actual time will vary depending on the system on which MonkeyLogic is running); but even if this were only a 0.1 millisecond delay, the same result would ensue. So long as the next frame has started by even the smallest amount of time, an extra frame must be added to be able to enact the next toggleobject command.

Importantly, these things hold true if one is using toggleobject with the "eventmarker" option, causing toggleobject to wait until a vertical blank before time-stamping that event to coincide with the actual stimulus onset or offset. If toggleobject were called without the eventmarker option, then there would be no gap (here, for example, between t = -5 and t = 0) and program execution would be allowed to continue immediately after the visual stimuli have been cued, even before the next video refresh actually took place. In that case, immediately calling the separate eventmarker command directly from the timing script would produce a time-stamp just after t = -5, and would not therefore indicate the actual time of visual stimulus onset.

In this example, to achieve the desired interval of exactly 100 ms, the best strategy is to specify an interval that is slightly shorter (approximately by half a video frame usually works). Here, that would mean specifying an interval of ~95 ms. That way, the second toggleobject command is issued sufficiently before the next refresh to queue the visual stimuli to turn off at the desired time (t = 100 ms).