Please note this Plugin is still a work in progress and may undergo drastic changes. Plugin is only supported on Windows OS at the moment.
To help you quickly get started using the Plugin check out these video tutorial.
For a Video version, see the 1st video in Video section
Once the Plugin is finalized, we plan to submit it to the MicroManager repository. Then the Plugin will come standard with every MicroManager install. Until then, interest users can ask ASI for a copy of the Plugin and manually install it. There are two parts to installing the Plugin,
mmpluginsfolder on your MicroManager install. On my computer, the full path is this
opencv_2411which contain the java relevant OPENCV 2.4.11 libraries. These libraries are needed by the Plugin to work properly. Copy the folder to
pluginsfolder on your MicroManager install. On my computer, the full path is this
Micro-Manager-1.4\plugins\opencv_2411\opencv-2411.jar Micro-Manager-1.4\plugins\opencv_2411\x64\opencv_java2411.dll Micro-Manager-1.4\plugins\opencv_2411\x86\opencv_java2411.dll Micro-Manager-1.4\mmplugins\asi_xyz_tracking.jar
If the Plugin isn't able to find and load OpenCV libraries, it will stall and crash it and will prevent other Plugins from loading too. You notice this when you go to the Plugin menu and the Plugin menu will not load.
The following version of firmware and software are compatible with each other
|Micro Manager||1.4.23 (nightly builds)|
Important things to note during hardware installation.
start Acqbutton. The Plugin would put the camera in TTL trigger mode, and start dithering the Tunable lens with a Single-axis functions. The Tunable Lens card will output a TTL pulse when it reaches the lower and upper peak of the dither. This triggers the camera to capture images at the lower and upper peak of the Tunable lens.
For a Video version, see the 2nd video in Video section
The Plugin along with a Tunable Lens works like a continuous focus device (like ASI's CRISP). The Plugin does this by dithering the Tunable lens and triggering the camera at the two ends of the dither. This gets the Plugin two images at different focal lengths without actually moving the objective. (You many want the Tunable lens on a 2nd optical path, as the lens is constantly dithering and always out of focus, the images from it may not be useful. It is important for the two optical paths to be parfocal).
By Calculating the focus scores of these two images and comparing them the Plugin is able to estimate the quality of focus at the main optical path, and if it shifts, what direction to move the Z drive to bring to back to focus.
Above graph is ideal , we like a nice slope (either sloping left or right), with it centered as close to the main optical path's best focus as possible. When there is a shift in main optical path's focus, we are picking up change in up and down images focus score, and we know the amount and direction the Z stage needs to be moved to bring back to original focus.
Larger the slope, more signal we have to work with and better the Z tracking performs. A relatively small slope like this could also works. Few things we can try to improve the slope is to reduce the binning, so images have more detail and focus scoring improves. Increase the TL dither amplitude. Or try a different Z focus scoring algorithm.
When Optical path with tunable lens and main Acquisition camera are not parfocal, we may see a slope like this. The Plugin won't be able to maintain focus, because the images on the tunable lens path are so blurry, any focus change on the main path will go unnoticed. We can compensate for the focus offset by using the Tunable lens at an offset with the “Set TLens Offset” button. With the Tunable lens not dithering, view the videos from both the optical paths. Using the focus control knob. Manually adjust the Tunable lens so both video are in focus. Then press the “Set TLens Offset” button. Tunable lens will dither with this offset position as new center. However because of the offset, we got some tunable lens travel.
These graphs are not ideal and the Z tracking won't work. When the delta is flat, it means the focus score of the images are same or very small. This could be when there is no feature to focus on or other issues like insufficient light, or filter blocking the light etc.
When it's a parabola, the Plugin is able to pickup focus change, but isn't able to deduce the direction to move to fix it. You may try using the TL Offset feature to shift the parabola and create a slope that falls on the best focus position.
* If you are getting the ideal graph, we can now proceed to Z Calibration, run it by pressing the “Z Calib” button. This move the Z stages a few microns (distance specified in “Calib step”). And returns to original position. The Plugin picks up the change in focus score and is able to detect slope of the delta to set “Z Pol” or polarity, and also min delta, Z offset, Min focus score settings in “Z” tab. If the Focus scoring algorithm is changed, “Z Calib” routine needs to run again. * Continue to lock by pressing the “Z lock”. Now move the XY stage and see how the Z tracking is working. If the correction are happening slowly, try increasing the “Z Gain” setting in “Z tab”, if the corrections are too frequent and the stage seems to be oscillating, reduce “Z gain”
That is it for get started with Z tracking.
For a Video version, see the 3nd video in Video section
The “Up image” window is the main interface for XY tracking. User can pick features of objects to track by left clicking on them, and enable or disable LOCK (or tracking) with a right click on the mouse.
XY tracking can be done along with Z tracking. If the user only requires XY tracking, then select “Free Running mode” in “Acquisition” tabs.
That is it for get started with XY tracking.
Note: most of the settings are saved between sessions.
1,2,3,4,5) Are Drop Down boxes which list the available XY, Z,Tunable Lens, Camera and TigerComm. User is then able to pick which of these devices the Plugin can and cannot use.
XY, is the XY stage holding the sample Z, is the objective mover or sample mover in Z, it's the focus axis Tunable Lens, this is ASIs Tunable lens that the Plugin dithers to get up and down image. Camera, this cannot the main data acquisition camera, as the Plugin will take total control of this camera. Use a secondary camera. Tiger Comm, this is the controller that is housing the XY, Z stage and Tunable lens.
6) Binning , this is the added level of binning the Plugin does to the image to shrink its size further. Calculating the focus score or pixel shift of the image is processor intensive. This settings give the user option to reduce the processing the time.
7) Up Img Scale, this fraction sets the size of the up image frame. Does not increase processing time.
8) Averaging, The Plugin averages the focus score and pixel shift of the images. This setting lets the user pick the Moving average length.
9) TL Dither Amp, Tunable lens Single-axis functions dither amplitude. Same as the Command:SAA. The focus change produced varies how your system is optically setup, but the biggest contributor is Objective magnification. On a 20x objective, 4000 dither produces 80 microns focus change. However on a 60x objective 4000 units of dither may only produce 8 microns focus change. Also Large dither amplitude limits maximum dither frequency. In our observations, at 4000 units dither, maximum dither frequency we could run at was 30 Hz. reducing the amplitude; let us dither beyond 30 Hz.
10) TL dither Freq (Hz), sets the dither frequency of the Tunable lens. Plugin converts this into a time period and applies it to the Tunable lens with the Command:SAF. Pick Dither frequency, set it at half of the camera frame rate. Say camera has a frame rate of 25 Hz; let the dither frequency at 10 or 12 Hz. One cycle of dither puts out two trigger pulses, so dither frequency should always be half of camera fps.
11) Dither Pattern, sets the shape of Single-axis functions dither. User can pick from a Triangle, Square or Sine pattern.
12) Manually control camera, if the user is using a PointGrey, TheImagingSource or a PCO pixel fly camera, the Plugin can automatically put these camera in TTL trigger mode. However if the user is using a different camera the Plugin won't be able to change the mode. User will need to manually set the camera in external trigger mode, and check this box; Plugin will not try change the camera state.
13) Log Tracker?, when checked Plugin generates a log file whenever XY or Z tracker is in
LOCK state. The file is saved in MicroManager Install folder in Program files. Like
C:\Program Files\Micro-Manager-1.4. This feature doesn't generate more serial communication traffic.
14) Log Name, name of the tracking data log file. Default name is
asi_xyz_tracker_log.txt. If a file exists with the same name, Plugin appends to it, and won't overwrite it.
15) Pressing the button open a “Save File dialog box”, lets user specify the location and name of the Tracking data log.
16) Free Running, when checked, Plugin won't put the camera in TTL mode, and won't try dither the Tunable lens. Suitable when the user only requires XY tracking
This tab holds settings related to the Z Tracker.
1) Z Algorithm, user can select which Algorithm the Plugin uses to generate the focus score. Each algorithm has its strengths and weakness, user is encouraged to experiment. Changing this option requires the Acquisition be restarted for changes to take effect. User might need to rerun
Z Calib. For more info on Algorithms refer Section on Z Focus Algorithms
An approximate idea of processing times, for a 640×480 pixel frame, with Binning set to 1. SharpEdges_mm will take 78 millisec, Tenengrad_mm takes 31 millisec and SharpEdges_CV takes 15 millisec.
2) Min Delta, its minimum error or delta (difference of the UP image focus score and DOWN image focus score) the Plugin needs to see before applying any Z corrections. This setting is meant to avoid tiny movements of the Z drive when image is in focus. Settings is automatically determined by Z Calib routine.
3) Z Gain, gain or multiplier used by the tracker to generate correction. If the correction is too slow, increase this setting. If the correction is too much that it overshoots then starts ringing or oscillating, decrease the settings.
4) Z Pol, Polarity of the error, either +1 or -1. Setting is automatically determined by
Z Calib routine. To determine this manually, put the Z track in Lock state, then if the Z drive moves away from best focus, then polarity needs to be inverted.
5) Z offset, the Z tracker routine uses the Delta in Up and Down images focus score to determine the correction needed when in Lock state. When the user presses the
Z Lock Button the Plugin zeros the current offset by copying it to this setting. User can adjust this setting if they need to slightly adjust the Z Lock position.
6) Min Focus Score, the tracker will look for a minimum focus score before making correction. This is to prevent the tracker from focusing on background. This setting is automatically determined during
Z Calib. To do this manually, move the field of view to just the background. Then hit the
Get Low Focus Score button. The current focus score will copy into this setting. Will need to be repeated if Z Algorithm is changed.
7) Max Z Run Away (um), this is the maximum distance in microns the Plugin is allowed to move the Z drive. This is to prevent run away.
When working with Objectives with small working distance or high NA. Reduce the Run away to avoid crashes.
8) Calib Step (um), lets user set the step size the Z stage is moved and the step size used for “Start Z Sweep” routine too. Lower magnification objectives have a longer depth of field so user may like to have longer step size. However on high magnification objectives have shorter depth of field, so user may want to reduce step size.
When working with Objectives with small working distance or high NA. Reduce the Calib Step size to avoid crashes.
9) Use ROI for Z Focus Score? , when checked while XY tracking is in
Lock state , instead of generating focus score for the entire image, Plugin will only generate focus score for a smaller ROI. This will help the Z Tracker stay locked on the specimen and not the background. User may need to run Z Calib again before Z
10) Serial Hack, when checked Plugin will leverage a new serial command on Tiger Controller to keep serial traffic to a minimum. Will only work on ASI systems with updated firmware (3.18+). If using a 3rd party system or older ASI firmware, leave unchecked.
This tab holds settings related to the Z Tracker.
1) XY Algorithm, user can select which Algorithm the Plugin uses to calculate pixel shift. Each algorithms has its strengths and weakness, user is encouraged to experiment. Changes to this option require XY tracker state to return to
IDLE for changes to take effect. Redoing XY Calibration isn't required. For more info on Algorithms refer Section on XY Tracking Algorithms
For an approximate idea of speeds, for a 640×480 pixel frame, with Binning set to 1. Meanshift will take less than 1 millisecond, LK takes 7 milliseconds and PhaseCorr takes 50 milliseconds.
2) XX, XY, YX, YY are coefficients used by the XY Tracker to figure out the orientation of the camera to the XY stage. This setting is automatically determined by the XY Calibration routine, which is initiated by pressing the
Start XY Calib.
3) XY Gain, gain or multiplier used by the tracker to generate correction. If the correction is too slow, increase this setting. If the correction is too much that it overshoots then starts ringing or oscillating, decrease the settings.
4) XY Error check, applies only to LK Algorithm. When checked, Plugin runs the Algorithm twice (once with corners of frame 1 to get new location of corners in frame 2, then again with corners in frame 2 to get location of old corners in frame 1). This helps remove any false pixel shift and error.
5) XY Track Radius applies mostly to LK and Meanshift Algorithm. These Algorithms need user to specify a feature to track. This setting specifies the distance from the mouse click the Plugin should consider feature to track.
6) Threshold for Meanshift, Meanshift Algorithm works on Binary image, this setting sets the Threshold when generating the binary image. Can be adjusted in real-time.
7) Color Margin, used by the ColorTracker and ColorMeanShift routine, its sets the color tolerance
8) Erode, used by the ColorTracker, ColorMeanShift, MotionMeanShift and MotionTracker routine. Once the binary image is generated, Plugin applies the Erode routine to reduce region of pixels left, it helps in reducing the noise and protrusions.
9) Dilate, used by the ColorTracker, ColorMeanShift, MotionMeanShift and MotionTracker routine. Is the converse of erode; it expands pixel regions left, smooth out concavities. Dilate is applied after erode.
10) Brightfield? , Meanshift Algorithm works on Binary image with a dark background. Brightfield images have a bright background and will need to be inverted, while fluorescent microscopy images do not. Enabling this setting inverts the intensity of the images before making it binary.
11) Max XY Run Away sets the maximum distance the Plugin is allowed to move the XY stage. When the Plugin reaches the maximum distance of travel, it cancels the lock and returns to
READY state. Helps prevent runaway state.
12) Move to Center, when checked, when XY tracker enters LOCK state, tracker will try to move the tracked feature to center of Up image frame.
13) X & Y offset, work with the “Move to Center” feature, lets user input a pixel offset from the center of up image frame they would like the tracking feature to be.
1) Test OpenCV, run a snippet of OPENCV code, used in verifying if OPENCV libraries are installed and working properly.
2) Serial, lets user type in a serial command to send to the “Tiger Comm device”. Pressing the return key sends the command
3) SEND, pressing the send button sends the command in “Serial” textbox to Tiger device. Output is printed in textbox#6
4, 5) Copy, Clear, Copies or clears the contents of textbox 6.
6) Textbox prints the results from the “Start Z Sweep”, “Test OpenCV”, “XY Calib”, and replies to the serial commands.
Results pane displays read only data about the state of the system
1, 2) Up, Down, displays the focus score of the Up and Down image
3) Delta, displays the Difference between the Up and down image's focus score and accounts for the Z Offset too.
Delta = (Up Image Score - Down Image Score)-Z Offset
4, 5, 6) X Corr, Y Corr, Z Corr shows the vector correction being applied to XY and Z stage.
7,8,9) X Pos,Y Pos ,Z Pos, shows the current X,Y and Z drive position in um. Only updates when XY or Z Tracker is in LOCK state.
10, 11) XY Status Status, shows the state of the XY and Z tracker, such as IDLE, READY, LOCK etc.
12, 13) Process Time (ms), shows how much time the Plugin is taking processing the frame. XY indicates the pixel shift processing time; Z shows the focus scoring time.
1) Halt, Stops XY and Z stage from moving and puts all Tracker to idle state.
2) Start Acq, Starts Acquisition of images. Puts Camera in TTL mode, and sets Tunable Lens to dither with TTL out. Button then changes state to
Stop Acq. When pressed shuts down acquisition and return all Trackers to idle state and stop any XYZ stage travel.
3) Get Low Focus Score, copies the current Focus score into the
Min Focus Score setting in the Z setting Tab. Z Tracker will not do any corrections if Focus score drops to this level. This helps the Plugin focusing on the background.
4) Get Z offset, saves the current delta
(Up Image Score - Down Image Score) into the
Z offset setting in Z tab.
5) Z Lock, when pressed Plugin copies the current Delta into
Z offset settings (same as “Get Z offset” button), and puts the Z Tracker in LOCK state. If Z Tracker is already in LOCK state, then it returns it to IDLE state. Tracker will start applying corrections to Z drive.
6) Move Z Back, Plugin moves Z drive back to the position before LOCK state. Helps recover from a runaway state.
7) Z Calib performs a series of moves that helps it find various settings like Min Delta, Z pol and Min Focus score. User will need to run this when Z Algorithm is changed.
8) Toggle XY Ready, moves the XY tracker from IDLE state to READY state, or back to IDLE state if already in READY state. In READY state the XY Tracker is active but not locked, it starts comparing the UP image frames to determine the pixel shift. LEFT clicking a spot on the UP Image will also put the XY tracker in READY state. LK and Meanshift algorithms need features to track. If “Toggle XY Ready” button is clicked instead of the LEFT licking on UP Image, Plugin will default to using the Center of image as feature to track.
9) Toggle XY Lock, moves the XY tracker from READY state to LOCK state, or back to READY state if already in LOCK state. Tracker will start applying correction to XY stage. RIGHT Clicking a spot on the UP Image will also put the XY tracker in LOCK state, and READY state if already in LOCK state.
10) Start XY Calib, performs a series of moves that help the Plugin determine the orientation and Imaging camera to that of the stage. This routine will automatically set the XX, XY, YX, YY settings in the XY settings tab. Only need to be performed if a new camera is installed or Objective is changed.
11) XYZ Lock Toggle, a quick macro buttons, puts both the XY and Z tracker in LOCK state. Same function as “Z Lock” and “Toggle XY Lock” button.
12) Set TLens Offset, sometimes the optical path with the tunable lens and main optical path with the imaging camera may not be parfocal. In this case the Tunable lens can be adjusted to bring both paths into focus. This buttons saves the setting, so when the Plugin dithers the Tunable lens. It dithers at this offset position.
13) Start Z Sweep, does a series of twenty Z moves with 5um set size, and prints the UP, DOWN image focus score, and delta into a test box at the end of each move. It also generates a graph like the one shown below. It helps troubleshoot and adjust the Z tracker. The step size for the Z sweep can be adjusted with “Calib Step” setting in “Z tab”
14) Save, saves all the Plugin settings in an xml file. When clicked, user is shown a file dialog box, user can select the save file name and location where it is saved it. Note: Plugin automatically saves and retains settings between sessions. This Buttons lets you copy these settings in a XML file
15) Load, lets user load settings from a previously saved XML file. User is shown a file dialog box, user can then select the save file to load from. Plugin will then quit, and when it is restarted, it will load with settings from the save file.
16) Manual, opens the default web browser on the PC and navigates it to this webpage.
Below is a list of Focus scoring algorithms available in the Plugin.
Provided by micro-manager; a slow routine. It is able to score very blurry images. Is sensitive to intensity variation.
Same as above but implemented with OpenCV library, routine is one of the quickest. However it is NOT able to score blurry images very well. Is sensitive to intensity variations.
Provided by micro-manager, routine is medium-quick, notably not sensitive to intensity variations.
Provided by micro-manager, routine is one of the quickest. Is suitable for most cases.
The “Start Z sweep” routine is a good way to compare the focus routines.
Below is a list of XY tracking algorithms provided in the Plugin:
OpenCV's implementation of the Phase Correlation routine. This routine takes in two consecutive images and outputs relative movement between them. It's the slowest routine in the pack as it processes the entire frame.
When XY tracker is in idle state, left clicking on the up image or clicking the “Toggle XY Ready” button starts the tracker and puts it in “Ready” mode. This routine doesn't need a reference pixel to begin tracking.
OpenCV'sLucas Kanade Optical flow routine. This routine outputs the relative movements of a set of pixels between two consecutive images. These pixels are called corners (pixels that have intensity variation in more than one direction). By selecting corner pixels around an interested object, this routine is able to track the position of this object in a frame. This routine is medium-quick, but isn't too robust.
Tracker is activated when user left clicks on the up image or clicks the “Toggle XY Ready” button. When activated the routine takes the current frame, make a circular mask of radius defined by the “XY Track Radius” setting around a reference pixel. If the user left clicked, it uses the mouse position as the reference pixel, and if “Toggle XY Ready”, uses the center of video as the reference pixel.
Routine runs OpenCV's GoodFeaturestoTrack on this sub image, which returns all the pixels that have corners. Routine will use these pixels to measure relative moments between video frames. User can track an object in a video frame, by setting the “XY Track Radius” to the size of this object, and then left click on the object, pixels about this object will be selected to track, and the tracker will follow this object.
At any time, even in Lock mode, user can add more pixels to track by left clicking on the Up image.
OpenCV's MeanShift routine. This routine works on binary images. User defines a rectangle, routine finds the center of mass in the rectangle, and shifts the center to it. Then on the next consecutive image, say the object that was inside the rectangle moved, routine calculates the center of mass and moves the rectangle to the new center. By seeing the change in the rectangle's position, we can get the relative movement between the two images. This routine is one of the quickest.
Tracker is activated when user left clicks on the up image or clicks the “Toggle XY Ready” button. When activated the routine takes the current frame, turns it into a binary image with the threshold value in “Threshold for Meanshift” setting (This setting can be adjust on the fly). The routine then generates a square with double the width given in “XY Track radius” settings with center around the reference pixel. If the user left clicked, it uses the mouse position as the reference pixel, and if “Toggle XY Ready”, uses the center of video as the reference pixel.
The routine then calculates the center of mass of the binary image within the square, and shifts the center of the square to that new center. It repeats this process on the same image a few times. If the user is interested in tracking an object, they can define the “XY Track Radius” to be slightly bigger than the object. Click near or on the object. The routine will generate a square and centers it on the object. Now we know where the object is in the current frame. On the next frame, if the object moved, the square is re-centered on the object again, and we now know the object's new position.
User can change the object they like to track anytime (even in lock mode) by Left clicking on a different object. If the routine loses the object, when the object is no longer is within the Square. If the object moves back into the square, it'll continue to track it, it can recover on its own making it robust.
Similar to the above Meanshift routine, uses a binary image to track too, instead of generating it with a Threshold value, it uses color.
When user selects a reference pixel (mouse position when user left clicks, or center of the video when they use the “Toggle XY Ready” button). The Routine registers the color of the pixel. Then it generates a lower and upper bound of color with help of the “Color Margin” settings. Routine then filters out every color in the frame that is not the reference pixel color or falls within the upper and lower bound. Generates a binary from this image, applies erode and dilate effect (amount given by the Erode and Dilate setting, setting it to “0”, doesn't apply that affect) Then generates a square and centers it on the reference pixel. From here the operation is very similar to that of Meanshift.
The added color processing doesn't increase processing time by much. This routine offers the same robustness and flexibility as regular Meanshift .
Like the ColorMeanshift routine, this too uses color of the reference pixel and filters out every color except that in the image. Applies erode and dilate affects too.
The Binary image that is now generated has these white objects. Instead of doing Meanshift, This routine now calls in OpenCv's Find Contours feature to scan the entire image and find all the Object. It then returns the location of the biggest object to the XY tracker.
This routine may have limited usage but it has some interesting features. Since it is looking at the entire image, it's less likely to lose the object. If it does lose the object, as soon as the object reappears in the frame, the routine will resume tracking. However if a bigger object enters the frame, the routine will lock on the new object and start tracking that.
Processing contours is slower than Meanshift, so this is a medium slow routine.
This routine begins by performing a Differential operation on two consecutive image frames from a video. The output of this operation is a new image with just the pixels that have changed between the two frames. Static objects and background is removed. This image is turned into a binary, and then Meanshift routine is performed to get the position of the moving object.
When user selects a reference pixel (mouse position when user left clicks, or center of the “Up video” if the user clicks the “Toggle XY Ready” button). The Routine is activated and performs the Differential operation between the current and next image, it draws the Mean Shift rectangle with the reference pixel at center with the width prescribed in the “XY Track Radius” setting. The operation that follows is similar to that of MeanShift routine.
This routine is as fast as the other Mean Shift routines.
Note, once the XY tracker is enabled and locked, sample is moved by the stage, static objects will also appear to move. This routine may have limited use.
Like the MotionMeanShift this routine also performs a Differential operation on two consecutive image frames from a video. The output of this operation is a new image with just the pixels that have changed between the two frames. Static objects and background is removed.
The image is turned into a binary, and instead of performing MeanShift, this routine instead calls in OpenCv's Find Contours feature to scan the entire image and find all the Object. It then finds the biggest object and gets its position.
This routine can also be activated with Left Click or “Toggle XY Ready” button. However it doesn't need a reference pixel. It scans the frame, and tracks the biggest moving object.
Note, once the XY tracker is enabled and locked, sample is moved by the stage, static objects will also appear to move. This routine may have limited use.