# Applied Scientific Instrumentation

### Products

 Custom Optics diSPIM FTP FW1000: Filter Wheel MPPI MS2000, MFC2000 and RM2000 Controller Piezo Systems RAMM, MIM and VTS Portal Serial Command Software TG-1000/Tiger Controller Tracking & Stabilization XY Stage Z-Drive and Focus Stage

### Tags

asi_xyz_tracker_plugin

# ASIs XYZ Tracker

ASI's XYZ Tracker is a MicroManager Plugin, That uses Tiger Controller , XY stage, Z focus drive and a Tunable Lens to keep a sample in view for extended period.

Please note this Plugin is still a work in progress and may undergo drastic changes. Plugin is only supported on Windows OS at the moment.

## Video

To help you quickly get started using the Plugin check out these video tutorial.

## Installation

### Software

For a Video version, see the 1st video in Video section

Once the Plugin is finalized, we plan to submit it to the MicroManager repository. Then the Plugin will come standard with every MicroManager install. Until then, interest users can ask ASI for a copy of the Plugin and manually install it. There are two parts to installing the Plugin,

2. Copy the Plugin jar asi_xyz_tracking.jar to the mmplugins folder on your MicroManager install. On my computer, the full path is this C:\Program Files\Micro-Manager-1.4\mmplugins
3. Along with the Plugin jar, ASI would have also sent you a folder called opencv_2411 which contain the java relevant OPENCV 2.4.11 libraries. These libraries are needed by the Plugin to work properly. Copy the folder to plugins folder on your MicroManager install. On my computer, the full path is this C:\Program Files\Micro-Manager-1.4\plugins
4. Your Plugin is now installed
Micro-Manager-1.4\plugins\opencv_2411\opencv-2411.jar
Micro-Manager-1.4\plugins\opencv_2411\x64\opencv_java2411.dll
Micro-Manager-1.4\plugins\opencv_2411\x86\opencv_java2411.dll
Micro-Manager-1.4\mmplugins\asi_xyz_tracking.jar

If the Plugin isn't able to find and load OpenCV libraries, it will stall and crash it and will prevent other Plugins from loading too. You notice this when you go to the Plugin menu and the Plugin menu will not load.

#### Compatibility

The following version of firmware and software are compatible with each other

 Tracker Plugin 0.3 3.18 2.4.11 1.4.23 (nightly builds)

### Hardware

Important things to note during hardware installation.

• Tunable lens should be put on a different optical path than the main image acquisition path because it is constantly dithering and produces blurry images.
• The two optical paths need to be parfocal and have centered field of view as possible. Or the Plugin which only looks at the Tunable lens path (or tracking path) when it makes its adjustments won't help the main optical path focus or hold the feature in position.
• The TTL out of the Tunable lens card should be connected to the camera on the Tunable lens optical path.
• Rest of the connections is typical. For info on RAMM assembly refer to RAMM, MIM and VTS
Connections

## Operation in Brief

• User begins by Selecting the Devices the Plugin has control over by picking XY stage, Z stage , Tunable Lens , Camera and Tiger Comm in the “Acquisition” Tab.
• User would start the Plugin by pressing the start Acq button. The Plugin would put the camera in TTL trigger mode, and start dithering the Tunable lens with a Single-axis functions. The Tunable Lens card will output a TTL pulse when it reaches the lower and upper peak of the dither. This triggers the camera to capture images at the lower and upper peak of the Tunable lens.
• The Plugin asynchronous collects these images from the camera and sorts and displays the images.
• The Plugin then calculates the focus score of these images, and figures out the direction and amount of correction needed to keep sample in focus.
• For XY tracking, the Plugin just looks at the images coming from the upper peak, compares the current image with the previous image and calculates relative movement between the two frames or objects within a frame
• Plugin then sends the correction command for XY and Z to Tiger controller to bring the sample back into view and in focus.

## Getting started with Z Tracking

For a Video version, see the 2nd video in Video section

The Plugin along with a Tunable Lens works like a continuous focus device (like ASI's CRISP). The Plugin does this by dithering the Tunable lens and triggering the camera at the two ends of the dither. This gets the Plugin two images at different focal lengths without actually moving the objective. (You many want the Tunable lens on a 2nd optical path, as the lens is constantly dithering and always out of focus, the images from it may not be useful. It is important for the two optical paths to be parfocal).

By Calculating the focus scores of these two images and comparing them the Plugin is able to estimate the quality of focus at the main optical path, and if it shifts, what direction to move the Z drive to bring to back to focus.

• Adjusting settings in “Acquisition” Tab.
• Begins by Selecting the Devices the Plugin has control over by picking XY stage, Z stage , Tunable Lens , Camera and Tiger Comm .
• Next Select Binning, default focus scoring routine “Volath” is pretty fast; even it chokes on large images. Start with a high binning; higher the binning faster is the processing time. Too much binning, the image loses its detail, and the focus scoring won't work so well.
• Then select the TL Dither Amp, maximum setting is 4000. Unless you have good reasons leave it at 4000. This is the amplitude tunable lens is dithered at, larger the amplitude, more focus change is produced. The focus change produced varies how your system is optically setup, but the biggest contributor is Objective magnification. On a 20x objective, 4000 dither produces 80 microns focus change. However on a 60x objective 4000 units of dither may only produce 8 microns focus change. Also Large dither amplitude limits maximum dither frequency. In observations, at 4000 units dither, maximum dither frequency we could run at was 30 Hz. reducing the amplitude; let us dither beyond 30 Hz.
• Pick Dither frequency, set it at half of the camera frame rate. Say camera has a frame rate of 25 Hz; set the dither frequency at 10 or 12 Hz. One cycle of dither puts out two trigger pulses, so dither frequency should always be half of camera fps.
• Manually control Camera, the Plugin supports PointGrey, PCO and TheImagingSource camera, i.e. it's able to automatically put these camera in external trigger mode, then pulses from Tunable Lens card will trigger the camera. If you are not using these cameras, then select this box, and manually set the camera in external mode.
• Leave “Free Running” unchecked or Z tracking won't work
• Next switch to the “Z” tab. If you have no preference for which Z Algorithm is used, pick Volath, should give good results in most cases.
• Set “Max Z Run Away”, if you have an objective with small working distance, reduce this. This is the maximum distance the Plugin is allowed to move the objective in Lock state. This setting will prevent objective from crashing into the sample.
• Same with “Calib Step”, default is 5um; this is used during “Z Calib” and “Start Z sweep”. Start Z sweep does 20 steps with that step size. So 5um results in -50um to +50um travel. If you have an objective with small working distance, reduce this.
• Leave “Use ROI…” unchecked, and “Serial Hack” checked.
• We can now begin acquisition by clicking “Start Acq” button. If everything worked correctly, you'll see two windows open, one called “Up img” and another “Down img”. These images will look blurry, focus score is calculated for these images using “Volath” or other algorithm you picked, and the result is displayed in the “Results” section. Time it took to focus score the image is shown under “Process time”, ideally we like this to be less than 20millisec. If it is a big image, try increasing binning, or picking another Z algorithm. This will require Acquisition to be restarted, so press “Stop Acq” and “start acq” again.
• Next manually focus the image out of your main optical path camera by moving the Z drive with a knob, if you like press zero button on the joystick box.
• Let's test the system by running the “Start Z Sweep” routine. Plugin then scans thru the sample with Z stage in 20 steps with the step size specified in “Calib Step (um)”. Say the Z position was 0um, and step size was 5um. Plugin moves the Z stage from -50um to +50um in 20 steps, of 5ums each. It records the Focus score of the up image and down image, and computes a delta of these scores and graphs them. A graph like the one below is generated.
Ideal Graph

Above graph is ideal , we like a nice slope (either sloping left or right), with it centered as close to the main optical path's best focus as possible. When there is a shift in main optical path's focus, we are picking up change in up and down images focus score, and we know the amount and direction the Z stage needs to be moved to bring back to original focus.

Ok, but slope is small

Larger the slope, more signal we have to work with and better the Z tracking performs. A relatively small slope like this could also works. Few things we can try to improve the slope is to reduce the binning, so images have more detail and focus scoring improves. Increase the TL dither amplitude. Or try a different Z focus scoring algorithm.

Graph when Main path and Tunable lens path aren't parfocal

When Optical path with tunable lens and main Acquisition camera are not parfocal, we may see a slope like this. The Plugin won't be able to maintain focus, because the images on the tunable lens path are so blurry, any focus change on the main path will go unnoticed. We can compensate for the focus offset by using the Tunable lens at an offset with the “Set TLens Offset” button. With the Tunable lens not dithering, view the videos from both the optical paths. Using the focus control knob. Manually adjust the Tunable lens so both video are in focus. Then press the “Set TLens Offset” button. Tunable lens will dither with this offset position as new center. However because of the offset, we got some tunable lens travel.

These graphs are not ideal and the Z tracking won't work. When the delta is flat, it means the focus score of the images are same or very small. This could be when there is no feature to focus on or other issues like insufficient light, or filter blocking the light etc.

When it's a parabola, the Plugin is able to pickup focus change, but isn't able to deduce the direction to move to fix it. You may try using the TL Offset feature to shift the parabola and create a slope that falls on the best focus position.

* If you are getting the ideal graph, we can now proceed to Z Calibration, run it by pressing the “Z Calib” button. This move the Z stages a few microns (distance specified in “Calib step”). And returns to original position. The Plugin picks up the change in focus score and is able to detect slope of the delta to set “Z Pol” or polarity, and also min delta, Z offset, Min focus score settings in “Z” tab. If the Focus scoring algorithm is changed, “Z Calib” routine needs to run again. * Continue to lock by pressing the “Z lock”. Now move the XY stage and see how the Z tracking is working. If the correction are happening slowly, try increasing the “Z Gain” setting in “Z tab”, if the corrections are too frequent and the stage seems to be oscillating, reduce “Z gain”

That is it for get started with Z tracking.

## Getting started with XY Tracking

For a Video version, see the 3nd video in Video section

The “Up image” window is the main interface for XY tracking. User can pick features of objects to track by left clicking on them, and enable or disable LOCK (or tracking) with a right click on the mouse.

XY tracking can be done along with Z tracking. If the user only requires XY tracking, then select “Free Running mode” in “Acquisition” tabs.

• User begins by Selecting the Devices the Plugin has control over by picking XY stage, Z stage , Tunable Lens , Camera and Tiger Comm in the “Acquisition” Tab.
• Next Select Binning, smaller the image faster the processing time.
• Pick Dither frequency, set it at half of the camera frame rate. Say camera has a frame rate of 25 Hz; let the dither frequency at 10 or 12 Hz. One cycle of dither puts out two trigger pulses, so dither frequency should always be half of camera fps.
• Manually control Camera, the Plugin supports PointGrey, PCO and TheImagingSource camera, i.e. it's able to automatically put these camera in external input mode, where they'll be triggered by plus from Tunable Lens card. If you're not using these cameras, then select this box, and manually set the camera in external mode.
• “Free Running” is up to the user, if they just need XY tracking, select Free Running.
• Next go to the “XY” tab and pick a “XY Algorithm”, detailed info on each algorithm are here.
• A simple tracker to use during XY calibration is “PhaseCorr”.
• Set “Max XY Runaway”, this is the maximum distance the Plugin is allowed to move the stage.
• Leave “Move to center” checked
• Set “XY Calib step size”, if you have a high magnification objective with small field of you, you may want to reduce it. On the flip size a small magnification objective, you may want to increase it.
• Now start Acquisition by clicking on “Start Acq” button. If free running is disabled, user will see two image windows, “up image” and “down image”. If free running is enabled, then just one image window “up image” opens.
• XY tracker begins in idle mode and it isn't doing any processing. To put XY tracker in ready mode, user can either click the “Toggle XY Ready” button or left click on the “Up image” window. Now you begin to see XY processing time. If it's too high we may want to increase binning or select a different XY tracking algorithm.
• To select another algorithm, click the “Toggle XY ready” button again to put YX tracker in IDLE mode. Unlike Z, Acquisition doesn't have to be restarted.
• PhaseCorr is a good routine for XY calibration, it's simple to use as it looks at the entire frame and no feature has to be selected prior. However it's slow and not ideal during tracking.
• With the XY tracker in READY mode, we can run XY calibration. Start it by pressing the “Start XY Calib” button. This routine moves the XY Stage in X then in Y with a step size specified in “XY Calib step size”, records the shift in frame and is able to detect the orientation of the Tracking camera to the XY stage. It then populates the XX, XY, YX, YY settings in XY tab.
• Unless the tracking camera is moved to a different port, this routine doesn't have to be run again, even if the user changes the XY tracking algorithm.
• Plugin is now ready to lock. User can either right click anywhere on the “Up image” window, or click the “Toggle XY Lock” button.
• When in LOCK state, Plugin will try to keep the feature selected or frame as centered as possible by applying correction. Another right click or clicking the “Toggle XY lock” bring XY tracker to Ready state. Object is still being tracked, but no correction is applied.
• In LOCK state, if the correction is too small or slow, increase the “XY Gain”, and on the flip side, if the corrections are too frequent, or the stage seems to run away, reduce the “XY Gain”. This can be changed on the fly even in LOCK state.

That is it for get started with XY tracking.

## Description of Plugin Controls

Note: most of the settings are saved between sessions.

### Acquisition Settings Tab

1,2,3,4,5) Are Drop Down boxes which list the available XY, Z,Tunable Lens, Camera and TigerComm. User is then able to pick which of these devices the Plugin can and cannot use.

XY, is the XY stage holding the sample Z, is the objective mover or sample mover in Z, it's the focus axis Tunable Lens, this is ASIs Tunable lens that the Plugin dithers to get up and down image. Camera, this cannot the main data acquisition camera, as the Plugin will take total control of this camera. Use a secondary camera. Tiger Comm, this is the controller that is housing the XY, Z stage and Tunable lens.

6) Binning , this is the added level of binning the Plugin does to the image to shrink its size further. Calculating the focus score or pixel shift of the image is processor intensive. This settings give the user option to reduce the processing the time.

7) Up Img Scale, this fraction sets the size of the up image frame. Does not increase processing time.

8) Averaging, The Plugin averages the focus score and pixel shift of the images. This setting lets the user pick the Moving average length.

9) TL Dither Amp, Tunable lens Single-axis functions dither amplitude. Same as the Command:SAA. The focus change produced varies how your system is optically setup, but the biggest contributor is Objective magnification. On a 20x objective, 4000 dither produces 80 microns focus change. However on a 60x objective 4000 units of dither may only produce 8 microns focus change. Also Large dither amplitude limits maximum dither frequency. In our observations, at 4000 units dither, maximum dither frequency we could run at was 30 Hz. reducing the amplitude; let us dither beyond 30 Hz.

10) TL dither Freq (Hz), sets the dither frequency of the Tunable lens. Plugin converts this into a time period and applies it to the Tunable lens with the Command:SAF. Pick Dither frequency, set it at half of the camera frame rate. Say camera has a frame rate of 25 Hz; let the dither frequency at 10 or 12 Hz. One cycle of dither puts out two trigger pulses, so dither frequency should always be half of camera fps.

11) Dither Pattern, sets the shape of Single-axis functions dither. User can pick from a Triangle, Square or Sine pattern.

12) Manually control camera, if the user is using a PointGrey, TheImagingSource or a PCO pixel fly camera, the Plugin can automatically put these camera in TTL trigger mode. However if the user is using a different camera the Plugin won't be able to change the mode. User will need to manually set the camera in external trigger mode, and check this box; Plugin will not try change the camera state.

13) Log Tracker?, when checked Plugin generates a log file whenever XY or Z tracker is in LOCK state. The file is saved in MicroManager Install folder in Program files. Like C:\Program Files\Micro-Manager-1.4. This feature doesn't generate more serial communication traffic.

14) Log Name, name of the tracking data log file. Default name is asi_xyz_tracker_log.txt. If a file exists with the same name, Plugin appends to it, and won't overwrite it.

15) Pressing the button open a “Save File dialog box”, lets user specify the location and name of the Tracking data log.

16) Free Running, when checked, Plugin won't put the camera in TTL mode, and won't try dither the Tunable lens. Suitable when the user only requires XY tracking

### Z Settings Tab

This tab holds settings related to the Z Tracker.

1) Z Algorithm, user can select which Algorithm the Plugin uses to generate the focus score. Each algorithm has its strengths and weakness, user is encouraged to experiment. Changing this option requires the Acquisition be restarted for changes to take effect. User might need to rerun Z Calib. For more info on Algorithms refer Section on Z Focus Algorithms

An approximate idea of processing times, for a 640×480 pixel frame, with Binning set to 1. SharpEdges_mm will take 78 millisec, Tenengrad_mm takes 31 millisec and SharpEdges_CV takes 15 millisec.

2) Min Delta, its minimum error or delta (difference of the UP image focus score and DOWN image focus score) the Plugin needs to see before applying any Z corrections. This setting is meant to avoid tiny movements of the Z drive when image is in focus. Settings is automatically determined by Z Calib routine.

3) Z Gain, gain or multiplier used by the tracker to generate correction. If the correction is too slow, increase this setting. If the correction is too much that it overshoots then starts ringing or oscillating, decrease the settings.

4) Z Pol, Polarity of the error, either +1 or -1. Setting is automatically determined by Z Calib routine. To determine this manually, put the Z track in Lock state, then if the Z drive moves away from best focus, then polarity needs to be inverted.

5) Z offset, the Z tracker routine uses the Delta in Up and Down images focus score to determine the correction needed when in Lock state. When the user presses the Z Lock Button the Plugin zeros the current offset by copying it to this setting. User can adjust this setting if they need to slightly adjust the Z Lock position.

6) Min Focus Score, the tracker will look for a minimum focus score before making correction. This is to prevent the tracker from focusing on background. This setting is automatically determined during Z Calib. To do this manually, move the field of view to just the background. Then hit the Get Low Focus Score button. The current focus score will copy into this setting. Will need to be repeated if Z Algorithm is changed.

7) Max Z Run Away (um), this is the maximum distance in microns the Plugin is allowed to move the Z drive. This is to prevent run away.

When working with Objectives with small working distance or high NA. Reduce the Run away to avoid crashes.

8) Calib Step (um), lets user set the step size the Z stage is moved and the step size used for “Start Z Sweep” routine too. Lower magnification objectives have a longer depth of field so user may like to have longer step size. However on high magnification objectives have shorter depth of field, so user may want to reduce step size.

When working with Objectives with small working distance or high NA. Reduce the Calib Step size to avoid crashes.

9) Use ROI for Z Focus Score? , when checked while XY tracking is in Ready or Lock state , instead of generating focus score for the entire image, Plugin will only generate focus score for a smaller ROI. This will help the Z Tracker stay locked on the specimen and not the background. User may need to run Z Calib again before Z Lock

10) Serial Hack, when checked Plugin will leverage a new serial command on Tiger Controller to keep serial traffic to a minimum. Will only work on ASI systems with updated firmware (3.18+). If using a 3rd party system or older ASI firmware, leave unchecked.

### XY Settings Tab

This tab holds settings related to the Z Tracker.

1) XY Algorithm, user can select which Algorithm the Plugin uses to calculate pixel shift. Each algorithms has its strengths and weakness, user is encouraged to experiment. Changes to this option require XY tracker state to return to IDLE for changes to take effect. Redoing XY Calibration isn't required. For more info on Algorithms refer Section on XY Tracking Algorithms

For an approximate idea of speeds, for a 640×480 pixel frame, with Binning set to 1. Meanshift will take less than 1 millisecond, LK takes 7 milliseconds and PhaseCorr takes 50 milliseconds.

2) XX, XY, YX, YY are coefficients used by the XY Tracker to figure out the orientation of the camera to the XY stage. This setting is automatically determined by the XY Calibration routine, which is initiated by pressing the Start XY Calib.

3) XY Gain, gain or multiplier used by the tracker to generate correction. If the correction is too slow, increase this setting. If the correction is too much that it overshoots then starts ringing or oscillating, decrease the settings.

4) XY Error check, applies only to LK Algorithm. When checked, Plugin runs the Algorithm twice (once with corners of frame 1 to get new location of corners in frame 2, then again with corners in frame 2 to get location of old corners in frame 1). This helps remove any false pixel shift and error.

5) XY Track Radius applies mostly to LK and Meanshift Algorithm. These Algorithms need user to specify a feature to track. This setting specifies the distance from the mouse click the Plugin should consider feature to track.

6) Threshold for Meanshift, Meanshift Algorithm works on Binary image, this setting sets the Threshold when generating the binary image. Can be adjusted in real-time.

7) Color Margin, used by the ColorTracker and ColorMeanShift routine, its sets the color tolerance

8) Erode, used by the ColorTracker, ColorMeanShift, MotionMeanShift and MotionTracker routine. Once the binary image is generated, Plugin applies the Erode routine to reduce region of pixels left, it helps in reducing the noise and protrusions.

9) Dilate, used by the ColorTracker, ColorMeanShift, MotionMeanShift and MotionTracker routine. Is the converse of erode; it expands pixel regions left, smooth out concavities. Dilate is applied after erode.

10) Brightfield? , Meanshift Algorithm works on Binary image with a dark background. Brightfield images have a bright background and will need to be inverted, while fluorescent microscopy images do not. Enabling this setting inverts the intensity of the images before making it binary.

11) Max XY Run Away sets the maximum distance the Plugin is allowed to move the XY stage. When the Plugin reaches the maximum distance of travel, it cancels the lock and returns to READY state. Helps prevent runaway state.

12) Move to Center, when checked, when XY tracker enters LOCK state, tracker will try to move the tracked feature to center of Up image frame.

13) X & Y offset, work with the “Move to Center” feature, lets user input a pixel offset from the center of up image frame they would like the tracking feature to be.

### Miscellaneous Tab

1) Test OpenCV, run a snippet of OPENCV code, used in verifying if OPENCV libraries are installed and working properly.

2) Serial, lets user type in a serial command to send to the “Tiger Comm device”. Pressing the return key sends the command

3) SEND, pressing the send button sends the command in “Serial” textbox to Tiger device. Output is printed in textbox#6

4, 5) Copy, Clear, Copies or clears the contents of textbox 6.

6) Textbox prints the results from the “Start Z Sweep”, “Test OpenCV”, “XY Calib”, and replies to the serial commands.

### Results Pane

Results pane displays read only data about the state of the system

1, 2) Up, Down, displays the focus score of the Up and Down image

3) Delta, displays the Difference between the Up and down image's focus score and accounts for the Z Offset too.

Delta = (Up Image Score - Down Image Score)-Z Offset

4, 5, 6) X Corr, Y Corr, Z Corr shows the vector correction being applied to XY and Z stage.

7,8,9) X Pos,Y Pos ,Z Pos, shows the current X,Y and Z drive position in um. Only updates when XY or Z Tracker is in LOCK state.

10, 11) XY Status Status, shows the state of the XY and Z tracker, such as IDLE, READY, LOCK etc.

12, 13) Process Time (ms), shows how much time the Plugin is taking processing the frame. XY indicates the pixel shift processing time; Z shows the focus scoring time.

### Buttons Pane

1) Halt, Stops XY and Z stage from moving and puts all Tracker to idle state.

2) Start Acq, Starts Acquisition of images. Puts Camera in TTL mode, and sets Tunable Lens to dither with TTL out. Button then changes state to Stop Acq. When pressed shuts down acquisition and return all Trackers to idle state and stop any XYZ stage travel.

3) Get Low Focus Score, copies the current Focus score into the Min Focus Score setting in the Z setting Tab. Z Tracker will not do any corrections if Focus score drops to this level. This helps the Plugin focusing on the background.

4) Get Z offset, saves the current delta  (Up Image Score - Down Image Score) into the Z offset setting in Z tab.

5) Z Lock, when pressed Plugin copies the current Delta into Z offset settings (same as “Get Z offset” button), and puts the Z Tracker in LOCK state. If Z Tracker is already in LOCK state, then it returns it to IDLE state. Tracker will start applying corrections to Z drive.

6) Move Z Back, Plugin moves Z drive back to the position before LOCK state. Helps recover from a runaway state.

7) Z Calib performs a series of moves that helps it find various settings like Min Delta, Z pol and Min Focus score. User will need to run this when Z Algorithm is changed.

8) Toggle XY Ready, moves the XY tracker from IDLE state to READY state, or back to IDLE state if already in READY state. In READY state the XY Tracker is active but not locked, it starts comparing the UP image frames to determine the pixel shift. LEFT clicking a spot on the UP Image will also put the XY tracker in READY state. LK and Meanshift algorithms need features to track. If “Toggle XY Ready” button is clicked instead of the LEFT licking on UP Image, Plugin will default to using the Center of image as feature to track.

9) Toggle XY Lock, moves the XY tracker from READY state to LOCK state, or back to READY state if already in LOCK state. Tracker will start applying correction to XY stage. RIGHT Clicking a spot on the UP Image will also put the XY tracker in LOCK state, and READY state if already in LOCK state.

10) Start XY Calib, performs a series of moves that help the Plugin determine the orientation and Imaging camera to that of the stage. This routine will automatically set the XX, XY, YX, YY settings in the XY settings tab. Only need to be performed if a new camera is installed or Objective is changed.

11) XYZ Lock Toggle, a quick macro buttons, puts both the XY and Z tracker in LOCK state. Same function as “Z Lock” and “Toggle XY Lock” button.

12) Set TLens Offset, sometimes the optical path with the tunable lens and main optical path with the imaging camera may not be parfocal. In this case the Tunable lens can be adjusted to bring both paths into focus. This buttons saves the setting, so when the Plugin dithers the Tunable lens. It dithers at this offset position.

13) Start Z Sweep, does a series of twenty Z moves with 5um set size, and prints the UP, DOWN image focus score, and delta into a test box at the end of each move. It also generates a graph like the one shown below. It helps troubleshoot and adjust the Z tracker. The step size for the Z sweep can be adjusted with “Calib Step” setting in “Z tab”

Graph generated at the end of Start Z Sweep routine

14) Save, saves all the Plugin settings in an xml file. When clicked, user is shown a file dialog box, user can select the save file name and location where it is saved it. Note: Plugin automatically saves and retains settings between sessions. This Buttons lets you copy these settings in a XML file

15) Load, lets user load settings from a previously saved XML file. User is shown a file dialog box, user can then select the save file to load from. Plugin will then quit, and when it is restarted, it will load with settings from the save file.

Dialog box shown to user when they are loading settings from a save file.

16) Manual, opens the default web browser on the PC and navigates it to this webpage.

## Z Focus Algorithms

Below is a list of Focus scoring algorithms available in the Plugin.

### sharpedges_mm

Provided by micro-manager; a slow routine. It is able to score very blurry images. Is sensitive to intensity variation.

### sharpedges_cv

Same as above but implemented with OpenCV library, routine is one of the quickest. However it is NOT able to score blurry images very well. Is sensitive to intensity variations.

Provided by micro-manager, routine is medium-quick, notably not sensitive to intensity variations.

### Volath_mm

Provided by micro-manager, routine is one of the quickest. Is suitable for most cases.

The “Start Z sweep” routine is a good way to compare the focus routines.

## XY tracking Algorithms

Below is a list of XY tracking algorithms provided in the Plugin:

### PhaseCorr

OpenCV's implementation of the Phase Correlation routine. This routine takes in two consecutive images and outputs relative movement between them. It's the slowest routine in the pack as it processes the entire frame.

When XY tracker is in idle state, left clicking on the up image or clicking the “Toggle XY Ready” button starts the tracker and puts it in “Ready” mode. This routine doesn't need a reference pixel to begin tracking.

### LK

OpenCV'sLucas Kanade Optical flow routine. This routine outputs the relative movements of a set of pixels between two consecutive images. These pixels are called corners (pixels that have intensity variation in more than one direction). By selecting corner pixels around an interested object, this routine is able to track the position of this object in a frame. This routine is medium-quick, but isn't too robust.

Tracker is activated when user left clicks on the up image or clicks the “Toggle XY Ready” button. When activated the routine takes the current frame, make a circular mask of radius defined by the “XY Track Radius” setting around a reference pixel. If the user left clicked, it uses the mouse position as the reference pixel, and if “Toggle XY Ready”, uses the center of video as the reference pixel.

Routine runs OpenCV's GoodFeaturestoTrack on this sub image, which returns all the pixels that have corners. Routine will use these pixels to measure relative moments between video frames. User can track an object in a video frame, by setting the “XY Track Radius” to the size of this object, and then left click on the object, pixels about this object will be selected to track, and the tracker will follow this object.

At any time, even in Lock mode, user can add more pixels to track by left clicking on the Up image.

### Meanshift

OpenCV's MeanShift routine. This routine works on binary images. User defines a rectangle, routine finds the center of mass in the rectangle, and shifts the center to it. Then on the next consecutive image, say the object that was inside the rectangle moved, routine calculates the center of mass and moves the rectangle to the new center. By seeing the change in the rectangle's position, we can get the relative movement between the two images. This routine is one of the quickest.

Tracker is activated when user left clicks on the up image or clicks the “Toggle XY Ready” button. When activated the routine takes the current frame, turns it into a binary image with the threshold value in “Threshold for Meanshift” setting (This setting can be adjust on the fly). The routine then generates a square with double the width given in “XY Track radius” settings with center around the reference pixel. If the user left clicked, it uses the mouse position as the reference pixel, and if “Toggle XY Ready”, uses the center of video as the reference pixel.

The routine then calculates the center of mass of the binary image within the square, and shifts the center of the square to that new center. It repeats this process on the same image a few times. If the user is interested in tracking an object, they can define the “XY Track Radius” to be slightly bigger than the object. Click near or on the object. The routine will generate a square and centers it on the object. Now we know where the object is in the current frame. On the next frame, if the object moved, the square is re-centered on the object again, and we now know the object's new position.

User can change the object they like to track anytime (even in lock mode) by Left clicking on a different object. If the routine loses the object, when the object is no longer is within the Square. If the object moves back into the square, it'll continue to track it, it can recover on its own making it robust.

### ColorMeanshift

Similar to the above Meanshift routine, uses a binary image to track too, instead of generating it with a Threshold value, it uses color.

When user selects a reference pixel (mouse position when user left clicks, or center of the video when they use the “Toggle XY Ready” button). The Routine registers the color of the pixel. Then it generates a lower and upper bound of color with help of the “Color Margin” settings. Routine then filters out every color in the frame that is not the reference pixel color or falls within the upper and lower bound. Generates a binary from this image, applies erode and dilate effect (amount given by the Erode and Dilate setting, setting it to “0”, doesn't apply that affect) Then generates a square and centers it on the reference pixel. From here the operation is very similar to that of Meanshift.

The added color processing doesn't increase processing time by much. This routine offers the same robustness and flexibility as regular Meanshift .

### ColorTracker

Like the ColorMeanshift routine, this too uses color of the reference pixel and filters out every color except that in the image. Applies erode and dilate affects too.

The Binary image that is now generated has these white objects. Instead of doing Meanshift, This routine now calls in OpenCv's Find Contours feature to scan the entire image and find all the Object. It then returns the location of the biggest object to the XY tracker.

This routine may have limited usage but it has some interesting features. Since it is looking at the entire image, it's less likely to lose the object. If it does lose the object, as soon as the object reappears in the frame, the routine will resume tracking. However if a bigger object enters the frame, the routine will lock on the new object and start tracking that.

Processing contours is slower than Meanshift, so this is a medium slow routine.

### MotionMeanShift

This routine begins by performing a Differential operation on two consecutive image frames from a video. The output of this operation is a new image with just the pixels that have changed between the two frames. Static objects and background is removed. This image is turned into a binary, and then Meanshift routine is performed to get the position of the moving object.

When user selects a reference pixel (mouse position when user left clicks, or center of the “Up video” if the user clicks the “Toggle XY Ready” button). The Routine is activated and performs the Differential operation between the current and next image, it draws the Mean Shift rectangle with the reference pixel at center with the width prescribed in the “XY Track Radius” setting. The operation that follows is similar to that of MeanShift routine.

This routine is as fast as the other Mean Shift routines.

Note, once the XY tracker is enabled and locked, sample is moved by the stage, static objects will also appear to move. This routine may have limited use.

### MotionTracker

Like the MotionMeanShift this routine also performs a Differential operation on two consecutive image frames from a video. The output of this operation is a new image with just the pixels that have changed between the two frames. Static objects and background is removed.

The image is turned into a binary, and instead of performing MeanShift, this routine instead calls in OpenCv's Find Contours feature to scan the entire image and find all the Object. It then finds the biggest object and gets its position.

This routine can also be activated with Left Click or “Toggle XY Ready” button. However it doesn't need a reference pixel. It scans the frame, and tracks the biggest moving object.

Note, once the XY tracker is enabled and locked, sample is moved by the stage, static objects will also appear to move. This routine may have limited use.

### Other manuals

For more info on Tunable Lens refer Tiger Tunable Lens Card (TGTLC)