Automation

OpenFIBSEM provides a number of modules to support automation.

Image Processing

We provide a number of image processing functions to support automation. These are available in fibsem.imaging or fibsem.acquire.

AutoGamma:

Adjusting the gamma of the image can improve the contrast of the image. This is useful for aligning images, and can be used to improve the contrast of the image for feature detection. The following figure shows the effect of adjusting the gamma of the image. When the autogamma flag is enabled in ImageSettings the gamma of the images taken will be automatically adjusted with this method.

Gamma Adjustment

AutoCLAHE:

Contrast Limited Adaptive Histogram Equalization (CLAHE) is a method for improving the contrast of an image. When the autoclahe flag is enabled in ImageSettings the gamma of the images taken will be automatically adjusted with this method. AutoCLAHE has many parameters that can be tuned. The following figures shows the difference between the different autogamma methods.

AutoGamma Functions

AutoGamma Functions

Alignment

The alignment module contains functions are aligning reference images, and tools and parameters for adjusting these alignments.

WIP: images

Cross Correlation

The primary alignment method is fourier cross-correlation to a reference image. The following helper function provides the the methods for alignment, and the required movements, as well as parameters for tuning.

def align_using_reference_images(
    microscope: SdbMicroscopeClient,
    settings: MicroscopeSettings,
    ref_image: AdornedImage,
    new_image: AdornedImage,
    ref_mask: np.ndarray = None,
    xcorr_limit: int = None,
    constrain_vertical: bool = False,
    beam_shift: bool = False,
    lp_px: int  = 128, 
    hp_px: int = 8,  
    sigma: int = 6,
) -> bool:
    """Align new image to reference image using crosscorrelation

    Args:
        microscope (SdbMicroscopeClient): microscope client
        settings (MicroscopeSettings): microscope settings
        ref_image (AdornedImage): reference image
        new_image (AdornedImage): new image
        ref_mask (np.ndarray, optional): reference mask. Defaults to None.
        xcorr_limit (int, optional): crosscorrelation limit. Defaults to None.
        constrain_vertical (bool, optional): constrain vertical movement. Defaults to False.
        beam_shift (bool, optional): use beam shift. Defaults to False.
        lp_px (int, optional): lowpass filter size. Defaults to 128.
        hp_px (int, optional): highpass filter size. Defaults to 8.
        sigma (int, optional): gaussian filter sigma. Defaults to 6.
    Returns:
        bool: True if alignment was successful, False otherwise
    """

In order to tune the alignment the user can adjust the following:

Filtering

The cross correlation alignment can be tuned by adjusting the bandpass filters; lowpass, highpass; and sigma. These filters are applied in fourier space. - lowpass: upper pixel distance bound - highpass: lower pixel distance bound - sigma: blur applied to bandpass mask boundary

The effect of these filters on the bandpass mask is shown below:

Masking

We provide a number of masking utilities:

• circle_mask: mask a circular area of the image.
• bandpass_mask: mask a bandpass area of the image (donut)
• rect_mask: mask a rectangular area of the image
• area_mask: mask quarters of the image specified by left, right, upper, and lower params
• vertical_mask: mask a central vertical area of the image
• invert: invert the mask

These masks can be passed to the ref_maskfor use in the alignment, and are available in fibsem.imaging.masks. Many of these masking functions contain additional parameters, so please see the implementation for details.

Movement

By default, the function will move the stage to align the reference image. The following parameters adjust this movement.

• constrain_vertical: this flag will constrain the stage to move only vertically (z). This is useful when attempting to correct the eucentric position using alignment.
• beam_shift: this flag will adjust the shift the beam instead of moving the stage. The beam shift is more precise than the mechanical stage, but has a much lower range of movement (~10um?, check this)

Utilities

We also provide a number of utility functions that can help with crosscorrelation. For example:

• rotate_image: rotate the image 180deg while preserving metadata
• fromFibsemImage: match the image settings used for reference image

These utilities are available in fibsem.imaging.utils.

Helper Functions

The fibsem.alignment module contains more helper functions for performing alignment workflows:

• correct_stage_drift: multi-step alignment
• eucentric_correct_stage_drift: multi-step alignment + eucentric alignment

Feature Detection

We provide some feature detection functions and workflows primarly centred around the liftout workflow. These are customisable to the user's needs.

The primary detection workflow uses the following to run the feature detection, transform coordinate systems, and optionally allows the user to validate the result. For more information on the user validation please see ml.md.

def detect_features_v2(
    microscope: SdbMicroscopeClient,
    settings: MicroscopeSettings,
    features: tuple[Feature],
    validate: bool = True,
    mask_radius: int = 256,
) -> DetectedFeatures:

    """Detect features in microscope image.

    Args:
        microscope (SdbMicroscopeClient): microscope client
        settings (MicroscopeSettings): microscope settings
        features (tuple[Feature]): features to detect
        validate (bool, optional): whether to validate features. Defaults to True.
        mask_radius (int, optional): radius of mask to apply to image. Defaults to 256.

    Returns:
        DetectedFeatures: detected features
    """
    ...

The supported Feature Types are as follows:

Bases: ABC

Source code in fibsem/detection/detection.py

@dataclass
class Feature(ABC):
    px: Point 
    feature_m: Point
    _color_UINT8: tuple = (255,255,255)
    color = "white"
    name: str = None

    @abstractmethod
    def detect(self, img: np.ndarray, mask: np.ndarray=None, point:Point=None) -> 'Feature':
        pass

Bases: Feature

Source code in fibsem/detection/detection.py

@dataclass
class LamellaCentre(Feature):
    feature_m: Point = None
    px: Point = None
    _color_UINT8: tuple = (255,0,0)
    color = "red"
    name: str = "LamellaCentre"

    def detect(self, img: np.ndarray, mask: np.ndarray = None, point:Point=None) -> 'LamellaCentre':
        self.px = detect_lamella(mask, self)
        return self.px

Bases: Feature

Source code in fibsem/detection/detection.py

@dataclass
class ImageCentre(Feature):
    feature_m: Point = None
    px: Point = None
    _color_UINT8: tuple = (255,255,255)
    color = "white"
    name: str = "ImageCentre"

    def detect(self, img: np.ndarray, mask: np.ndarray=None, point:Point=None) -> 'ImageCentre':
        self.px = Point(x=img.shape[1] // 2, y=img.shape[0] // 2)
        return self.px

Bases: Feature

Source code in fibsem/detection/detection.py

@dataclass
class LamellaRightEdge(Feature):
    feature_m: Point = None
    px: Point = None
    _color_UINT8: tuple = (255,165,0)
    color = "orange"
    name: str = "LamellaRightEdge"

    def detect(self, img: np.ndarray, mask: np.ndarray = None, point:Point=None) -> 'LamellaRightEdge':
        self.px = detect_lamella(mask, self)
        return self.px

Bases: Feature

Source code in fibsem/detection/detection.py

@dataclass
class LamellaLeftEdge(Feature):
    feature_m: Point = None
    px: Point = None
    _color_UINT8: tuple = (255,0,255)
    color = "magenta"
    name: str = "LamellaLeftEdge"

    def detect(self, img: np.ndarray, mask: np.ndarray = None, point:Point=None) -> 'LamellaLeftEdge':
        self.px = detect_lamella(mask, self)
        return self.px

Bases: Feature

Source code in fibsem/detection/detection.py

@dataclass
class NeedleTip(Feature):
    feature_m: Point = None
    px: Point = None
    _color_UINT8: tuple = (0,255,0)
    color = "green"
    name: str = "NeedleTip"

    def detect(self, img: np.ndarray, mask: np.ndarray = None, point:Point=None) -> 'NeedleTip':
        self.px = detect_needle_v5(mask)
        return self.px

Bases: Feature

Source code in fibsem/detection/detection.py

@dataclass
class LandingPost(Feature):
    feature_m: Point = None
    px: Point = None
    _color_UINT8: tuple = (255,255,255)
    color = "cyan"
    name: str = "LandingPost"

    def detect(self, img: np.ndarray, mask: np.ndarray = None, point:Point=None) -> 'LandingPost':
        self.px = detect_landing_post_v4(mask, point)
        # self.px = detect_landing_post_v3(img, landing_pt=None)
        return self.px

Machine Learning

We provide a basic machine learning pipeline and workflow. Currently only segmentation models are supported, but more will be added in the future.

For detailed information on the machine learning tools, please see Machine Learning.

Movement

The movement module contains functions for moving the microscope stage.

We provide a number of advanced movement functions that are useful for correcting and maintianing the eucentricity of the microscope.

Eucentric Movement

Eucentric movements correct the eucentric position of the microscope. This is useful when initially setting up the microscope, or when the eucentric position has been lost. The eucentric movements move the stage vertically in the chamber to correct the eucentric position.

To use eucentric movements, use:

microscope.eucentric_move(settings, dy)

Move the stage vertically to correct eucentric point

Parameters:

Name	Type	Description	Default
settings	MicroscopeSettings	microscope settings	required
dy	float	distance in y-axis (image coordinates)	required

Source code in fibsem/microscope.py

def eucentric_move(
    self,
    settings: MicroscopeSettings,
    dy: float,
    dx: float = 0.0,
    static_wd: bool = True,
) -> None:
    """
    Move the stage vertically to correct eucentric point

    Args:
        settings (MicroscopeSettings): microscope settings
        dy (float): distance in y-axis (image coordinates)
    """
    _check_stage(self.hardware_settings)
    wd = self.connection.beams.electron_beam.working_distance.value
    image_rotation = self.connection.beams.ion_beam.scanning.rotation.value
    if np.isclose(image_rotation, 0.0):
        dy = dy
    elif np.isclose(image_rotation, np.pi):
        dy = -dy
    # TODO: does this need the perspective correction too?

    z_move = dy / np.cos(np.deg2rad(90 - self.stage_settings.tilt_flat_to_ion))  # TODO: MAGIC NUMBER, 90 - fib tilt

    move_settings = MoveSettings(link_z_y=True)
    z_move = FibsemStagePosition(
        x=dx,
        z=z_move, coordinate_system="Specimen"
    ).to_autoscript_position()
    self.connection.specimen.stage.relative_move(z_move, move_settings)
    logging.info(f"eucentric movement: {z_move}")

    # x-move
    # self.move_stage_relative(FibsemStagePosition(x=dx, y=0, z=0))

    if static_wd:
        self.connection.beams.electron_beam.working_distance.value = (
            settings.system.electron.eucentric_height
        )
        self.connection.beams.ion_beam.working_distance.value = (
            settings.system.ion.eucentric_height
        )
    else:
        self.connection.beams.electron_beam.working_distance.value = wd
    self.connection.specimen.stage.link()

where dy is the distance to move in the image plane (eucentric movements are always calculated from the ion beam perspective).

Stable Movement

Stable movements maintain the sample at the eucentric position, allowing for movement along the sample plane without losing the eucentric position. This is useful for performing liftout, and other tasks that require movement along the sample plane.

To use stable movements, use:

microscope.stable_move(settings, dx, dy, beam_type)

where dx, dy are the distance to move in the image plane, and beam_type is the beam type to use for the movement.

Calculate the corrected stage movements based on the beam_type, and then move the stage relatively.

Parameters:

Name	Type	Description	Default
settings	MicroscopeSettings	microscope settings	required
dx	float	distance along the x-axis (image coordinates)	required
dy	float	distance along the y-axis (image coordinates)	required
beam_type	BeamType	beam type to move in	required
_fixed	bool	whether to fix the working distance. Defaults to False.	False

Source code in fibsem/microscope.py

def stable_move(
    self,
    settings: MicroscopeSettings,
    dx: float,
    dy: float,
    beam_type: BeamType,
    _fixed: bool = False, 
) -> None:
    """
    Calculate the corrected stage movements based on the beam_type, and then move the stage relatively.

    Args:
        settings (MicroscopeSettings): microscope settings
        dx (float): distance along the x-axis (image coordinates)
        dy (float): distance along the y-axis (image coordinates)
        beam_type (BeamType): beam type to move in
        _fixed (bool, optional): whether to fix the working distance. Defaults to False.
    """
    _check_stage(self.hardware_settings)
    wd = self.connection.beams.electron_beam.working_distance.value

    if beam_type == BeamType.ELECTRON:
        image_rotation = self.connection.beams.electron_beam.scanning.rotation.value
    elif beam_type == BeamType.ION:
        image_rotation = self.connection.beams.ion_beam.scanning.rotation.value

    if np.isclose(image_rotation, 0.0):
        dx = dx
        dy = dy
    elif np.isclose(image_rotation, np.pi):
        dx *= -1.0
        dy *= -1.0

    # calculate stage movement
    x_move = FibsemStagePosition(x=dx, y=0, z=0)
    yz_move = self._y_corrected_stage_movement(
        settings=settings,
        expected_y=dy,
        beam_type=beam_type,
    )

    # move stage
    stage_position = FibsemStagePosition(
        x=x_move.x,
        y=yz_move.y,
        z=yz_move.z,
        r=0,
        t=0,
        coordinate_system="RAW",
    )
    logging.info(f"moving stage ({beam_type.name}): {stage_position}")
    self.move_stage_relative(stage_position)

    # adjust working distance to compensate for stage movement
    if _fixed:
        wd = settings.system.electron.eucentric_height

    self.connection.beams.electron_beam.working_distance.value = wd
    self.connection.specimen.stage.link()

    return stage_position