2.1.

Cell Classification in High-Plex IF Images

High-plex IF imaging is a powerful tool for investigating cells in situ, but characterizing cells in high-plex image data comes with challenges. Unlike single-cell experiments, classifying and characterizing cells in tissue can be affected by tissue autofluorescence, true signal from neighboring cells, inaccurate cell segmentation, image pre-processing artifacts, and batch effects from staining and imaging.^15,25,26 Most existing methods for cell annotation in high-plex image data perform cell localization and segmentation first then characterize the detected cells by evaluating the pixel intensities within each cell mask across all relevant image channels.¹⁵ One popular method for cell annotation is reminiscent of flow cytometry analysis: applying manual or semi-automatic gates to MPI across the population or detected cells, following decision-tree logic until all cells have reached a terminal branch.^14,27 Clustering algorithms are also commonly used to separate cells into specific phenotypes.^17,28

These decision tree and clustering algorithms are dataset-specific and typically not generalizable to new data. Decision trees are known to be unstable, as small changes in the data distribution can produce drastically different results.²⁹ Even on the same data with the same normalization, slightly different manual gates affect downstream decision points and cell classifications. Practically, these methods are typically iterative, requiring multiple rounds of human intervention. Brbić et. al.³⁰ proposed a graph-based deep learning method that can potentially generalize to new datasets and minimize the need for human intervention.¹⁷ This graph-based method also includes cell location to improve classification. All of these “cell-first” methods for cell classification share limitations related to batch effects and signal overlap from neighboring cells. Some algorithms, such as REDSEA,²⁵ have been developed to try to mitigate some of these limitations. Normalization and batch effect correction are also integral steps for ensuring accurate classification with these methods.¹⁵

2.2.

Spectral Angle Mapping and Pixel Classification

Hyperspectral imaging, originally known as imaging spectroscopy, is a decades-old imaging technique developed for remote sensing.³¹ Spectral angle mapping (SAM) was first applied for hyperspectral image and pixel classification and clustering in 1993.²⁴ To achieve accurate classifications of pixels in remote sensing data, a cosine similarity metric [Eq. (1)] was used to compute the angular distances between a detected spectrum (A) and spectra of known ground surface materials (B). As the advent of this pixel classification scheme, SAM and its variants have been used for pixel classification in remote sensing images and novel applications of hyperspectral imaging, such as label-free imaging of biospecimens^22,32–35

Clear parallels can be drawn between hyperspectral image data and high-plex IF data. Rather than wavelength along the channel dimension of the image, high-plex IF has protein-specific markers. Pixel vectors from a high-plex IF image do not contain spectra in the physical sense, but the variable intensity across channels can be interpreted as such. Lui et al. developed Pixie,¹ a method for pixel-level annotation (classification) in high-plex IF images. Pixie relies on clustering for pixel classification. However, due to the immense number of pixels in a whole-slide image, current hardware limitations require sampling of pixels, and therefore, Pixie is only trained on a small fraction of pixel vectors from the image. This potentially limits the ability of the algorithm to detect rare classes of pixels.

pSAM leverages the interpretability and simplicity of spectral angle-based pixel classification methods as a novel approach to solving the cell classification problems presented by high-plex IF imaging. This method allows researchers to incorporate known biology into cell classification while mitigating many of the challenges that come from a “cell-first” classification method. These “cell-first” approaches prioritize reducing the image into objects (cells) then extract features of those cells for classification. Conversely, pSAM takes a “class-first” approach, maintaining full spatial resolution but compressing the channel dimension of the image into class maps prior to pulling cells out of the data. This “class-first” approach overcomes many previously mentioned limitations of the “cell-first” approach, including batch effects and the spatial bleed through of signal into neighboring cell segmentations.

3.1.

Sample and Image Acquisition

Thirteen formalin-fixed, paraffin-embedded (FFPE) colon biopsy samples from patients diagnosed with primary sclerosing cholangitis (PSC) and/or inflammatory bowel disease (IBD) were acquired from the Human Tissue Resource Center at the University of Chicago (HTRC). In addition, three kidney samples were selected from existing datasets also acquired from the HTRC. One kidney sample was from a patient diagnosed with lupus nephritis (LuN), one from a patient with mixed rejection of renal allograft [referred to later as kidney transplant rejection (KTR)], and one from a patient with angiomyolipoma. All kidney samples were also FFPE.

A $5\text{-}\mu \mathrm{m}$ section of each biopsy was de-paraffinized and mounted on a functionalized coverslip for iterative staining and imaging. Colon samples were iteratively stained and imaged with a 13-marker panel (Table S1 in the Supplementary Material), and kidney samples were iteratively stained and imaged with a 43-marker panel (Table S2 in the Supplementary Material), which included 10/13 markers from the colon panel. The PhenoCycler protocol was used for iterative staining.³⁶ All samples were imaged on an Andor Dragonfly spinning disk confocal microscope with a $40\times$ objective lens. For the colon dataset, each cycle of imaging included a nucleus stain [4′,6-diamidino-2-phenylindole (DAPI)] imaged at 405 nm and three other stains from the panel imaged at 488, 561, and 637 nm. For the kidney samples, each imaging cycle contained DAPI and four other stains, with an additional channel at 730 nm. Blank imaging cycles (no fluorescence reporters except for DAPI) were acquired before and after all staining cycles to capture background tissue autofluorescence. The resulting full-section images had a $0.1507\text{-}\mu \mathrm{m}$ pixel size.

3.2.

Image Preprocessing

Images were acquired in $2048\times 2048\mathrm{pixel}$ fields of view with a 205-pixel ($\sim 10\%$) overlap in each dimension with neighboring tiles. The Ashlar stitching and alignment software³⁷ was used to stitch all image tiles into full-section composites and align all imaging cycles to the DAPI channel from the first staining cycle.³⁷ Ashlar performance was visually checked across all samples. After aligning all image channels, the first blank cycle of imaging was used for background subtraction and spectral normalization of all stained images. First, each channel of the blank cycle was subtracted from the corresponding fluorescence channel in all imaging cycles. Each imaging wavelength has a different dynamic range, so the subtracted images were also divided by the standard deviation of the background image to standardize dynamic range across imaging wavelengths. After standardization relative to imaging wavelength, images were min–max normalized to the 99th percentile.

3.3.

Defining Reference Pseudo-Spectra

The specific reference pseudo-spectra for each panel were generated through a literature review and with expert immunologist input to reflect the protein expression patterns for each cell class. Because certain markers are known to localize to different compartments of the cell (i.e., nucleus, cytoplasm, or membrane), compartment-specific references were defined (Fig. 2). For the PSC dataset, reference (or “ideal”) pseudo-spectra were defined with binary values (0 or 1) for each marker (Fig. S1 in the Supplementary Material). In more complex panels such as the kidney panel, there are situations in which a marker is expected to be expressed at multiple levels (high/low/no expression). In these cases, a value of 0.5 was used to capture low expression (Fig. S2 in the Supplementary Material). Importantly, marker order was held consistent across all defined references. After visual quality checks of the images, some references were adapted, and new references were generated due to the cross-reactivity of some markers. The “tubule nucleus” reference spectra used for the kidney data defined CD21 as high in the nucleus compartment of several tubule cells in these pathological states.

Fig. 2

Reference pseudo-spectra for the PSC dataset were designed for specific cell compartments such as the nucleus (a) or cell membrane (b), based on the molecular target of the marker.

Given the wealth of literature defining immune cells based on their gene or protein expression,^38–41 and the help of expert immunologists, we generated ideal pseudo-spectra as models of the cell types probed by two high-plex staining panels. For example, an exhausted CD8 T cell will have high expression of the markers CD45, CD3, CD8, and PD1 on the surface of the cell, whereas a $T$ regulatory cell will have high expression of CD45, CD3, and CD4 on the surface of the cell and high expression of Foxp3 in the nucleus of the cell. In addition, images were examined for cross-reactivity of the antibodies. In the kidney dataset, CD21 was selected to identify germinal center B cells. However, this marker showed a notably high signal in many tubule cell nuclei of the kidney. Therefore, we also generated a reference pseudo-spectrum for a “tubule nucleus” class, even though an antibody specific to tubule cell nuclei was not used in the staining panel.

In the 13-marker panel used for staining colon biopsies from PSC and IBD patients, three markers probed cell nuclei, resulting in three unique ideal pseudo-spectra for the nucleus compartment [Fig. 2(a)]. The remaining 10 markers are expected to be expressed on the cell membrane or within the cytoplasm of cells,^42–44 collectively referred to here as the membrane compartment. Three of the 13 reference pseudo-spectra for this compartment are shown in Fig. 2(b), with all reference profiles displayed in Fig. S1 in the Supplementary Material. In total, 16 unique reference pseudo-spectra were generated from the 13-marker panel used in the PSC imaging experiment. Notably, there are more cell classes that could exist in these inflamed colon samples which cannot be captured by this 13-marker staining panel.

The kidney dataset contained 43 markers, 30 of which were used to define reference pseudo-spectra. Thirty-eight unique references were generated from these 30 markers: 34 membrane-associated pseudo-spectra and 4 nucleus-associated pseudo-spectra (Fig. S2 in the Supplementary Material). For the kidney dataset, we also expect a contingent of unclassified cells: some that are not captured by the staining panel and some that are not captured by this set of references. We generated 38 total references from this panel, but more references could be generated from these 43 markers that would further sub-type some of our selected classes.

3.4.

Pseudo-Spectral Angle Mapping

All image channels were consistently stacked in the order defined by the reference pseudo-spectra. Each channel of the multi-channel stack was normalized to match the range of the reference pseudo-spectra (0 to 1). Pixel vectors extracted from the image data were defined as the value of all markers at a single ($x,y$) position. Each pixel vector $A$ in the image was compared with all reference pseudo-spectra $B$ by computing the cosine similarity between the two vectors [Eq. (1)]. After all vector similarity calculations were complete, class maps for all references were stacked. Pixel class was defined as the argmax of the class map stack ($\mathrm{\Theta }$) at each location [Eq. (2)]

3.5.

Instance Segmentation of Cells

The DAPI channel of each image was used to generate instance segmentations of all cells in each full-section image. The “nuclei” model in Cellpose2.0⁴⁵ was fine-tuned using human-in-the-loop training for each dataset. The three kidney samples used for the generalization experiments originated from separate datasets. Separate models were fine-tuned for the lupus nephritis and kidney transplant rejection datasets. The model fine-tuned for lupus nephritis was also applied to the angiomyolipoma sample. Nucleus segmentations were dilated by $\sim 1\mu \mathrm{m}$ (7 pixels) to achieve approximate whole-cell segmentations. Voronoi tessellation was used to avoid overlap of the expanded nuclei in areas of crowded cells.

3.6.

Cell Classification

To optimize our cell classification, a threshold of 0.5 was applied to all class maps to reject low cosine similarity scores. Importantly, this threshold is below the minimum value of the maximum intensity projection (MIP) of all class maps. Cell membrane segmentations were computed by subtracting the nucleus segmentation from the whole-cell segmentation. For each cell predicted by the fine-tuned Cellpose model, we computed a score for all computed class maps using the following protocol (Fig. 3).

Fig. 3

Class maps are combined with instance segmentations of cells to classify cells. The segmented nuclei and a dilated ring around each nucleus are used to mask class maps to generate class scores for each detected cell. The maximum score in each compartment provides a compartment-specific class for every cell.

A “soft” mask of each nucleus segmentation was created using an inverted distance transform to blur the edges of the binary mask. These “soft masks” were multiplied by all class maps associated with the nucleus compartment. The membrane mask (dilated nucleus segmentation mask—nucleus segmentation mask) was also blurred using a distance transform and multiplied with all class maps associated with the membrane compartment. The mean score of the masked class maps was computed for all reference pseudo-spectra for each cell. Each cell was given a nucleus classification and a membrane classification based on the maximum score for each compartment. A cell received a membrane class of “unclassified” if it scored zero for all membrane classes. If the maximum score for the nucleus compartment was zero, the cell received a nucleus classification of “generic nucleus.”

4.1.

pSAM Collapses High-Plex Images to Class Maps

pSAM is a method for computing spatial likelihood maps for cell types probed by the markers used in a custom staining panel. In pSAM, a cosine similarity metric is used to evaluate how close a pixel vector is to a reference vector in an $N$-dimensional space, where $N$ is the number of channels in an image.

Class similarity maps for a single reference pseudo-spectrum can compress information from all markers in the panel. Class maps of select reference pseudo-spectra for a full biopsy section are displayed in Fig. 4(a). A maximum intensity projection of all 16 class maps, colored by the argmax [Fig. 4(b)] revealed definitive structures—potentially even tertiary lymphoid structures, and object-level (or cell-level) variability in densely packed areas [Fig. 4(b), inset]. Importantly, the class of the maximum score corresponded to the correct compartments, with nucleus-associated classes scoring highest in cell nuclei, and membrane-associated classes scoring highest in the pixels surrounding cell nuclei. As expected, the highest prevalence pixel class is the “generic nucleus” class, where prevalence is the fraction of the entire population assigned to a class [Fig. 4(c)]. In addition, the “other $T$ cell” class is also very low prevalence, which is expected, as $T$ cell subsets that are negative for both CD4 and CD8 are generally rare. Although very few pixels are most similar to the “other $T$ cell” reference pseudo-spectrum, the average class score (or average cosine similarity across the pixels in this class) for this category was relatively high [Fig. 4(d)]. Conversely, the pSTAT3+ nucleus class showed high pixel prevalence and a relatively low average class score. Within the maximum intensity projection, the areas corresponding to the lumen of the intestine were associated with this class [Fig. 2(b), orange]. The intestinal lumen can be interpreted as background, as there is no tissue in this area.

Fig. 4

The pSAM class maps display the similarity of pixels to each cell type without the need to compare relative expression across multiple image channels (a). Given the maximum value at each $x/y$ location, individual cells of different populations are clearly visible, even in densely packed areas (b). For the sample shown in panel (b), the prevalence of each pixel class within the tissue was calculated (c). The average cosine similarity score for all classes of pixels in this example image is shown in panel (d). Prevalence (c) and average score (d) are calculated over the pixels that fall within the tissue mask, whereas the slide background is filtered out as background [black in panel (b)].

4.2.

Pixel Class Maps Aid in Cell Classification

Cellpose2.0 was fine-tuned to detect the cell nuclei in each dataset. After fine-tuning, the performance of each tuned model was evaluated in an independent test set from each dataset (Fig. 5). These test sets comprised 10 non-overlapping image tiles from five separate biopsies for each dataset. Because the angiomyolipoma sample was the only sample from that pathology, the model tuned for the lupus dataset was also used to segment cell nuclei for the angiomyolipoma sample. Note that we demonstrate pSAM in three kidney samples from a larger dataset, so the five samples used for the evaluation of instance segmentation performance are not included in pSAM analysis. In a test set of 50 image tiles per dataset ($512\times 512\mathrm{pixels}$), separately fine-tuned models achieved an $F1$-score of 0.83 ± 0.13 for the PSC dataset, $0.86\pm 0.11$ for the LuN dataset, and $0.70\pm 0.16$ for the KTR dataset.

Fig. 5

Representative results from three separately fine-tuned Cellpose models on the PSC, LuN, and KTR datasets. Ground truth segmentations (top, yellow) agree well with the fine-tuned models (bottom, magenta) on data from previously unseen patients.

By collapsing high-plex image data into pixel-level class representations, we were also able to infer likely cell phenotypes—merging pixel-level descriptors with predicted object (cell) masks. Class maps computed using pSAM were combined with instance segmentations of cell nuclei to assign cell classes in the PSC dataset [Fig. 6(a)]. The predicted classes agreed well with the class representations and the fluorescence images themselves [Figs. 2(b) and 6(a)]. These cell phenotype scores were validated by analyzing the MPI for each category of cells across all thirteen colon samples [Fig. 6(b)]. The intensity profiles for each detected cell phenotype corresponded well with the expected relative protein expression for the assigned cell type. This population-level validation demonstrates the potential of pixel-level analysis of highly multiplexed images to improve cell classification. Variation in cell class prevalence was seen across colon samples. Many samples had a plurality of IgG+ plasma cells, whereas others had a much smaller proportion of these cells [Fig. 6(c)]. Across all samples, a strong majority of cells were classified as having a “generic” nucleus [neither Foxp3+ or pSTAT3+, Fig. 6(d)]. Foxp3+ nuclei were much more common than pSTAT3+ nuclei; however, it is worth noting that many samples showed markedly high Foxp3 signal in the villi of the intestine, which may have elevated this prevalence. In agreement with visual inspection of pSTAT3 images, many samples had no pSTAT3+ nuclei, and those that did had a very small proportion. Note that we did not combine membrane and nucleus classes into overall cell classes for this analysis (i.e., regulatory T cells would be CD4T cells or inducible co-stimulator (ICOS)+ CD4T cells with a Foxp3+ nucleus).

Fig. 6

Cell membrane classifications are shown for sample S1 in panel (b). Cell classes were validated using the MPI for all channels for each cell. MPI was $Z$-scored across the full PSC dataset (13 biopsies, $\sim 885\mathrm{K}$ cells), and the average $Z$-score for each channel is calculated for each cell class across all samples (b). Note that each cell received a membrane class (top 13 rows) and a nucleus class (bottom 3 rows). Across the 13 samples in the PSC dataset, pSAM detected varying distributions of cell classes in the membrane compartment (c) and the nucleus compartment (d). Notably, the generic nucleus class (gray) is a strong majority of cells, and pSTAT3+ nuclei (orange) are rare but present and detectable in a few samples. The color bar in panel (a) is also applicable to panel (c).

4.3.

Comparing pSAM and Other Cell Classification Methods for High-Plex IF Imaging

Current mainstream methods for classifying cells classification in high-plex IF images draw from single-cell analyses such as flow cytometry and single-cell sequences. Decision trees are generally used in flow cytometry analyses, passing cell populations through “gates” or hand-drawn thresholds at each decision point. In single-cell sequencing, clustering algorithms are generally used to group cells into similar populations. The resulting clusters are manually, iteratively tuned to reach biologically sound classifications. These two methods can also be implemented to classify cells in images, using the MPI of each segmented cell in all channels for gating in a decision tree or as input features into a clustering algorithm.

To compare these methods to pSAM, we have designed a decision tree for classifying cells in the 13-plex PSC data for comparison to the pSAM classification results [Fig. 7(a)].⁴⁶ Thresholds at each decision point were calculated semi-automatically by first implementing a multi-Otsu thresholding then manually selecting one of the calculated thresholds for each decision point in the tree. We also implemented $K$-means clustering using 11 of the 13 cell markers in the panel. Cell nucleus markers Foxp3 and pSTAT3 were excluded for both the decision tree and clustering to draw direct comparisons to the pSAM “membrane-based” classification.

Fig. 7

Comparison of pSAM to a decision tree and $K$-means clustering for cell classification. (a) A decision tree was designed to classify cells in the PSC dataset. Cells are gated at each decision point using a threshold of MPI for the listed marker. UMAP dimensionality reduction was used to compress the 13-dimension feature space into two dimensions, colored by cell class defined by the decision tree (b), $K$-means clustering (c), and pSAM (d). Cell classes were also mapped back to the global coordinate space [gray and yellow boxes in panels (b)–(d)]. These regions correspond to the gray and yellow insets from the image in panel (e) (selected channels shown).

Uniform manifold approximation and projection (UMAP) dimensionality reduction was used to visualize all 13 MPI features in a two-dimensional feature space, and the resulting plot was colored by the cell classification used by the decision tree [Fig. 7(b)], $K$-means clustering [Fig. 7(c)], and pSAM [Fig. 7(d)]. Similar areas within the UMAP space correspond to the same cell classes across the three methods. The decision tree and $K$-means clustering result in a larger group of unclassified cells relative to pSAM. In addition, the “other $T$ cell” class [yellow in Figs. 7(b)–7(d)] was over-represented in $K$-means clustering, as it should be a rare cell class. In addition, the decision tree showed comparatively fewer plasma cells and IgG+ plasma cells relative to clustering and pSAM. Notably, IgG+ plasma cells have been shown to be an abundant cell type in PSC.⁴⁴

Visualizations of the classified cells from each method [Figs. 7(b)–7(d)] show that pSAM accurately depicts a stark boundary between B cells and plasma cells [Fig. 7(e), gray box], whereas both the decision tree and $K$-means clustering predict a region of “other immune cells” between the B cell and plasma cell regions. Similarly, pSAM leaves fewer cells unclassified in the villi of the intestine [Figs. 7(b)–7(e), yellow inset].

4.4.

pSAM is Readily Generalizable to New Imaging Experiments

High-plex IF imaging experiments vary widely in the protein marker panel used to probe cells and tissue. Therefore, to evaluate the generalizability of pSAM, we applied the same method used on the 13-marker PSC dataset to a pilot set of three kidney samples imaged with the same 43-marker staining panel. The higher plex staining panel used in the kidney dataset allowed for more than double the number of cell types and states to be detected in this dataset relative to the 13-plex PSC dataset. Pixels and cells of a given class were consistently identified across all three kidney samples, despite coming from patients with different diagnoses. In addition, the pixel class prevalence trend as expected; nucleus-associated classes had the highest pixel prevalence, and more inflamed samples showed a higher prevalence of pixels with high similarity to immune cell reference pseudo-spectra (Fig. S3 in the Supplementary Material). Specifically, the KTR sample has by far the highest prevalence of lymphocyte-associate pixels (T and B cells), and this sample has three large clusters of lymphocytes visible in the image [Fig. 8(a), white arrows]. Cell classes were also computed using the instance segmentation masks and the pixel class representations.

Fig. 8

pSAM is readily applied to new datasets. (a) pSAM was used to classify pixels and cells in three kidney samples. Each cell received a class assignment for one of 34 membrane classes and one of four nucleus classes (data not shown). White arrows in the KTR sample denote immune cell aggregates. Insets at the location of the yellow box are shown to the right of the whole-section class map MIPs. Predicted cell classes for the inset are shown at the far right. (b) The $Z$-score of the population MPI for all stains in the 43-marker panel is compared across all cell membrane and nucleus classes. Note that all cells receive both a membrane class and a nucleus class. (c) Comparison of the relative populations in each sample shows that the immune populations detected show a diverse and abundant population of immune cells, corresponding with visual inspection of the images. (d) Tubule and generic nuclei (gold and gray, respectively) are by far the most abundant nuclei in all three samples. The color bar in panel (a) is also applicable for panel (c).

Validation with MPI across each predicted class of cells showed that each cell phenotype had the expected relative protein expression [Fig. 8(b)]. The trends for protein expression of a given cell type are similar across these samples from different pathological conditions (Fig. S4 in the Supplementary Material). All 43 markers were used for MPI validation, whereas only 30 markers were used for cell classification. Therefore, a few surprising trends were found with the remaining 13 markers. For example, RAR-related orphan receptor-$\gamma \mathrm{t}$ (ROR-$\gamma \mathrm{t}$) (not used in cell classification) was relatively high in SLAMF7+ plasma cells. Further investigation is needed to determine whether this finding is associated with cross-reactivity of antibodies or spatial overlap of signal from neighboring cells. Further validation of the pSAM cell classification showed that the prevalence of immune cell populations is highest in the transplant rejection sample, which corresponds with visual analysis of the full biopsy sections [Fig. 8(c)]. In addition, the generic nucleus class is by far the most abundant across samples, as expected [Fig. 8(d)].

4.5.

Integrating Cell Classification with Interstitial Pixel Classes

One of the benefits of pSAM is that both pixel and cell classifications can be calculated. Therefore, the complexity of the interstitial space between cells can also be analyzed. This is particularly important for contextualizing interstitial signal from irregularly shaped myeloid cells, such as macrophages and dendritic cells, as their protrusions are likely present in image planes where their cell nucleus cannot be captured. To demonstrate the utility of pSAM for this type of analysis, we have developed a “proximity score” metric for evaluating whether pixels of a given class preferentially fall near cells of a given class, suggesting that out-of-plane cells might be interacting with cells in the imaging plane. To calculate this proximity score, a proximity map for a given cell class is calculated [Fig. 9(a)]. First, pixels in the interstitial space (pixels that are in the tissue but lie outside of cell segmentations) are weighted with a Gaussian decay with respect to distance from the cell edge [Eq. (3)]. Sigma ($\sigma$) was selected to be 22.36 pixels, so that pixels $4\mu \mathrm{m}$ from the cell boundary will be weighted at 0.5. If a cell has no neighbors within the vicinity, this results in a smooth decay away from the cell [Fig. 9(a), left]. However, if there are cells from other classes that reside near cells in the class of interest, the pixels within those cell segmentations are weighted at zero, as they are not interstitial pixels [Fig. 9(a), right].

Fig. 9

Evaluating interstitial pixels through proximity mapping to cells. (a) For each cell class, proximity maps were calculated by weighting interstitial pixels with a Gaussian decay away from the boundaries of cells from that class. For sparse areas of cells (left), this results in a smooth decay. In areas of crowded cells (right), pixels belonging to cells of other classes are weighted at zero, as they are not “interstitial.” (b) By combining interstitial pixel class maps with the proximity maps from panel (a), each class of interstitial pixels can be scored relative to a given cell class. (c) Six interstitial pixel class maps are combined with the proximity map from CD4T cells to demonstrate the variability in proximity scores for different classes of interstitial pixels. CD4T interstitial pixels are generally located near the CD4T cell boundaries, as nucleus dilation is not a perfect segmentation strategy. $\mathrm{M}\phi$ pixels (both CD14+ $\mathrm{M}\phi$ and M2) are in close proximity to CD4T cells, suggesting that out-of-plane cells might be interacting CD4T cells. Interstitial pixels from DC classes also show infrequent high proximity scores. All interstitial pixel classes were scored in relation to all cell classes for the kidney transplant rejection sample (c) and the lupus nephritis sample (d).

Next, for each class defined by pSAM, a binary mask of pixels belonging to this class was multiplied by the proximity map defined for each cell class [Fig. 9(b)]. For each unique combination of cell and pixel class, the proximity score was calculated as the mean non-zero value. Proximity scores were calculated for the lupus nephritis sample and the kidney transplant rejection sample from above. As expected, the highest proximity scores are for matched cell and pixel classes along the diagonal [Figs. 9(c) and 9(d)]. Other high values indicate that pixels from one class are preferentially near cells of another, suggesting potential interactions. This type of analysis could augment a cell-centric analysis that infers interactions from the proximity of cell centroids. Importantly, this proximity score, facilitated by pSAM, can incorporate contextual information from the image that does not rely solely on detected cells, which can help to infer cell:cell interactions between in-plane and out-of-plane cells.