Protein Representation Learning with
Sequence Information Embedding:
Does it Always Lead to a Better Performance?
^††thanks: This work was supported by the National Natural Science Foundation of China (11974239; 62302291), the Innovation Program of Shanghai Municipal Education Commission (2019-01-07-00-02-E00076), Shanghai Jiao Tong University Scientific and Technological Innovation Funds (21X010200843), the Student Innovation Center at Shanghai Jiao Tong University, and Shanghai Artificial Intelligence Laboratory.

Consider three arbitrary peptide chains ${\mathcal{P}}_{1}$, ${\mathcal{P}}_{2}$, and ${\mathcal{P}}_{3}$, where ${\mathcal{P}}_{1}$ and ${\mathcal{P}}_{2}$ share a similar global structure. While ${\mathcal{P}}_{3}$ has a significantly different overall structure, it contains a common substructure ${\mathcal{P}}^{\prime}$ with ${\mathcal{P}}_{1}$ and ${\mathcal{P}}_{2}$. A global structure matching evaluates the overall similarity of peptide chain pairs, e.g., assigns a high similarity score to the $({\mathcal{P}}_{1},{\mathcal{P}}_{2})$ pair and a low similarity score to both $({\mathcal{P}}_{1},{\mathcal{P}}_{3})$ and $({\mathcal{P}}_{2},{\mathcal{P}}_{3})$.

Define ${\mathcal{G}}=({\mathcal{V}},{\mathcal{E}})$ the graph representation of a peptide chain’s backbone. Each node $v\in{\mathcal{V}}$ represents an amino acid, and spatially closed nodes (i.e., Euclidean distance smaller than 10Å) are connected by directed edges $e\in{\mathcal{E}}$. For the $i$th amino acid, the node feature is composed of scalars and vectors, i.e., ${\bm{H}}_{v}^{i}=({\bm{N}}_{S}^{i},{\bm{N}}_{V}^{i})$. The scalar feature ${\bm{N}}_{S}^{i}$ contains one-hot encodings of structure tokens, such as DSSP-based secondary structure [20] or FoldSeek embedding [21]. The vector feature ${\bm{N}}_{V}^{i}\in\mathbb{R}^{3\times 3}$ summarizes the spatial relationship of neighborhood heavy atoms along the sequence, including two directional vectors by the coordinates of the $C_{\alpha}$ atoms (${\bm{N}}_{V,1}^{i}=C_{\alpha_{i+1}}-C_{\alpha_{i}}$; ${\bm{N}}_{V,2}^{i}=C_{\alpha_{i-1}}-C_{\alpha_{i}}$) and a tetrahedral geometry unit vector

where $\mathbf{n}=N_{i}-C_{\alpha_{i}}$ and $\mathbf{c}=C_{i}-C_{\alpha_{i}}$.

Similarly, on the edge of two connected nodes from $v_{i}$ to $v_{j}$, we define edge features ${\bm{H}}_{e}^{ij}$ by scalar features and vector features. The scalar feature ${\bm{E}}_{S}^{ij}\in\mathbb{R}^{32}$ concatenates the radial basis functions (RBF) representations ¹¹1We use 16 Gaussian radial basis functions with centers evenly spaced between 0 and 20Å. of $\|C_{\alpha_{j}}-C_{\alpha_{i}}\|_{2}$ and sinusoidal positional encoding ²²2We use the positional encoding method described in Transformer [22]. of the relative Euclidean distance between $v_{i}$ and $v_{j}$. The vector feature ${\bm{E}}_{V}^{ij}\in\mathbb{R}^{3}$ is defined by the direction of $C_{\alpha_{i}}-C_{\alpha_{j}}$.

ProtLOCA implements geometric vector perceptrons (GVP) [8] to extract scalar and vector features from the nodes and edges of protein graphs. For an arbitrary protein graph ${\mathcal{G}}$, a GVP layer computes embeddings for the scalar feature ${\bm{H}}_{S}$ and the vector feature ${\bm{H}}_{V}$, i.e.,

The key to a $\rm GVP(\cdot)$ layer is composed of multiple iterations of scalar-vector propagations, defined in (1). At the $(\ell+1)$th ($0\leq\ell<L$) iteration,

Here ${\bm{W}}_{1},{\bm{W}}_{1},{\bm{W}}_{3},{\bm{W}}_{4}$ and ${\bm{b}}$ are learnable parameters for this layer, $\odot$ denote row-wise multiplication, ${\rm norm}(\cdot)$ denotes row-wise ${\mathbb{L}}_{2}$ normalization, and $\sigma(\cdot)$ represents the sigmoid activation function. At $\ell=0$, the initial input $({\bm{H}}_{S}^{(0)},{\bm{H}}_{V}^{(0)})=({\bm{H}}_{S},{\bm{H}}_{V})$. At the last layer when $\ell+1=L$, it outputs $({\bm{H}}_{S}^{\prime},{\bm{H}}_{V}^{\prime})=({\bm{H}}_{S}^{(L)},{\bm{H}}_{V}^{(L)})$. We set $L=3$ in each of the $\rm GVP(\cdot)$ layers.

The separately encoded scalar and vector representations $({\bm{H}}_{S}^{\prime},{\bm{H}}_{V}^{\prime})$, before sending to further prediction, are combined to obtain an AA-level matrix representation. This requires additional transformations, which we define as a GVP Transform layer. As introduced below, we first define a concatenated feature ${\bm{H}}={\rm concat}({\bm{H}}_{S}^{\prime},{\bm{H}}_{V}^{\prime})$. For the $i$th node and the edge of connected nodes $i\rightarrow j$, we define:

where each feature vector $\mathbf{h}$ is a concatenation of scalar features ${\bm{H}}_{S}^{\prime}$ and vector features ${\bm{H}}_{V}^{\prime}$, $k$ is the number of incoming messages from $i$’s neighbors, and ${\bm{h}}_{v}^{i}$ and ${\bm{h}}_{m}^{ij}$ are the embedding of scalars and vectors for node $i$ and edge $i\to j$.

We also add an extra feed-forward layer when updating the node representation

where ${\rm GVP}^{\prime}$ denotes a GVP layer with $L=2$. We use superscripts to distinguish it from the previous GVP layers in (1) and (3), which includes three layers of scalar-vector propagations defined in (2). In comparison, ${\rm GVP}^{\prime}$ in (4) only applies 2 layers of scalar-vector propagation.

The stack of GVP convolution and feed-forward transformation defined in (1)-(4) constructs a GVP-GNN block. The block is repeated multiple times to obtain expressive node representations. The feed-forward layer is applied at the end of every GVP-GNN block except for the last block. In implementation, we set the reputation to $6$. See ablation studies in Section IV) for more details.

The node representation is sent to readout layers for label prediction. In the training phase, a dense layer is employed to recovery the input tokens:

For prediction tasks, i.e., global structure matching, we obtain vector representation for the input protein with an normalized average pooling layer, i.e.,

To measure the similarity of a protein pair $({\mathcal{P}}_{1},{\mathcal{P}}_{2})$ with the respective learned vector representations $({\bm{h}}_{1},{\bm{h}}_{2})$, we define the cosine similarity:

Training ProtLOCA only involves the scalar and vector features extracted from backbone coordinates, excluding inputs directly related to amino acid types. The model is trained in a self-supervised learning manner with the objective of denoising the perturbed node features. Two types of corruption approaches are considered for adding noise, including masking and permutation. In the former, values in ${\bm{N}}_{S}$ are set to $0$ with a probability $p$; in the latter, values in ${\bm{N}}_{S}$ are randomly replaced by another ${\bm{N}}_{S}$ value with a probability of $1-p$. Additional discussion on the tunable parameter $p$ can be found in Section IV.

Consider two arbitrary peptide chains ${\mathcal{P}}_{1}$ and ${\mathcal{P}}_{3}$ with different sequence length and overall structure and a common substructure ${\mathcal{P}}^{\prime}$. An local structure alignment task aims to identify highly similar local regions ${\mathcal{P}}^{\prime}$ in the input data $({\mathcal{P}}_{1},{\mathcal{P}}_{3})$.

In the global structure matching task, we employ average pooling on the amino acid representations of the protein pairs to obtain vector representations for comparing protein-level similarity. However, this simplified method cannot provide insights into the alignment of protein local regions. While functionally similar proteins may only have similar active regions and differ in overall structure, discovering local alignments of proteins could be essential for functional region identification and analysis. To this end, we introduce a modified ProtLOCA with a simple heuristic algorithm to highlight similar regions for protein pairs. After extracting the hidden representation for nodes by (4), we conduct the following three steps for local alignment identification.

We construct CATH-aligns, a new benchmark with standard quantitative evaluation criteria. We process the dataset from CATH 4.3 ³³3Official dataset can be found at http://download.cathdb.info/cath/releases/all-releases/v4_3_0/, a comprehensive dataset with experimentally determined protein domain structures. All structures are labeled with a four-level CATH classification code [25] that classifies the protein’s structural type from different perspectives. We remove incomplete protein entities that include missing atomic coordinates for $C_{\alpha}$ and $N$. All proteins are below $20\%$ of sequence identity to each other. A total of $14,654$ are left for constructing the independent test set CATH-aligns.

For structure alignment prediction, we define a binary classification task with the split test subset from CATH4.3. We consider two levels of classification difficulty and name them as CATH-aligns and CATH-aligns+, respectively. The former CATH-aligns defines negative pairs as protein domains with all the four-level CATH classification codes being different and positive pairs as any of the four codes being identical. The latter CATH-aligns+ defines a more difficult task, where structure pairs with identical CATH codes at all four levels are considered positive sample pairs, while pairs differing at any level are considered negative sample pairs. To ensure computational efficiency and balance the number of positive and negative samples, we prepare three folds for evaluation, each containing $10,000$ positive and $10,000$ negative pairs that are randomly sampled from the complete $14654\times 14654$ pairs of CATH-aligns. The prediction results are assessed using the AUC (area under the curve) metric, where an AUC closer to $1$ indicates better predictive performance.

The helix-turn-helix (HTH) motif is a crucial structural component in DNA binding proteins, including transcription factors regulating gene expression [18]. It comprises two alpha-helices joined by a ‘turn’, with the second helix, known as the recognition helix (Fig. 4A), specifically interacting with DNA (Fig. 4B). This interaction is essential for gene regulation, as it enables proteins containing the HTH motif to control the transcription process by attaching to DNA’s promoters or operators [27]. We first use TM-align to identify the HTH motif in the phage lambda cII protein (Fig. 4C) [28] and the transcriptional regulator PA2196 (Fig. 4D) [29]. In the phage lambda cII protein, the HTH motif identified by TM-align is located differently in protein compared to its position in the ground truth. For transcriptional regulator PA2196, the HTH motif identified by TM-align is much shorter than the one in the ground truth. These cases demonstrate TM-align’s limitations in accurately identifying the correct HTH motif in DNA binding proteins. However, ProtLOCA can effectively identify the correct HTH motif in these two proteins, despite their different overall folds. Thus, ProtLOCA demonstrates better performance than TM-align in identifying critical motifs in proteins with the same functions when their overall structures vary.

With the growth of protein sequences and advancements in natural language modeling methods, the most commonly used approaches in protein representation learning typically involve unsupervised training on protein sequences, without considering protein structural information. For example, ESM2 [23], ESM-1v [30], and ESM-1b [31] use different redundancy levels of the Uniref dataset [32], employing the BERT [33] architecture and a masked language modeling unsupervised training objective to train models for downstream tasks related to representation learning or zero-shot mutation tasks in protein engineering. ProtTrans [34] has introduced a series of protein language representation models, such as ProtBert, ProtT5, and ProtAlBert, based on BERT [33], T5 [35] or AlBert [36] architectures, primarily applied to various downstream tasks of representation learning. Ankh [37] uses an asymmetric encoder-decoder approach and explores a series of training parameters to train language models that perform well on downstream tasks. Additionally, methods like CARP [38] and ProteinBert [39] use 1D-CNN instead of the attention mechanism to improve training efficiency for processing longer sequences.

With the increase in crystal structures and advancements in folding techniques [40, 23], protein structure databases have become increasingly large [41]. Currently, mainstream methods use sequence information as the training target for structural inputs or as auxiliary node features, with few models considering only protein structures while discarding amino acid types. For instance, GearNet [10] uses contrastive learning to enhance representation quality for protein enzyme commission (EC) number prediction, and ProtLGN [42] employs multi-task learning and denoising training objectives to improve zero-shot prediction capabilities for protein mutations. Additionally, models like GVP [8] and EGNN [43] use graph neural networks to model the equivariance and invariance of proteins for protein quality prediction tasks. Some inverse folding methods use protein structures as input to restore amino acid information, achieving structure-aware training. For example, ESM-IF [14] uses GVP to initialize transformer node features, and ProstT5 [24] uses Foldseek’s structural tokens as input and amino acid sequences as output (or vice versa) for machine translation training. Furthermore, some approaches combine language models and graph neural networks to enhance the quality of representation learning. Examples include MIF-ST [15], which integrates CARP [38] and Struct GNN, ProtSSN [11], which combines ESM2-650M and EGNN structures, and LM-GVP [12].