Using Database Schemas of Legacy Applications for Microservices Identification: A Mapping Study

We will use the following digital libraries for searching the related studies based on popularity among users.

Our strategy is to start with a Search query that targets a broader scope of microservices identification strategies/methods/approaches, and then using snowballing technique to find relevant papers that can answer our research questions. The search engines used are presented in Table 1

Search String

"MICROSERVICE IDENTIFICATION" OR "MICROSERVICE DISCOVERY" OR "MICROSERVICE EXTRACTION"

We will not directly include secondary studies in our analysis phase to have a more homogeneous result set, but we will include any relevant to our research context primary studies that are referenced to these studies. Our inclusion and exclusion criteria are presented in table 2.

Our search query was submitted on the 2^th of May 2022 and yielded 241 results. We excluded 21 duplicate results so the number of studies before applying inclusion/exclusion criteria was 220.

After applying inclusion/exclusion criteria, 14 studies were relevant to our research questions. To make sure that we did not miss some relevant study we applied the snowballing method. In this method, each research paper's list of references is examined in the terms of the previously applied inclusion/exclusion criteria [24].

We observe in Figure 1 that the research topic of service discovery from database artefacts is very active, with a regular number of papers each year. Most common type of publication is Conference Papers as we see in Figure 2

Figure 1:Number of included studies per year

Figure 2: Number of included studies per study type

In this section we will discuss the relevant papers discovered in the previous step.

Del grosso et al [10] used a bottom up approach (i.e. from the database toward the user interface in a layered architectural style), capturing SQL query interactions with database by using the JDBC APIs and Formal Concept Analysis in a two-step process to extract and cluster candidate services. This approach produces fine-grained granularity services, and the system must be orchestrated in a specific manner for the steps to be performed. Also, the study due to its date of conduction does not refer to the microservices architectural style but is concerned with services in general.

Baghdadi et al. [4] proposed a method to generate web services interfaces based on the database tables. The main idea was to create a façade for the underlying monolithic system that can be accessed through messaging protocols (HTTP) and create a set of CRUD (Create, Read, Update, Delete) interfaces for each table. These interfaces then need to be implemented by the developer. This approach produces low level wrapper candidate services that ignores the business-logic aspects of the system and the related source code. Intercommunication between services is not taken into account and the study is lacking evaluation.

Amiri et al. [3] proposed a multifaceted service identification process that combined three strategies, a. Business process-driven, b. Goal-Requirements-driven and c. data-driven approach, to find and extract candidate services from a legacy system. This approach is using task-task matrices that are based on each facet and are generated through modeling algorithms that are using business process, goals and data models as input. Weights are applied to each of the created matrices with the help of domain experts and a training dataset, and the three matrices are combined to a single final task-task matrix. A genetic algorithm and a fitness function is used to cluster the elements of the result matrix to candidate services.

This process has a strong rationale, but it was not evaluated in a real-world scenario specifically with a large application. The evaluation provided does not appear to be generalizable. Furthermore, the target system needs to be properly documented for the business-goals-requirements-entities models to be created. This is highly unlikely for a legacy system. Also, microservices are not referenced explicitly although at 2016 they were already present in literature.

[22] proposed a method that was similar to [10]. They used the source code and specifically the persistence layer as input to detect functions that are responsible for database operations with the help of an algorithm. Then, they generated wrapper functions that expose the inner functionality of the target system. These functions must be stateless and to be able to do this, any state dependencies that are not part of the function must be passed as an argument. This can be proven challenging in large legacy applications that are usually monoliths with a global state. Also, to make the functions stateless the source code needs to change. No automation is used in this approach and microservices are not mentioned. Also, the resulted services are at the lowest level (i.e., provide simple CRUD functionality) and to create higher-level business services, these low-level services must be orchestrated in a specific way.

[6] and [7] introduced a semi-automatic procedure to extract candidate microservices from a source system using heuristic algorithms and structural-behavioral analysis, capturing SQL queries and schema analysis to identify relationships between database tables and business object relations. The produced artefacts are clustered and optimized to reduce intercommunications with the use of a genetic algorithm, more specifically NSGA II [9]. The result is a microservices-based new system. This study is much closer to our research questions. The methodology is well documented, and a case study evaluation is performed in 2 open-source projects. However, there are some questions left open regarding the generality of the approach and the level of the automation of the process. Although the process is characterized as automated by the authors, it seems to be semi-automated since it requires considerable manual work on extracting the required artefacts and configuring the genetic algorithm. Furthermore, the evaluation process provides some performance metrics about the new microservices system in a microservice level, but not as a whole integrated system where one microservice can call another.

[14] proposed a method to identify Entity, Task and Process services in a target system using the Master Data Lifecycle and state machine diagrams obtained by reverse engineering the database and/or from system documentation when available. The result is CRUD operations for each type (Entity, Task, Process) that can be grouped resulting to a lower granularity service. There is an absence of implementation details of the case study and also this method is lacking any automation. In addition, there is no reference to the microservices architectural style. The authors later released a tool, that automated some of the process steps but still all the related diagrams must be produced.

[18] introduced a procedure to extract processes from a relational database with the help of entity lifecycle logs. These logs must be mapped manually with the corresponding data tables and data tables are clustered using weighted k-means algorithm to create high level business processes. The connection between tables is measured by their entropy and inner-schema dependencies (e.g., foreign keys) which in many cases of a production system is not accurate and the authors propose some techniques to rediscover foreign keys between them. Furthermore, to create the lifecycle of entities, timestamped attributes must exist in the database.

One of the problems of this approach is that the number of extracted artefacts is given by the user. This can often lead to inconsistent process design and a extensive experimentation is needed, especially in large systems where various domains exist in one schema. Microservices are not mentioned in this study.

[21] introduced a roadmap of a legacy system migration to a new microservices based one, using a hybrid domain-driven approach. This study is an empirical approach to the steps required for an architecture modernization, but it does not provide any implementation details of the actual steps or any tool to support this process. We decided to include this paper in our study because it specifically identifies the database as an input for such a process

[16] proposed a technique for converting a legacy service-oriented system to microservices. The process involves the decomposing of the database into groups of tables of the same business context (subsystems) which, after a call graph analysis, are mapped to existing facades from the old system making the migration to microservices transparent to the end users.

The presentation layer must be decoupled from the business logic for this transparency to work, also the process can be done incrementally and the authors state that the two systems can coexist. They evaluated their technique in a large banking system with promising results. This approach requires deep knowledge of the target domain and the legacy system, and no automation is used in any level.

[2] Proposed a method similar to [3], utilizing semantic and syntactic elements from the source code along with database schema relationships . The derived artefacts are used to identify potential microservices with the use of a clustering algorithm. A case study involving 2 popular open-source software systems was performed. This method is in the same spirit like [3, 6, 18] .

[13] described a process that uses legacy system's store procedures to identify microservices candidates. The authors analyzed the process in depth, and they also applied it in a large legacy system, with promising results. One of the problems with this approach, is that it only considers store-procedure based systems. Also, the proposed process is manual since no automation is involved.

[20] introduced a data-centric process for identifying microservices that uses the database schema artefact names (table names, attribute names), enriched further by semantically related words. This method is close to our research questions. One of the issues is that this approach is strongly coupled to the naming scheme of the targeted system. This can lead to misleading results and the assistance of a system expert is required.

A summarized view of each study and their corresponding artefacts are given on Table 3

RQ1: During the process of re-architecting a legacy system, are data-driven artefacts like database schema and state of data used for identifying potential microservices?

Based to our current research we have found that data-driven elements (such as the database schema or state of data) have been considered in the past as artefacts that can help the re-architecting of a legacy system but rarely as the sole factor [13] [4] [16]

RQ1.1: Can the process of identifying services/microservices from database artefacts be in a complete or partial way automated?

Based to our findings there is no proposed method that can achieve full automation. This is expected due to the complexity of a system migration. We found 9 studies that provide some degree of automation (semi-automation or partial automation), and 5 studies that proposed completely manual processes without tool support (Figure 3). Furthermore, the semi-automatic proposed methods still involve considerable number of manual steps and significant amount of work to parametrize the automation mechanisms.

Figure 3: Comparison of automation degree

RQ2: How data-driven artefacts, like the database, are used in the process of a software migration to services/microservices based architecture?

Based on our findings in the literature we observe that schema analysis and data access analysis from the source code are the most common patterns of utilizing data-driven artefacts in a migration process (Figure 4). Other methods like data mining and SQL transactions runtime analysis are not used often. Source code is needed in almost all of the proposed methods (10 of 14).

Figure 4: Database artefacts used so far for microservices discovery

Half (7/14) of the proposed methods have microservices awareness. With the popularity and the growth of cloud native applications we anticipated more results to be focused on this aspect. We also see a research gap in architecture migration without the use of source code, for example with data-mining techniques.

Furthermore, we can perform data analysis/mining processes to find features or aspects of the application that are not in use. This information cannot be given easily from the source code but only from the domain experts of the supported system with support of the state of data (e.g., through database analysis we can discover unused tables). Excluding not used features of a system presents benefits including reduction of the development time and the corresponding cost of the new system while at the same time making it more streamlined to the business or organization's purposes. In addition, it can also assist with the prioritization of features during the migration process.

Because the inclusion/exclusion of a study is manual executed from each author, there is a possibility to misjudge a study. To mitigate this threat, we prior discussed the process in depth so there is a common understanding of the inclusion/exclusion criteria. Furthermore, during the process of reading the studies, if one author wasn't sure about the inclusion or exclusion of a study, there was a discussion between the authors and the decision was made in agreement of both authors.

Bias or other issue of data extraction can affect the facet classification and further analysis of the selected studies. To reduce this threat, we thoroughly discussed the definition of the data items and agreed in the current form.

We may have missed some studies that were relevant to our subject. To reduce this threat, we searched in the broader topic of microservices identification, and then we applied the method of snowballing to further reduce this possibility.