CN111190537B - Method and system for managing sequential storage disk in additional writing scene - Google Patents
Method and system for managing sequential storage disk in additional writing scene Download PDFInfo
- Publication number
- CN111190537B CN111190537B CN201911278697.8A CN201911278697A CN111190537B CN 111190537 B CN111190537 B CN 111190537B CN 201911278697 A CN201911278697 A CN 201911278697A CN 111190537 B CN111190537 B CN 111190537B
- Authority
- CN
- China
- Prior art keywords
- block
- chunk
- chunkmeta
- disk
- list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0674—Disk device
- G06F3/0676—Magnetic disk device
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The invention discloses a method and a system for managing sequential storage disks in an additional writing scene, wherein the method comprises the following steps: firstly, defining a disk structure, initializing a disk, applying for chunkServer for chunk when data is written for the first time, and initializing chunkMeta information of chunk; applying a block for the chunk at the same time of applying for the chunk, wherein the block is used as a starting block of the chunk; converting the data offset and size of the chunk into a write offset and size within the block; when the size of the written data exceeds the size of one block, a new block is allocated to the ChunkServer; when the block is successfully allocated, the id of the block is written into the ordered list block_list of ChunkMeta, and meanwhile, the id of the next block is written into the last data segment of the previous block, so that the disc dropping operation is completed. The invention can effectively manage the bare disc in a block mode, flexibly define the size of the block and improve the storage speed by using an additional writing method.
Description
Technical Field
The invention relates to the technical field of disk management, in particular to a method and a system for managing sequential storage disks in an additional writing scene.
Background
With the advent of storage media such as SSD (Solid State Disk), disk IO efficiency is greatly broken through, and a storage mode has become the key point of cloud storage bottleneck optimization. Sequential reading and writing is an important way to increase the storage speed, and additional writing is a way to implement sequential reading and writing. In the prior art, for single-machine disk data management, two modes are generally adopted, one is to use a local file system for data management; another method is to perform data management on a data block, divide a disk into blocks of a fixed size (commonly referred to as chunk) at the time of system initialization, allocate a plurality of chunk (minimum storage unit provided by a distributed storage system to an external opening) on the disk by an algorithm to store a file when the system stores the file data, record the correspondence between chunk and the file, and finally persist the correspondence to a data disk or a database.
The main defects of the existing single-machine disk processing technology in the distributed storage scene are as follows:
1. if the setting of the chunk is too large, the system allocates a fixed size to each chunk in advance, and when storing small files, great waste is caused. For example, for a 1G chunk, 1GB is not actually written, and still 1GB of space is occupied.
2. If the setting of the chunk is too small, the system caches the context information of the chunk for fast indexing of the chunk, so that the chunk context information occupies more memory space, metadata needs to be persisted once for each chunk allocation, and the cost of persisting operation is high, so that the system is not suitable for a high-performance storage system.
3. If the local file system is used for storage, reading and writing of file metadata are needed on an IO path, so that writing amplification is caused, a large amount of CPU and disk IO resources are consumed, and the local file system has the risk of losing data in a power failure in some scenes.
4. The data file storage position is maintained by the file, and when the metadata is lost, the data storage position cannot be recovered.
Therefore, how to effectively manage the sequential storage disk in the additional writing scene is a problem to be solved urgently.
Disclosure of Invention
In view of this, the present invention provides a method for managing sequential storage disks in an additional write scenario, which can effectively manage bare disks in a block manner, flexibly define the size of a block, and increase the storage speed by using an additional write method.
The invention provides a method for managing sequential storage disks under an additional writing scene, which comprises the following steps:
dividing the disk into DiskMeta, chunkMeta and BlockData; wherein, the BlockData consists of disk blocks with fixed sizes, and is distinguished by physical disk offset;
the method comprises the steps of loading DiskMeta and ChunkMeta data information from a disk;
based on the initial block_id in the ChunkMeta, loading all block_ids of the Chunk into an ordered list block_list of the ChunkMeta according to blocks;
scanning the ordered list block_list of the ChunkMeta to obtain an idle block queue and a block allocation bitmap block_bit_map of the block;
when data is written for the first time, chunkServer applies for chunk, and ChunkMeta information of the chunk is initialized;
applying a block for the chunk at the same time of applying for the chunk, wherein the block is used as a starting block of the chunk;
converting the data offset and size of the chunk into a write offset and size within the block;
when the size of the written data exceeds the size of one block, a new block is allocated to the ChunkServer;
when the block allocation is successful, writing the id of the block into the ordered list block_list of the ChunkMeta, and simultaneously writing the id of the next block into the last data segment of the previous block to finish the disc dropping operation.
Preferably, when the write data size exceeds the size of one block, allocating a new block to the ChunkServer includes:
judging whether the free block queue free_block_list is empty, if yes, returning a state code of being full of the disk, and if not, returning the state code of being full of the disk:
acquiring a first block from the free block queue free_block_list;
judging whether the block is used or not through an allocation bitmap block_bit_map of the block, and if not, judging whether the block is used or not:
the block is successfully allocated, and the allocation bit map block_bit_map of the block is allocated at the position 1.
Preferably, the method further comprises:
calculating the number of blocks of the read data on the chunk according to the offset on the chunk;
based on a calculation result, acquiring the id of a block in an ordered list block_list in the ChunkMeta;
calculating an offset address on a disk according to the id of the block, and reading data according to the offset;
and judging whether the size of the read data exceeds the current block, if not, finishing the reading of the returned data, and if so, acquiring the next block from the ordered list block_list in the ChunkMeta.
Preferably, the method further comprises:
deleting ChunkMeta metadata;
and releasing the block when deleting the ChunkMeta metadata is successful.
Preferably, when deleting the ChunkMeta metadata is successful, releasing the block includes:
adding the id of the block to the end of a free block queue free_block_list;
and setting the allocation position of the allocation bitmap block_bit_map of the block to 0.
A system for sequential storage disk management in an append write scenario, comprising:
the definition module is used for dividing the disk into DiskMeta, chunkMeta and BlockData; wherein, the BlockData consists of disk blocks with fixed sizes, and is distinguished by physical disk offset;
the first loading module is used for loading DiskMeta and ChunkMeta data information from the disk;
the second loading module is used for loading all block_ids of the chunk into an ordered list block_list of the chunkMeta according to the block based on the initial block_id in the chunkMeta;
the scanning module is used for scanning the ordered list block_list of the ChunkMeta to obtain an idle block queue and a block allocation bitmap block_bit_map of the block;
the initialization module is used for applying for the chunkServer for chunk and initializing chunkMeta information of the chunk when the data is written for the first time;
the application module is used for applying a block for the chunk at the same time of applying for the chunk, and the block is used as a starting block of the chunk;
the conversion module is used for converting the data offset and the size of the chunk into the write offset and the size in the block;
the distribution module is used for distributing a new block to the ChunkServer when the size of the written data exceeds the size of one block;
and the writing module is used for writing the id of the block into the ordered list block_list of the ChunkMeta when the block is successfully allocated, and simultaneously writing the id of the next block into the last data segment of the previous block to finish the disc dropping operation.
Preferably, the allocation module, when executing the allocation of a new block to the ChunkServer when the size of the write data exceeds the size of one block, includes:
a first judging unit, configured to judge whether the free block queue free_block_list is empty;
the return unit is used for returning the state code of the disk full when the free block queue free_block_list is empty;
an obtaining unit, configured to obtain a first block from an idle block queue free_block_list when the idle block queue free_block_list is not empty;
a second judging unit, configured to judge whether the block is used by using a block_bit_map of an allocation bitmap of the block;
and the setting unit is used for successfully distributing the block when judging that the block is not used through the distribution bitmap block_bit_map of the block, and setting the distribution position of the distribution bitmap block_bit_map of the block to be 1.
Preferably, the system further comprises:
the computing module is used for computing the blocks of the read data on the chunk according to the offset on the chunk;
the first acquisition module is used for acquiring the id of a block in an ordered list block_list in the ChunkMeta based on a calculation result;
the reading module is used for calculating an offset address on the disk according to the id of the block and reading data according to the offset;
the judging module is used for judging whether the size of the read data exceeds the current block;
the return module is used for finishing the return data when the size of the read data does not exceed the current block;
and the second acquisition module is used for acquiring the next block from the ordered list block_list in the ChunkMeta when the read data size does not exceed the current block.
Preferably, the system further comprises:
the deleting module is used for deleting the ChunkMeta metadata;
and the releasing module is used for releasing the block when the ChunkMeta metadata is successfully deleted.
Preferably, the release module is specifically configured to:
adding the id of the block to the end of a free block queue free_block_list;
and setting the allocation position of the allocation bitmap block_bit_map of the block to 0.
In summary, the invention discloses a method for managing sequentially stored disks in an additional write scene, when the sequentially stored disks in the additional write scene are required to be managed, disk data information of DiskMeta and ChunkMeta are loaded from the disks, and then all block_ids of the chunk are loaded into an ordered list block_list of the ChunkMeta according to blocks based on a starting block_id in the ChunkMeta; scanning an ordered list block_list of ChunkMeta to obtain an idle block queue and an allocation bitmap block_bit_map of the block; when data is written for the first time, chunkServer applies for chunk, and ChunkMeta information of the chunk is initialized; applying a block for the chunk at the same time of applying for the chunk, wherein the block is used as a starting block of the chunk; converting the data offset and size of the chunk into a write offset and size within the block; when the size of the written data exceeds the size of one block, a new block is allocated to the ChunkServer; when the block allocation is successful, writing the id of the block into the ordered list block_list of the ChunkMeta, and simultaneously writing the id of the next block into the last data segment of the previous block to finish the disc dropping operation. The invention can effectively manage the bare disc in a block mode, flexibly define the size of the block and improve the storage speed by using an additional writing method.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for managing sequential storage disks in an append write scenario according to embodiment 1 of the present disclosure;
FIG. 2 is a flowchart of a method for managing sequential storage disks in an append write scenario according to embodiment 2 of the present invention;
FIG. 3 is a flowchart of a method for managing sequential storage disks in an append write scenario according to embodiment 3 of the present invention;
FIG. 4 is a schematic diagram of a system embodiment 1 for sequential storage disk management in an append write scenario according to the present disclosure;
FIG. 5 is a schematic diagram of a system embodiment 2 for sequential storage disk management in an append write scenario according to the present disclosure;
fig. 6 is a schematic structural diagram of a system embodiment 3 of sequential storage disk management in an append write scenario according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, a flowchart of a method of embodiment 1 of a method for sequential storage disk management in an append write scenario according to the present disclosure may include the following steps:
s101, dividing a disk into DiskMeta, chunkMeta and BlockData; wherein, the BlockData consists of disk blocks with fixed sizes, and is distinguished by physical disk offset;
when it is required to manage the sequentially stored magnetic disk in the additional write scene, in this embodiment, the magnetic disk is divided into a layered magnetic disk metadata segment (DiskMeta), a chunk raw data segment (ChunkMetas) and a BlockData segment; wherein, blockData is composed of fixed-size disk blocks, and distinguished by disk physical offsets. Specifically, a file is composed of multiple blocks, a block is composed of multiple blocks, an ordered list (block_list) of block data to be stored in the block, and each block data is represented by one block id in turn.
Wherein, chunk is a custom name, which represents the minimum storage unit provided by the distributed storage system to the outside; block is a custom name, which means that a disk is uniformly divided into N blocks on a single machine of a distributed storage system, and each Block is called a Block. The BlockData is a custom name, and a data Block representing the Block is called BlockData; diskMeta is metadata information for storing a disk, and includes the size of the disk, the number of blocks in the disk, and the like. ChunkMeta is the meta information that stores chunk, including index of chunk, length that chunk has used, length that can be used, id of block of disk that has been allocated.
S102, loading DiskMeta and ChunkMeta data information from a disk;
then, the disk is initialized according to the preset DiskMeta size, chunkMeta size and BlockData size. Specifically, when initializing a disk, disk data information of DiskMeta and ChunkMeta is first loaded from the disk.
S103, based on the initial block_id in the ChunkMeta, loading all block_ids of the chunks into an ordered list block_list of the ChunkMeta according to the blocks;
then, all block_ids of the chunk are loaded into the ordered list block_list of chunkMeta in terms of the starting block_id in chunkMeta in terms of blocks.
S104, scanning an ordered list block_list of ChunkMeta to obtain an idle block queue and an allocation bitmap block_bit_map of the block;
then, the ordered list block_list of ChunkMeta is scanned to obtain a free Block queue (free_block_list) and a Block allocation bitmap. Wherein the Block allocation bitmap is a two-byte bitmap block_bit_map, and whether the Block is used is identified by 0, 1.
S105, when data is written for the first time, the ChunkServer applies for chunk, and the ChunkMeta information of the chunk is initialized;
in the process of data writing, when writing for the first time, the ChunkMeta information of the Chunk is initialized through ChunkServer application chunk. Wherein Chunkserver is a custom name, is a service for managing chunk, chunkMeta, block, and is responsible for receiving and processing file storage requests of clients.
S106, applying a block for the chunk at the same time of applying for the chunk, and taking the block as a starting block of the chunk;
then, at the same time as the chunk is applied, a block needs to be applied for the chunk, and the block is used as the initial block of the chunk.
S107, converting the data offset and the size of the chunk into the write offset and the size in the block;
the data offset and size of the chunk is then converted to a write offset and size within the block.
S108, when the size of the written data exceeds the size of one block, distributing a new block to the ChunkServer;
if the write size exceeds the size of one block, the ChunkServer will attempt to allocate a new block to ChunkServer.
S109, when the block allocation is successful, writing the id of the block into the ordered list block_list of ChunkMeta, and simultaneously writing the id of the next block into the last data segment of the previous block, so as to complete the disc dropping operation.
If the block allocation is successful, the id of the block is written into the block_list of ChunkMeta, and meanwhile, the id of the next block is written into the last data segment of the previous block. And then, completing the tray dropping operation.
To sum up, in the above embodiment, when the disk sequentially stored in the additional write scene needs to be managed, the diskMeta and ChunkMeta data information are loaded from the disk first, and then all the block_ids of the chunk are loaded into the ordered list block_list of ChunkMeta according to the block based on the initial block_id in ChunkMeta; scanning an ordered list block_list of ChunkMeta to obtain an idle block queue and an allocation bitmap block_bit_map of the block; when data is written for the first time, chunkServer applies for chunk, and ChunkMeta information of the chunk is initialized; applying a block for the chunk at the same time of applying for the chunk, wherein the block is used as a starting block of the chunk; converting the data offset and size of the chunk into a write offset and size within the block; when the size of the written data exceeds the size of one block, a new block is allocated to the ChunkServer; when the block allocation is successful, writing the id of the block into the ordered list block_list of the ChunkMeta, and simultaneously writing the id of the next block into the last data segment of the previous block to finish the disc dropping operation. The invention can effectively manage the bare disc in a block mode, flexibly define the size of the block and improve the storage speed by using an additional writing method.
Specifically, in the above embodiment 1, when the size of the write data exceeds the size of one block, one implementation of allocating a new block to ChunkServer is shown in fig. 2, and may include the following steps:
s201, judging whether the free block queue free_block_list is empty, if yes, entering S202, otherwise entering S203:
at the time of allocating new blocks
S202, returning a state code of full disk;
s203, acquiring a first block from an idle block queue free_block_list;
s204, judging whether the block is used or not through an allocation bitmap block_bit_map of the block, and if not, entering S205:
s205, the block is successfully allocated, and the allocation bit map block_bit_map of the block is allocated at the position 1.
Specifically, the processing of data may be further implemented on the basis of the above embodiment 1, as shown in fig. 3, and may include the following steps:
s301, calculating the blocks of the read data on the chunk according to the offset on the chunk;
s302, acquiring the id of a block in an ordered list block_list in ChunkMeta based on a calculation result;
s303, calculating an offset address on a disk according to the id of the block, and reading data according to the offset;
s304, judging whether the read data size exceeds the current block, if not, entering S305, if yes, entering S306:
s305, reading end return data;
s306, acquiring a next block from an ordered list block_list in ChunkMeta;
s307, deleting ChunkMeta metadata;
s308, releasing the block when the ChunkMeta metadata is deleted successfully.
Specifically, the release of the block can be realized by adding the id of the block to the end of the free block queue free_block_list and setting the allocation position of the allocation bitmap block_bit_map of the block to 0.
The method for managing sequential storage disks in the additional writing scene provided by the invention can be applied to disk storage media including but not limited to storage media such as SSD and HDD.
In summary, the invention writes the id of the next block into the end of the data of the last block, thereby reducing the number of ChunkMeta persistence operations. Meanwhile, the data storage position can be restored under the condition that ChunkMeta metadata is lost. The free blocks are managed through the block_bit_map and the free_block_list, so that the blocks can be rapidly allocated and recovered, the context information of block management is simplified, and the space of cache management is reduced. The size of the block can be flexibly defined, and the disk space waste of block management in a scene of sequential storage is reduced. Under the condition of additional reading and writing, the disk blocks with flexibly defined sizes can be dynamically allocated for users to use.
As shown in fig. 4, a schematic structural diagram of a system embodiment 1 of sequential storage disk management in an append write scenario according to the present invention is disclosed, where the system may include:
a definition module 401, configured to divide a disk into DiskMeta, chunkMeta and BlockData; wherein, the BlockData consists of disk blocks with fixed sizes, and is distinguished by physical disk offset;
when it is required to manage the sequentially stored magnetic disk in the additional write scene, in this embodiment, the magnetic disk is divided into a layered magnetic disk metadata segment (DiskMeta), a chunk raw data segment (ChunkMetas) and a BlockData segment; wherein, blockData is composed of fixed-size disk blocks, and distinguished by disk physical offsets. Specifically, a file is composed of multiple blocks, a block is composed of multiple blocks, an ordered list (block_list) of block data to be stored in the block, and each block data is represented by one block id in turn.
Wherein, chunk is a custom name, which represents the minimum storage unit provided by the distributed storage system to the outside; block is a custom name, which means that a disk is uniformly divided into N blocks on a single machine of a distributed storage system, and each Block is called a Block. The BlockData is a custom name, and a data Block representing the Block is called BlockData; diskMeta is metadata information for storing a disk, and includes the size of the disk, the number of blocks in the disk, and the like. ChunkMeta is the meta information that stores chunk, including index of chunk, length that chunk has used, length that can be used, id of block of disk that has been allocated.
A first loading module 402, configured to load DiskMeta and ChunkMeta data information from a disk;
then, the disk is initialized according to the preset DiskMeta size, chunkMeta size and BlockData size. Specifically, when initializing a disk, disk data information of DiskMeta and ChunkMeta is first loaded from the disk.
A second loading module 403, configured to load all block_ids of the chunk into an ordered list block_list of ChunkMeta according to a block based on a starting block_id in ChunkMeta;
then, all block_ids of the chunk are loaded into the ordered list block_list of chunkMeta in terms of the starting block_id in chunkMeta in terms of blocks.
A scanning module 404, configured to scan the ordered list block_list of ChunkMeta to obtain an idle block queue and a block allocation bitmap block_bit_map of the block;
then, the ordered list block_list of ChunkMeta is scanned to obtain a free Block queue (free_block_list) and a Block allocation bitmap. Wherein the Block allocation bitmap is a two-byte bitmap block_bit_map, and whether the Block is used is identified by 0, 1.
An initialization module 405, configured to apply for ChunkServer for chunk when writing data for the first time, and initialize chunkMeta information of chunk;
in the process of data writing, when writing for the first time, the ChunkMeta information of the Chunk is initialized through ChunkServer application chunk. Wherein Chunkserver is a custom name, is a service for managing chunk, chunkMeta, block, and is responsible for receiving and processing file storage requests of clients.
An application module 406, configured to apply a block for the chunk as a starting block of the chunk while applying for the chunk;
then, at the same time as the chunk is applied, a block needs to be applied for the chunk, and the block is used as the initial block of the chunk.
A conversion module 407, configured to convert the data offset and size of the chunk into a write offset and size in the block;
the data offset and size of the chunk is then converted to a write offset and size within the block.
An allocation module 408, configured to allocate a new block to the ChunkServer when the size of the write data exceeds the size of a block;
if the write size exceeds the size of one block, the ChunkServer will attempt to allocate a new block to ChunkServer.
And a writing module 409, configured to write the id of a block into the ordered list block_list of the ChunkMeta when the block allocation is successful, and simultaneously write the id of the next block into the last data segment of the previous block, so as to complete the disc dropping operation.
If the block allocation is successful, the id of the block is written into the block_list of ChunkMeta, and meanwhile, the id of the next block is written into the last data segment of the previous block. And then, completing the tray dropping operation.
To sum up, in the above embodiment, when the disk sequentially stored in the additional write scene needs to be managed, the diskMeta and ChunkMeta data information are loaded from the disk first, and then all the block_ids of the chunk are loaded into the ordered list block_list of ChunkMeta according to the block based on the initial block_id in ChunkMeta; scanning an ordered list block_list of ChunkMeta to obtain an idle block queue and an allocation bitmap block_bit_map of the block; when data is written for the first time, chunkServer applies for chunk, and ChunkMeta information of the chunk is initialized; applying a block for the chunk at the same time of applying for the chunk, wherein the block is used as a starting block of the chunk; converting the data offset and size of the chunk into a write offset and size within the block; when the size of the written data exceeds the size of one block, a new block is allocated to the ChunkServer; when the block allocation is successful, writing the id of the block into the ordered list block_list of the ChunkMeta, and simultaneously writing the id of the next block into the last data segment of the previous block to finish the disc dropping operation. The invention can effectively manage the bare disc in a block mode, flexibly define the size of the block and improve the storage speed by using an additional writing method.
Specifically, in the embodiment 1 of the system, when the size of the write data exceeds the size of one block, the allocation module allocates a new block to the ChunkServer, as shown in fig. 5, may include:
a first determining unit 501, configured to determine whether the free block queue free_block_list is empty;
a returning unit 502, configured to return a state code that the disk is full when the free block queue free_block_list is empty;
an obtaining unit 503, configured to obtain, when the free block queue free_block_list is not empty, a first block from the free block queue free_block_list;
a second judging unit 504, configured to judge whether the block is used by using the allocation bitmap block_bit_map of the block;
a setting unit 505, configured to, when it is determined that the block is not used by using the allocation bitmap block_bit_map of the block, successfully allocate the block, and set the allocation position of the allocation bitmap block_bit_map of the block to 1.
Specifically, the processing of data may be further implemented on the basis of the system embodiment 1, as shown in fig. 6, which may include:
a calculating module 601, configured to calculate, according to the offset on the chunk, what block of the chunk the read data is on;
a first obtaining module 602, configured to obtain, based on a calculation result, an id of a block in an ordered list block_list in ChunkMeta;
a reading module 603, configured to calculate an offset address on the disk according to the id of the block, and read data according to the offset;
a judging module 604, configured to judge whether the read data size exceeds the current block;
a return module 605, configured to, when the read data size does not exceed the current block, end reading the returned data;
a second obtaining module 606, configured to obtain, when the read data size does not exceed the current block, a next block from the ordered list block_list in ChunkMeta;
a deletion module 607 for deleting ChunkMeta metadata;
a releasing module 608, configured to release the block when the ChunkMeta metadata is deleted successfully.
Specifically, the releasing module may add the id of the block to the end of the free block queue free_block_list, and place 0 in the allocation position of the allocation bitmap block_bit_map of the block, so as to realize the release of the block.
The method for managing sequential storage disks in the additional writing scene provided by the invention can be applied to disk storage media including but not limited to storage media such as SSD and HDD.
In summary, the invention writes the id of the next block into the end of the data of the last block, thereby reducing the number of ChunkMeta persistence operations. Meanwhile, the data storage position can be restored under the condition that ChunkMeta metadata is lost. The free blocks are managed through the block_bit_map and the free_block_list, so that the blocks can be rapidly allocated and recovered, the context information of block management is simplified, and the space of cache management is reduced. The size of the block can be flexibly defined, and the disk space waste of block management in a scene of sequential storage is reduced. Under the condition of additional reading and writing, the disk blocks with flexibly defined sizes can be dynamically allocated for users to use.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for sequential storage disk management in an append write scenario, comprising:
dividing the disk into a layer disk metadata segment DiskMeta, chunk original data segment ChunkMeta and BlockData; wherein, the BlockData consists of disk blocks with fixed sizes, and is distinguished by physical disk offset; the BlockData is a custom name, and a data Block representing Block is called BlockData; the DiskMeta is metadata information for storing a disk and at least comprises the size of the disk and the number of blocks in the disk; the ChunkMeta is the meta information of the stored chunk, and comprises index of the chunk, used length of the chunk, usable length and id of the block of the allocated disk; the chunk is a custom name and represents a minimum storage unit provided by the distributed storage system for external opening; the block is a custom name, and represents that a single machine of a distributed storage system can divide a magnetic disk into N blocks uniformly, and each block is called a block;
the method comprises the steps of loading DiskMeta and ChunkMeta data information from a disk;
based on the initial block_id in the ChunkMeta, loading all block_ids of the Chunk into an ordered list block_list of the ChunkMeta according to blocks; wherein the block_id represents the id of the block of the allocated disk;
scanning the ordered list block_list of the ChunkMeta to obtain an idle block queue and a block allocation bitmap block_bit_map of the block;
when data is written for the first time, chunkServer applies for chunk, and ChunkMeta information of the chunk is initialized; wherein, the ChunkServer is a custom name, is a service for managing chunk, chunkMeta, block, and is responsible for receiving and processing the file storage request of clients;
applying a block for the chunk at the same time of applying for the chunk, wherein the block is used as a starting block of the chunk;
converting the data offset and size of the chunk into a write offset and size within the block;
when the size of the written data exceeds the size of one block, a new block is allocated to the ChunkServer;
when the block allocation is successful, writing the id of the block into the ordered list block_list of the ChunkMeta, and simultaneously writing the id of the next block into the last data segment of the previous block to finish the disc dropping operation.
2. The method of claim 1 wherein when the write data size exceeds the size of a block, assigning a new block to ChunkServer comprises:
judging whether the free block queue free_block_list is empty, if yes, returning a state code of being full of the disk, and if not, returning the state code of being full of the disk:
acquiring a first block from the free block queue free_block_list;
judging whether the block is used or not through an allocation bitmap block_bit_map of the block, and if not, judging whether the block is used or not:
the block is successfully allocated, and the allocation bit map block_bit_map of the block is allocated at the position 1.
3. The method as recited in claim 2, further comprising:
calculating the number of blocks of the read data on the chunk according to the offset on the chunk;
based on a calculation result, acquiring the id of a block in an ordered list block_list in the ChunkMeta;
calculating an offset address on a disk according to the id of the block, and reading data according to the offset;
and judging whether the size of the read data exceeds the current block, if not, finishing the reading of the returned data, and if so, acquiring the next block from the ordered list block_list in the ChunkMeta.
4. A method according to claim 3, further comprising:
deleting ChunkMeta metadata;
and releasing the block when deleting the ChunkMeta metadata is successful.
5. The method of claim 4, wherein releasing the block when deleting the ChunkMeta metadata is successful comprises:
adding the id of the block to the end of a free block queue free_block_list;
and setting the allocation position of the allocation bitmap block_bit_map of the block to 0.
6. A system for sequential storage disk management in an append write scenario, comprising:
the definition module is used for dividing the disk into a layer disk metadata segment DiskMeta, chunk original data segment ChunkMeta and BlockData; wherein, the BlockData consists of disk blocks with fixed sizes, and is distinguished by physical disk offset; the BlockData is a custom name, and a data Block representing Block is called BlockData; the DiskMeta is metadata information for storing a disk and at least comprises the size of the disk and the number of blocks in the disk; the ChunkMeta is the meta information of the stored chunk, and comprises index of the chunk, used length of the chunk, usable length and id of the block of the allocated disk; the chunk is a custom name and represents a minimum storage unit provided by the distributed storage system for external opening; the block is a custom name, and represents that a single machine of a distributed storage system can divide a magnetic disk into N blocks uniformly, and each block is called a block;
the first loading module is used for loading DiskMeta and ChunkMeta data information from the disk;
the second loading module is used for loading all block_ids of the chunk into an ordered list block_list of the chunkMeta according to the block based on the initial block_id in the chunkMeta; wherein the block_id represents the id of the block of the allocated disk;
the scanning module is used for scanning the ordered list block_list of the ChunkMeta to obtain an idle block queue and a block allocation bitmap block_bit_map of the block;
the initialization module is used for applying for the chunkServer for chunk and initializing chunkMeta information of the chunk when the data is written for the first time; wherein, the ChunkServer is a custom name, is a service for managing chunk, chunkMeta, block, and is responsible for receiving and processing the file storage request of clients;
the application module is used for applying a block for the chunk at the same time of applying for the chunk, and the block is used as a starting block of the chunk;
the conversion module is used for converting the data offset and the size of the chunk into the write offset and the size in the block;
the distribution module is used for distributing a new block to the ChunkServer when the size of the written data exceeds the size of one block;
and the writing module is used for writing the id of the block into the ordered list block_list of the ChunkMeta when the block is successfully allocated, and simultaneously writing the id of the next block into the last data segment of the previous block to finish the disc dropping operation.
7. The system of claim 6, wherein the allocation module, when executing the allocation of a new block to ChunkServer when the write data size exceeds the size of one block, comprises:
a first judging unit, configured to judge whether the free block queue free_block_list is empty;
the return unit is used for returning the state code of the disk full when the free block queue free_block_list is empty;
an obtaining unit, configured to obtain a first block from an idle block queue free_block_list when the idle block queue free_block_list is not empty;
a second judging unit, configured to judge whether the block is used by using a block_bit_map of an allocation bitmap of the block;
and the setting unit is used for successfully distributing the block when judging that the block is not used through the distribution bitmap block_bit_map of the block, and setting the distribution position of the distribution bitmap block_bit_map of the block to be 1.
8. The system of claim 7, further comprising:
the computing module is used for computing the blocks of the read data on the chunk according to the offset on the chunk;
the first acquisition module is used for acquiring the id of a block in an ordered list block_list in the ChunkMeta based on a calculation result;
the reading module is used for calculating an offset address on the disk according to the id of the block and reading data according to the offset;
the judging module is used for judging whether the size of the read data exceeds the current block;
the return module is used for finishing the return data when the size of the read data does not exceed the current block;
and the second acquisition module is used for acquiring the next block from the ordered list block_list in the ChunkMeta when the read data size does not exceed the current block.
9. The system of claim 8, further comprising:
the deleting module is used for deleting the ChunkMeta metadata;
and the releasing module is used for releasing the block when the ChunkMeta metadata is successfully deleted.
10. The system according to claim 9, wherein the release module is specifically configured to:
adding the id of the block to the end of a free block queue free_block_list;
and setting the allocation position of the allocation bitmap block_bit_map of the block to 0.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911278697.8A CN111190537B (en) | 2019-12-10 | 2019-12-10 | Method and system for managing sequential storage disk in additional writing scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911278697.8A CN111190537B (en) | 2019-12-10 | 2019-12-10 | Method and system for managing sequential storage disk in additional writing scene |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111190537A CN111190537A (en) | 2020-05-22 |
| CN111190537B true CN111190537B (en) | 2023-08-25 |
Family
ID=70707314
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911278697.8A Active CN111190537B (en) | 2019-12-10 | 2019-12-10 | Method and system for managing sequential storage disk in additional writing scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111190537B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035069B (en) * | 2020-09-17 | 2024-02-27 | 上海二三四五网络科技有限公司 | Control method and device for judging whether file is completely dropped |
| CN112540731B (en) * | 2020-12-22 | 2023-08-11 | 北京百度网讯科技有限公司 | Data append writing method, device, equipment, medium and program product |
| CN114579062B (en) * | 2022-04-29 | 2022-08-05 | 深圳市杉岩数据技术有限公司 | Disk optimization method and device based on distributed storage system |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006331076A (en) * | 2005-05-26 | 2006-12-07 | Hitachi Ltd | Data storage system and storage method |
| JP2008140396A (en) * | 2007-12-10 | 2008-06-19 | Sharp Corp | File managing device |
| JP2009116946A (en) * | 2007-11-06 | 2009-05-28 | Hitachi Maxell Ltd | Storage device and data write control method |
| CN102169419A (en) * | 2011-04-02 | 2011-08-31 | 无锡众志和达存储技术有限公司 | RAID (redundant array of independent disks) data block splitting and assembling method based on SATA (serial advanced technology attachment) controller |
| CN102779180A (en) * | 2012-06-29 | 2012-11-14 | 华为技术有限公司 | Operation processing method of data storage system and data storage system |
| CN104035729A (en) * | 2014-05-22 | 2014-09-10 | 中国科学院计算技术研究所 | Block device thin-provisioning method for log mapping |
| CN105630808A (en) * | 2014-10-31 | 2016-06-01 | 北京奇虎科技有限公司 | Distributed file system based file reading and writing method and node server |
| CN108958660A (en) * | 2018-07-02 | 2018-12-07 | 深圳市茁壮网络股份有限公司 | Distributed memory system and its data processing method and device |
| CN109614054A (en) * | 2018-12-20 | 2019-04-12 | 广州鼎甲计算机科技有限公司 | The read method and system of data |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6089844B2 (en) * | 2013-03-22 | 2017-03-08 | 富士通株式会社 | Control device, storage device, and control program |
-
2019
- 2019-12-10 CN CN201911278697.8A patent/CN111190537B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006331076A (en) * | 2005-05-26 | 2006-12-07 | Hitachi Ltd | Data storage system and storage method |
| JP2009116946A (en) * | 2007-11-06 | 2009-05-28 | Hitachi Maxell Ltd | Storage device and data write control method |
| JP2008140396A (en) * | 2007-12-10 | 2008-06-19 | Sharp Corp | File managing device |
| CN102169419A (en) * | 2011-04-02 | 2011-08-31 | 无锡众志和达存储技术有限公司 | RAID (redundant array of independent disks) data block splitting and assembling method based on SATA (serial advanced technology attachment) controller |
| CN102779180A (en) * | 2012-06-29 | 2012-11-14 | 华为技术有限公司 | Operation processing method of data storage system and data storage system |
| CN104035729A (en) * | 2014-05-22 | 2014-09-10 | 中国科学院计算技术研究所 | Block device thin-provisioning method for log mapping |
| CN105630808A (en) * | 2014-10-31 | 2016-06-01 | 北京奇虎科技有限公司 | Distributed file system based file reading and writing method and node server |
| CN108958660A (en) * | 2018-07-02 | 2018-12-07 | 深圳市茁壮网络股份有限公司 | Distributed memory system and its data processing method and device |
| CN109614054A (en) * | 2018-12-20 | 2019-04-12 | 广州鼎甲计算机科技有限公司 | The read method and system of data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111190537A (en) | 2020-05-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10289315B2 (en) | Managing I/O operations of large data objects in a cache memory device by dividing into chunks | |
| US8239648B2 (en) | Reclamation of thin provisioned disk storage | |
| US8095728B2 (en) | Method and system for power aware I/O scheduling | |
| CN111190537B (en) | Method and system for managing sequential storage disk in additional writing scene | |
| US7945587B2 (en) | Random allocation of media storage units | |
| US10338833B1 (en) | Method for achieving sequential I/O performance from a random workload | |
| US11314689B2 (en) | Method, apparatus, and computer program product for indexing a file | |
| CN110858162B (en) | Memory management method and device and server | |
| CN113568582B (en) | Data management method, device and storage equipment | |
| CN108139972B (en) | Method and apparatus for managing memory fragmentation in hardware assisted data compression | |
| US20060136779A1 (en) | Object-based storage device with low process load and control method thereof | |
| US20080005206A1 (en) | Method for automatically managing disk fragmentation | |
| CN115878373A (en) | Resource allocation for synthetic backups | |
| US7644232B2 (en) | Cache method and cache system for storing file's data in memory blocks divided from cache memory | |
| US10387043B2 (en) | Writing target file including determination of whether to apply duplication elimination | |
| CN111444114B (en) | Method, device and system for processing data in nonvolatile memory | |
| CN109144403B (en) | Method and equipment for switching cloud disk modes | |
| US8918621B1 (en) | Block address isolation for file systems | |
| JP4997858B2 (en) | Data recording apparatus and data recording program | |
| CN108334457B (en) | IO processing method and device | |
| CN109508140B (en) | Storage resource management method and device, electronic equipment and system | |
| US9170929B2 (en) | Memory controller | |
| US20210191851A1 (en) | System and method for facilitating reduction of latency and mitigation of write amplification in a multi-tenancy storage drive | |
| CN115878025B (en) | Tree structure node compression priority | |
| JP4204405B2 (en) | Memory management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |