Bundesarchiv

The German Bundesarchiv is digitizing more and more of its documents and facing considerable challenges in the process. Every year it receives large volumes of digitized data that must be checked for integrity before being incorporated into the archive. The state archives is assisted in this process by outside service providers. In its search for a way to incorporate digitized files more quickly into the archive, the government agency came across FAST LTA’s Silent Bricks. After a particularly short implementation process, the Bundesarchiv now has 70 high-performance systems consisting of SSD storage media and transportable storage containers with which it can securely transfer digitized data to the archive in Koblenz in an extremely short time.

“With the Silent Bricks, we achieve transfer speeds of 500 to 700 MB per second for large files. Previously this was only 80 MB. This offers enormous advantages and streamlines the process considerably. We were able to take the technology into use and roll it out to our external service providers extremely easily without any modifications.”
Timo Dommermuth, IT specialist at the Bundesarchiv

Ten times faster storage speeds thanks to Silent Bricks:

How the German Bundesarchiv is digitizing the nation’s memory

Cinema newsreels from World War II, films such as “Rosen für den Staatsanwalt” (1959), documents from the end of the 14th century and, since 2021, the Stasi archives as well: the German Bundesarchiv stores the nation’s memory. More and more of the archives on paper or microfiche are being digitized. As of 2024, this should happen at a rate of some 80 million pages per year. FAST LTA’s Silent Bricks play a key role in this as fast and transportable storage media. As a result, the Bundesarchiv was able to increase storage speeds by a factor of ten over its previous solution. And thanks to this new storage system, digitized information is also available twice as fast. The Bundesarchiv and its external service providers store the digitized files on transportable and virtually indestructible storage media. These are transferred to Koblenz and stored there for eternity on magnetic tape. With each new request for archival material, the principle of digitization on demand is now applied. In other words, every paper document touched for this purpose is immediately digitized.

Challenges

1. Digitize up to 10 petabytes of archival data per year
2. Traditional hard drives represented a bottleneck
3. Secure transport of large amounts of data over long distances

Every era brings new insights. This makes it all the more important to re-evaluate documents from the past from the point of view of the present zeitgeist and state of knowledge. Archives such as those of the Bundesarchiv are indispensable for this purpose. “Every archive is a place that preserves individual and collective memory. In this age of information and digital transformation, archives safeguard popular sovereignty,” said Prof. Dr. Hollmann, president of the Bundesarchiv.

Mountains of data

The Bundesarchiv has the legal mandate to store and make usable German archival assets for the long term. Files, maps, images, posters, films and tape recordings are preserved in analog and digital form. Some 2,300 employees currently work at the Bundesarchiv. About a hundred of them are IT professionals. Digitizing the archives is a truly herculean task, as storage volumes grow by five to 10 petabytes every year. Within the archives, there are many analog sources. At the end of 2020, for example, there were 428 linear kilometers of written material and more than a million rolls of film alone. Since mid-2021, the Bundesarchiv also manages the Stasi archives of the “Staatssicherheitsdienst,” the internal secret service of the GDR, which were previously under the management of the Gauck-Behörde. This creates a further need for digitization.

In addition, all cinema films whose makers receive government subsidies from the German government are stored in the Bundesarchiv. However, the lion’s share of the archival material consists of files, such as policy documents of the German government and various government agencies. These documents trace back to the Holy Roman Empire of the German Nation. Finally, secret documents are also stored in the Bundesarchiv.

Bottlenecks within the digitization process

The first consultation regarding the project took place in mid-2018. “We were faced with the problem that we were receiving an enormous amount of data from external service providers. This trend was increasing noticeably. We received this data on USB drives or hard drives, and transferring it involved a huge overhead,” says Timo Dommermuth, head of IT at the Bundesarchiv. The data then had to be downloaded to a special storage environment for further processing.

“Copying the data from these data carriers, in most cases USB disks, proceeded at best at only 80 MB per second. This resulted in extremely long processing times, because we receive not just a few terabytes of data, but up to ten petabytes per year,” Dommermuth said. Moreover, it took up to two weeks to evaluate the data on such disks and fully incorporate it into the archive. In fact, an integrity check took place in the dedicated storage environment and the data were converted into TAR files. The resulting data sets were then stored on tape. The discussion was therefore centered on how this process could be dramatically streamlined, simplified and accelerated.

Elimination of copying process and noticeably faster data transfer

This was no easy task, as there seemed to be no practically feasible solution available on the market. However, the IT experts remembered a presentation on FAST LTA’s storage system they had attended some time before. It was about the classic concept of Silent Bricks as an energy-efficient cold storage archive solution. But in view of the enormous data volumes the Bundesarchiv faces, that would have been a utopia, according to Dommermuth. However, the experts got a bright idea: what if they simply used this technology to exchange data with external service providers?

“We remembered that the Silent Bricks could be inserted directly into the controllers and put into use. This allowed us to speed up the ingest and copy process considerably.”

In late 2018, they launched a proof of concept (POC) in collaboration with IT service provider Bechtle AG, a partner of FAST LTA. One initially started with Silent Bricks in two different sizes, an SSD variant and the corresponding controllers. “During the POC, it turned out that with the Silent Bricks we could achieve transfer speeds of 500 to 700 megabytes per second for files with a large size. This offers enormous advantages and streamlines the process considerably,” says Dommermuth. Processing times were thus more than halved. As a result, the digitized information is available faster in the archive.

The data is written by the service providers directly to the Silent Bricks. After transport to the Bundesarchiv, the data carriers are pushed into the controller and linked to the archive via the NFS protocol. The copying process is thus completely eliminated.

During the initial project phase, 20 Silent Bricks were purchased in combination with two controllers, each offering five slots for Silent Bricks. Because of the faster storage performance, the Bundesarchiv chose the SSD variant.

Easy integration of digitization partners

Already during the POC, the new storage technology was put to the test in cooperation with several service providers who were involved in the project alongside Bechtle. “We were able to start using the FAST LTA technology right off the shelf, so to speak, and without any modifications. Our digitization partners informed us that they could start using it immediately without any problems,” recalls the IT manager. The time between the initial concept and the final implementation was only six months. “That is exceptionally fast for an IT project that involves technology never used before,” Dommermuth says. The Bundesarchiv team is responsible for about a third of all digitization. The remaining part falls to external service providers. At the Berlin-Lichterfelde location, the team stores new data on Silent Bricks that are transferred to Koblenz every two weeks.

Highly secure shipping containers are used for this purpose. “We bet on the right horse,” says Timo Dommermuth. “There is flawless contact with Bechtle and the support from FAST LTA is also excellent. Both partners are extremely flexible. There have been no problems so far.”

Implementation

1. A successful proof of concept that involved external service providers

2. A particularly fast and trouble-free start to the project with 20 Silent Bricks

3. A continuously growing storage system

Safety as a driving force

Another important reason why the State Archives’ choice of Silent Bricks fell on Silent Bricks was the robustness of the system, which is almost completely fail-safe. After all, there are regular transports between the external service providers and the Bundesarchiv, and not always gently. Within each storage medium, erasure coding is used to prevent data loss due to failures. “Thanks to this integrated security concept, the risk of data loss is virtually zero, whereas in the past we were forced to re-digitize documents when failures occurred on USB drives,” says Timo Dommermuth.

Digitizing archives is not always easy. Especially when nitrate films are involved. World War II cinema newsreels alone take up 20 terabytes in digitized form. Moreover, it takes an unadulterated fingerspitzengefühl to ensure that the films are not damaged or put people in danger. The coating of nitrocellulose, also known as shooting cotton, falls under the explosives law. For this reason, the nitrate films must be stored at a certain temperature so that they cannot spontaneously ignite. Even the storage of the digitized nitrate films on Silent Bricks is now done in secure rooms.

A growing storage system

The modular storage system, which can be expanded flexibly based on the building block principle, has grown more and more over the years. The Bundesarchiv is currently in a new procurement phase. In the near future, another six G5000 controllers and around 30 Silent Bricks, each with a gross SSD storage capacity of 24 TB, are to be purchased. Things are clearly moving in the direction of on-demand digitization within the State Archives: as soon as documents are requested as part of a new information request, they are digitized. This is all the more reason to keep expanding the Silent Brick infrastructure. The experts at the Bundesarchiv expect to be able to digitize some 60 to 80 million pages per year starting in 2024. To date, 75 million pages are available in digital form. This is extremely high even by international standards.

The reasons why people call on the archive vary widely. They often involve scientists or doctoral students who need documents for their research. But there are also repeated requests for information to clarify people’s fates. The generation of grandchildren increasingly wants to come to terms with family history during the time of National Socialism. But there are also less common requests. For example, game producers need blueprints of World War II battleships for their game designs. Nearly all of these users now receive their information in digital form rather than on paper.

Results

1. Transfer speed was increased to 700 MB per second

2. Turnaround time for recording digitized files is more than halved

3. There is no longer any data loss due to problems in transit

Partner

• Bechtle IT-Systemhaus Bonn/Köln

About the Bundesarchiv

The Bundesarchiv is a German government institution with the legal mandate to store and make usable state archives for the long term. It stores files, maps, posters, images, films and tape recordings in analog and digital form. About 2,300 employees currently work at the Bundesarchiv. About a hundred of these are IT professionals. The archive was founded in 1952 in Koblenz. Before that, the state archives on the Brauhausberg in Potsdam served as the central archive.

After World War II, the Allies transferred confiscated documents to the Bundesarchiv. Since 1955, the State Archives has also been responsible for the long-term storage of the military history of the Federal Republic of Germany and its predecessors. In June 2021, responsibility for managing the documents of the Ministry for State Security (MfS) of the former GDR additionally fell to the Bundesarchiv.