Open-source DBMS for data historization and impact of flash memories
Archiving industrial data is a complex issue: a large volume of data has to be stored for several decades while meeting performance requirements for real-time insertions, along with retrieval and analysis queries. For these applications, niche products have specialized in this market segment: data historians. Data historians are proprietary solutions, with license fees of tens of thousands of dollars, and whose internal mechanisms are not documented. Therefore, we first emphasized data historian specificities, with regards to functionalities as much as performance. However, archiving industrial data can occur in very different contexts. IGCBoxes for example are industrial mini PCs using MySQL for short-term data archiving in hydroelectric power stations at EDF. These equipments expose distinctive features, mainly on their storage system based exclusively on flash memory, for its reliability in an industrial environment and its low energy consumption. For historical reasons, DBMS include many hard disk drive-oriented optimizations, and the lack of adjustment for flash memories can significantly decrease performance. This type of memory thus had notable consequences on insert performance, with a substantial drop compared with hard disk drives. We therefore designed Chronos, a DBMS for historization data management on flash memories. For that purpose, we especially identified an efficient “quasi-sequential” write pattern on flash memories, along with buffer and index management techniques optimized for historization typical workloads. Experimental results demonstrate improved performance for insertions over different solutions, by a factor of 20 to 54. Chronos is therefore competitive when insertions make up an extensive part of the workload. For instance, Chronos stands out with the typical workload of IGCBoxes, with global performance improved by a factor of 4 to 18 compared with other solutions.