I am currently working on the compression, as it is not complete yet. The 100 MB is indeed excessive but the window is dynamic and can differ from file to file since it is written to a `*.hblock` file which is stored next to the object in the object database https://github.com/Snowtrack/SnowFS/blob/03e5f839326e666c891...
Let me explain where the 100 MB window comes from as its not only related to the upcoming compression implementation. Some graphic applications touch the timestamps of their files for no reason, making it harder to detect if a file changed. But some file formats always change their 'header' or 'footer'. Means, comparing the hash of the first or last 100 MB of a file that is 8GB in size gives a great performance boost to detect if a file got modified.
Let me explain where the 100 MB window comes from as its not only related to the upcoming compression implementation. Some graphic applications touch the timestamps of their files for no reason, making it harder to detect if a file changed. But some file formats always change their 'header' or 'footer'. Means, comparing the hash of the first or last 100 MB of a file that is 8GB in size gives a great performance boost to detect if a file got modified.