Compressing Files

 

Why Compressing Files is a Good Idea

While optimizing our deployment process we came across some applications which had an absurdly large deployment package: a 750 MB zip file is a lot of data to be pushing around, particularly if you’re running this several times a day.

Because of how this particular deployment process works, there are a lot of duplicated files contained in the archive. One might expect that this will be automatically handled by the archiving tool, however, this is not the case for the most common .zip files. There are two common formats that are able to handle this, .7z and .rar. Both of the formats support what is known as a “solid archive” where the files are stored contiguously. In the 7zip UI this is called “Solid block size” in WinRAR this appears as “Create solid archive”.

 

What Do We Find?

Using the solid archive option on directories, changing no other options, our deployment size goes down from 750MB to 150MB. However, this is still not ideal because our build process creates .war files, which are already application bundles. Default compression options aren’t able to handle putting multiple large .war files inside a .rar file: This means we still end up with a 750MB archive! At this point, we come across another setting available in both the .rar and .7z format: dictionary size. Cranking this all the way up to 1GB takes the result back down to 150MB.

Both 7zip and WinRAR provide the option of creating self-extracting executables (SFX archives). This means that even though we are using less common tools, we are not introducing any additional dependencies during deployment.

The full command we run when creating the archive looks like this:

winrar.exe a out-path-here my/folder/path/* -r -ep1 -ma5 -s -sfx -md1g -y -t -zsfxcomment.txt -ibck

The parameters are as follows:

 a      = create archive
-r      = recurse path for files
-ep1    = don't add container path in archive
-ma5    = use winrar v5 format
-s      = solid archive
-sfx    = self-extracting .exe
-md1g   = use a 1GB dictionary
-y      = answer yes to all prompts
-t      = test files after archiving
-z      = add a comment from a file, which provides ext

with the content of the sfxcomment.txt file:

;The comment below contains SFX script commands
Silent=1
Overwrite=1

At Greenfinch Technology we are always actively working to keep up with modern technology. In our opinion. to stand still is to fall behind. You can be sure we have no intention of letting that happen to ourselves or to our clients.

If you would like a modern, forward-thinking IT company to help with your custom enterprise software solutions, web or mobile app development, a BPM such as Siskin or any Cloud technologies please feel free to contact us at 01 818 2949 or email us using our contact form. We will help guide you safely through the process of advancing your company beyond your competitors.

Greenfinch Technology

Greenfinch Technology is a software development company based in Dublin, Ireland. Our focus is on using cutting-edge technologies to build world-class mobile and web applications for the BES (Business Enterprise Sector).