Check out the FiniteStateEntropy/test/probaGenerator.c code.
It builds a table of character sequences based on a probability. If you feed it a p=0.5 you get a sequence that halves each time:
0000000011111223 (shortened example, the real total length is 4096 so you get 2048x0, 1024x1 ...)
This is then used as an alphabet probability to build random sequences in the test file.
The algorithm scores better than huff for higher P, meaning that the alphabet shortens with a few dominant ones. This means that (statistically) you get a lot of words in your dictionary that resemble each other. And that's where the fractional probability plays its role.
To try to answer your question I'd guess that this technique shows its muscle for data sets in which the dictionary is large and consists of similar sequences. Example: a csv file with a lot of numbers, gis or kml datasets, Gerber files, ...
It builds a table of character sequences based on a probability. If you feed it a p=0.5 you get a sequence that halves each time:
0000000011111223 (shortened example, the real total length is 4096 so you get 2048x0, 1024x1 ...)
This is then used as an alphabet probability to build random sequences in the test file.
The algorithm scores better than huff for higher P, meaning that the alphabet shortens with a few dominant ones. This means that (statistically) you get a lot of words in your dictionary that resemble each other. And that's where the fractional probability plays its role.
To try to answer your question I'd guess that this technique shows its muscle for data sets in which the dictionary is large and consists of similar sequences. Example: a csv file with a lot of numbers, gis or kml datasets, Gerber files, ...