2.1.1 -> 2.1.2

- AlignDocumentIterator (syntax: ^) makes it possible to align
  document/interval iterators from different indices. Using this
  feature MG4J can easily support queries based on semantic tagging.

- Fixed another bug in Snowball stemmers: calling processTerm() with
  a null argument would have caused an exception.

- Now Scan and IndexBuilder accept parseable objects as sequences.
  The same happens for the WORDREADER property of some factories,
  making it possible to create a moderately command-line-configurable
  FastBufferedReader as WordReader.

- UNICODE_INPUT is now set in SimpleParser.jj, making it possible to
  write wild Unicode queries again.

- QueryServlet now forces UTF-8 for output.

- We now distribute the javacc-generated files for easier installation.

- More liberal Velocity template-resolution setup, now documented in
  the HttpQueryServer Javadoc.

2.1 -> 2.1.1

- Major fix: the Snowball stemmers would generated empty strings, and
  Combine would choke (generating empty indices) on empty strings.

- Removed obsolete PorterStemmerTermProcessor.

2.0.1 -> 2.1

- WARNING: Most utility classes have been moved to dsiutils. Old versions
  are still here and deprecated, but you'll have some problems when importing
  this version. Always check which version you're using!

- WARNING: TermMap has been replaced by StringMap (in dsiutils). PrefixMap
  exists, but the dsiutils signature is completely different from the old one.

- Lots of stemmers coming from Snowball. We actually made some improvements
  to the Java Snowball compiler to get this working at a reasonable speed.

- New (somewhat experimental) feature: you can get the terms that caused
  an interval to be emitted.

- Sequential scan was not working for high-performance indices if positions
  were not read. The problem was evident when combining high-performance indices
  specifying -cPOSITIONS:NONE.

- Fixed a couple of NullPointerException in index construction (thanks to
  Marko Srdanovic for reporting these bugs).

- Fixed missing call to super.close() in AbstractIndexClusterIndexReader that
  was causing spurious warnings.

- Now Query has multiplex on by default.

- Fixed bug in MutableString.subSequence() (thanks to Espen Amble Kolstad
  for reporting this bug). MutableString is now in dsiutils.

- New allLines() method in LineIterator. LineIterator is now in dsiutils.

2.0 -> 2.0.1

- Can you believe that? Fast.leastSignificantBit() under very peculiar
  circumstance was returning random data, but apparently this was causing
  no warm. I don't wanna know.

- Better memory handling: buffer reallocation logic in index construction
  could cause out-of-memory errors. Now we retry a small reallocation after
  dumping the content in a temporary file, and record the event so the
  Scan process can dump the current batch.

- Fixed old minor bug in Combine: term files and global-counts files were
  not closed, leading to bizarre and spurious too-many-open-files errors.

- Fixed derelativisation when using FileSystemItem.

1.1.3 -> 2.0

- METAWARNING: This release has so many changes and so many new features
  that we strongly suggest to read carefully all information below
  and the manual.

- WARNING: there are performance improvements due to fixed-point
  computation of Golomb moduli (yes, it *really* slows down things), but
  unfortunately all indices have to be rebuilt.

- WARNING: virtual fields have changed in a completely incompatible
  way, and the same happened to AnchorExtractor. This was necessary
  to get finally rid of problems with System.identityHashCode()
  (see below).

- WARNING: BitStreamIndexIterator will now throw an UnsupportedOperationException
  when positions or intervals are retrieved on an index without positions.
  Previously, getting positions would have produced the same effect, but
  getting intervals would have returned TRUE. This was causing a very confusing
  behaviour with ordered AND, consecutivity, etc., as they were returning
  false positives.

- WARNING: a great deal of work has gone into making all relevant iterators
  fully lazy. Please use DocumentIterator.nextDocument() and
  IntervalIterator.nextInterval(), after reading the related Javadoc
  documentation. The change has produced significant performance
  improvements.

- WARNING: IOExceptions are now rethrown by most index-access methods.
  Previously, they would have been catched and wrapped into
  RuntimeException, but this behaviour was slightly slowing down methods
  called very often like nextDocument().

- WARNING: The old sequential reading methods (e.g., readDocumentPointer())
  are no longer available (I guess nobody was using them anyway). They are
  replaced by an IndexReader.nextIterator() method that returns an index
  iterator on the term after the current one, until exhaustion.

- WARNING: Quanta are now restricted to powers of two.

- Completely new kind of index (high-performance). It uses the Lucene idea
  of keeping positions in a separate file, and enriches it with MG4J skip
  structures. It is now the default index type.

- Completely rewritten index reading. Now a ruby script generate different
  readers for different combination of flags, increasing significantly
  performance due to the reduced logic overhead. A generic class is always
  available, but for production sites wired index readers are the right
  choice. The wired, faster class is fetched automagically by reflection
  if available.

- Completely new, memory-adaptive index construction strategy. Just specify
  a number of *documents* per batch and let MG4J do the rest. Please read the
  Scan class documentation.

- New payload-based indices. Now it is possible to index dates, integers,
  or any other payload. By default we supply range queries.

- Significant improvements in performance. System.identityHashCode() turned
  out to be *deadly* slow, so we dropped reference-based open hash map and
  started using brute-force array maps (you need fastutil >= 5.0.7)
  whenever we have to have to manipulate very small sets. The gains are
  suprising, in particular for queries containing frequent terms.

- Even more improvement due to parallel reimplementation of all operators for
  the special case in which all document iterators are index iterators. In
  this case all intervals have length 1 and can be retrieved eagerly. In some
  cases performance is almost doubled.

- New low-level coded-integer skipping methods have further increased performance
  in certain situations (e.g., phrasal queries containing stopwords).

- Now we use precomputed bit codes for 65536 words, uniformly. This
  requires 4MiB of memory just for precomputed words, but it almost doubles
  decoding speed (as the logic is much, much simpler).

- New bulk reading methods for integers in gamma, shifted gamma and delta
  coding. They make readDocumentPositions() several times faster as most
  decodings do not require a method call.

- Many fixes to the code involving generics.

- Fixed stupid bugs in PartitionLexically.

- Moved sizes into Index (brom BitStreamIndex) and added new SIZES property
  that makes it possible to specify a global sizes file. This way, it is
  possible to use BM25 on clusters.

- Major fixes to documentally clustered document iterators.

- Fixed subtle semantic issue in LowPassDocumentIterator: TRUE iterators
  now make the iterator valid.

- Fixed subtle semantic issue in subclasses of AbstractOrderedIntervalIterator:
  how TRUE subiterators are considered as always matching (so the actual
  interval matching is performed just on non-TRUE iterators).

- Fixed bug in ScoreDocumentBoundedSizeQueue that was causing enqueuing
  of documents with score equal to the minimum.

- Improved implementation of MinimalPerfectHash. By fixing deterministically
  the perfect hash functions we reduce to virtually zero the trials during
  the construction (thanks to djam8193ah@hotmail.com for suggesting the idea).

- Fixed old copy-and-paste bug in non-scored requests to QueryEngine: offset
  was not used at all (but I guess nobody was using that method anyway).

- Completely new support for query expansion. A MultiTermIndexIterator
  behaves in all respects like an IndexIterator, but it's actually built
  by merging the index iterators of several terms. The "frequency" is
  settable so to solve term-dependency problems in IDF-based ranking schemes.
  For debugging purposes, + can be used (instead of |) to cause the
  constructon of a MultiTermIndexIterator.

- Brouwerian difference is now supported. It kills all intervals of the minuend
  that appear in the subtrahend. It can be used for searching for terms forcing
  however the context in which they are found *not* to contain some
  terms, or more generally a query. It can also be used to modify index granularity
  by subtracting 2-element intervals that cross section boundaries.

- ConsecutiveDocumentIterator now support gaps that can be used to match arbitrary
  words. This is particularly useful to perform phrasal queries in indices where
  some terms have not being indexed. Gaps are specifiable using $ instead
  of a term in the built-in parser.

- New methods to access the front of a subclass of AbstractUnionDocumentIterator,
  that is, the indices of the component iterators positioned on the current document.
  They are used by all union-based iterators, providing a significant performance
  improvements on large unions.

- New metadata-only mode for Combine and related subclasses. Mainly useful for getting
  the global sizes, terms, etc. of a cluster.

- The array-writing methods of OutputBitStream now take a long for the
  bit length/offset, and correspondingly return a long. The old methods are
  still present, but they are deprecated (just to avoid proliferation).

- Deprecated all minimal perfect hashing constructors using the platform default
  encoding. They are just an endless cause of problems. There are now constructors
  with just a filename and an encoding (which can be null to mean the platform
  encoding, but you have to explicitly ask for it).

- Now all TermMap implementations have a constructor accepting an Iterable<CharSequence>.

- New constructors and main method options for minimal perfect hash tables, prefix
  dictionaries and front-coded lists that support reading gzip'd files.

- Query provides a clearer selection between *no interval selection* and
  *no intervals*.

- Fixed bug in ImmutableBinaryTrie: prefixes of the first binary string
  would have generated an empty approximated interval (instead of [0]).

- Fixed bug in writeShiftedGamma()/readShiftedGamma(), and modified test.bsh
  so that it detects the bug.

- The SPIRE 2006 algorithms are by now obsolete--we have new, provably optimally
  lazy algorithms. The code reflects this.

- Lots, lots, lots of unit tests.

1.1.2 -> 1.1.3

- New score(digits) method for ResultItem for easier display.

- Now JdbcDocumentCollection works with factories featuring more than one field.

- Reintroduced the JavaBeans Activation Framework in dependencies.

- Fixed lack of calls to close() in some document factory, generating
  spurious warnings.

- Fixed static fields in QueryServlet.

1.1.1 -> 1.1.2

- Fixed default values of K_1 and B in BM25 scorer following
  Büttcher & Clarke's paper.

- Fixed interval methods for nonsense calls on the empty interval.

- Dumped jline--we now suggest using rlwrap.

- More sensible hash for intervals. As a consequence, the serialUID
  had to be bumped.

- Fixed serious bug in OrderedAndDocumentIterator that was dropping
  several correct intervals (thanks to Fabien Campagne for finding
  this bug).

- Fixed very old bug in InputBitStream.read(byte[], int)--reads of
  full length would have caused an ArrayIndexOutOfBoundsException
  (thanks to Kevin Dorff for finding this bug).

- OrDocumentIterator was using an indirect queue instead of a 
  semi-indirect queue, maybe for historical reasons.

- Complete rewrite of interval operators due to new algorithms, to
  be included in the revised SPIRE 2006 paper. On TREC data this led
  to an average 3% increase in speed. Now the algorithms used by MG4J
  are provably optimally lazy.

- The BulletParser now accepts element-type names with dashes, etc., and
  moreover parses correctly explicit CDATA sections (thanks to Kevin Dorff
  for finding these bugs).

- Support for unsigning signed minimal perfect hash maps.

- New Shift-Add-Xor-based signed minimal perfect hashes (even with long
  signatures). Moreover, now all signed hashes have a main() method
  generating by default instances of that hash.

- Massive speed improvements in OutputBitStream: finally we write
  precomputed words for small integers, analogously to what happens
  in InputBitStream.

1.1 -> 1.1.1

- Better loading of InputBitStream data, working also with multiple class
  loaders, and serialisability of SelectedInterval (fixed by the Twease
  people).

- AbstractAggregator was not setting up the equalisation factors when
  equalisation was not required, resulting in divisions by 0.

- CountScorer is now a DelegatingScorer (as it should have always been).

- The empty-constructor interval selector wasn't really letting out *all*
  interval--overlapping intervals would have been discarded.

- Fixed a *very old* bug in the computation of minimal-interval semantics.
  Now the code is fully aligned with our SPIRE 2006 paper.

1.0.2 -> 1.1

- IMPORTANT: IndexWriter.close() no longer save automagicall
  properties--you have to fetch them with IndexWriter.properties().

- Java 5 only.

- Probably the largest rewrite and extension in the history of MG4J. Too
  many changes, fixes and optimisations to be described here. Almost nothing
  is backward-compatible.

- We are starting to distribute unit tests with each release. We have
  actually many more tests, but they are not cast inside JUnit and rather
  undocumented. You are welcome to donate unit tests.

1.0.1 -> 1.0.2

- Fixed bug in InputStreamDocumentCollection: the document index (and thus 
  the title) was never incremented.
- New parsing factory for the BulletParser: you decide how to parse your names (an
  idea by Fabien Campagne).
- Now we use 1.26n integers to minimally hash n words. 1.25n is in fact the
  threshold--you need something larger than that. The change should be fully
  backward-compatible.
- Now FileLinesCollection returns a Closeable FileLinesIterator.
- BloomFilter does not implement any longer the nonsensical size() method. add()
  is more efficient and does not return a value.

1.0.0 -> 1.0.1

- Fixed bug in Paste if the size of size lists differ (now we extend to zero).
- The "field" property was not propogated by Combine.
- A missing throws clause in AbstractDocumentCollection's implemention of iterator() 
  was making it impossible to throw exceptions in implementing subclasses.
- New, efficient single-query iterator for JdbcDocumentCollection.
- NULLs do not generate null pointer exceptions any longer in JDBC document collections.
  They're converted to empty input streams.

0.9.2 -> 1.0.0

- Too much to be written.

0.9.1 -> 0.9.2

- IMPORTANT: To avoid clashes with List, the get() method of TermMap
  has been changed to getTerm(). We're sorry for this inconvenience.

- Now we support prefixes by means of a PrefixMap. There are easy
  (ternary search trees) and very sophisticated (semi-external tries)
  implementation. If you have a PrefixMap you can search for things
  like "foo*" (meaning "starts with foo"), provided that the terms
  starting with "foo" do not exceed a constant defined in QueryParser.

- Interval has new methods that compare to points.

- Fixed stupid bug in ClarkCormack scorer: we were comparing the document
  indices, not the scores. Ouch.

- Fixed ScoredDocumentBoundedSizeQueue: now stability is forced by making
  the order an actual order (not a preorder) so it is possible to get the
  k-th to (k+j)-th ranked documents in a consistent way. The new version
  is, unfortunately, completely incompatible with the old one.

- New CachingDocumentIterator: it decorates a DocumentIterator so that
  you can get several times its interval iterators.

- FastBufferedOutputStream was NOT flushing. The flush() method was inherited,
  but of course that didn't work. FastBuffered{In,Out}putStream are
  now deprecated as they have been moved to fastutil.

- OrDocumentIterator would have caused IllegalStateException in some circumstances
  (the array of underlying iterators was assumed to have null'd position for
  empty component iterators, but this wasn't happening).

- InputBitStream is a boolean iterator, and OutputBitStream has a method
  accepting a boolean iterator. This opens a new world of possibilities 8^).

- New replace() and delete() methods in MutableString for handling more
  easily deletions or substitution of a class of characters.

- readLine() no long empties its argument on end of file.

0.9.0 -> 0.9.1

- A couple of missing methods in it.unimi.dsi.mg4j.util.Fast were necessary for
  WebGraph.

0.8.2 -> 0.9.0

- IMPORTANT: Int2IntArrayMap and Int2LongArrayMap no longer exist: offsets and
  sizes now are type-specific array lists (and they can be easily generated using
  fastutil wrappers).

- Fixed stupid, stupid bug in state handling in IndexReader. Sequential
  reads of an entire index would have thrown an IllegalStateException.

- Changed a few array static methods to faster fastutil counterparts.

- Fixed small glitch in lastIndexOf() semantics--searches from negative
  offset of the empty string would have returned 0 instead of -1.

- Bunch of new methods in MutableString ((last)indexOfAnyBut, (co)span).

- Golomb read/write methods now support modulus 0, 0 being the only valid
  argument (and the result returned upon reads).

- Minimal perfect hashes support lists with less then 16 elements, by
  storing them transparently in a vector.

- Signed hashes have an incompatible format (sorry).

- Minimal perfect hashes support optimal weight length computation for
  sorted term collections.

- New left/right trim methods. Moreover, trim methods preverve looseness
  and compactness.

- Literally zillions of new features, everywhere.

- Experimental support for multi-index minimal-interval semantics and
  skipping towers.

0.8.1 -> 0.8.2

- New methods for starting and stopping a progress meter with messages.

- New FastMultiByteArrayInputStream class that can hold 256 PiB (256 PiB = 2^28
  GiB) and expose them as a repositionable stream.
 
0.8 -> 0.8.1

- Modified imports and class name for compliance with fastutil 3.0.

- Relicensed under the LGPL.

0.7.1 -> 0.8

- New NullInputStream to support new InputBitStream direct array wrapping.

- position() in InputBitStream will always work if the new position is
  within the current buffer.

- Removed unused buffer in InputBitStream, and made unget buffer allocation
on-demand.

- Eliminated finalizers from streams.

- New debugging class.

- Fixed bug in position().

- Now the ProgressMeter gives items/s at the first printout.

- New methods for variable-length nibble coding.

- New methods for zeta coding (a new code!).

- Fixed erroneous serialisation of CRC32SignedMinimalPerfectHash.

- Completely renewed hashing scheme for minimal perfect hashing: supports
  the empty string and it is faster to compute. Moreover, MinimalPerfectHash
  now has an offline builder that never loads the words actually in RAM,
  thus allowing to hash very large sets (albeit slowly), and checks 
  a suitable system property to provide optional verbose logging. The
  serialisation, unfortunately, is incompatible.


0.7 -> 0.7.1

- Removed experimental classes.

- Fixed two bad bugs introduced during the in 0.7 during optimisation.

0.6 -> 0.7

- IMPORTANT: MG4J now uses the new fastutil package name (i.e., no
  more fastutil). If you use parts of MG4J that require fastutil, you
  should upgrade.

- New replace() methods for MutableString that entirely replace
  the string content. New copy() method to obtain easily a compact
  copy of a mutable string.

- New RepositionableStream interface to mark streams that can
  be repositioned by bit streams.

- New FastByteArrayInputStream to read memory blocks as bit
  streams.

- New unsynchronised FastBufferedReader.

- ProgressMeter count value has now setters and getters.

- Programmable meter quantum for FirstPass.

- Some optimisations.

0.5 -> 0.6

- IMPORTANT: streams and self-delimiting string formats are not
  binary compatible with previous versions. Please read the docs.

- Fixed bug with serialisation of empty strings and set serialVersionUID.

- Too many addition to be described in a file, but in short: optimised
  indexOf() family of methods, flexible index construction, QuickSearch
  fast searches.

0.4 -> 0.5

- MutableString has now a more coherent policy for compactness and looseness.
  They are preserved by all operations.

- StringBuffer-specific methods have been killed to reduce code duplication.
  You need to recompile so that java uses the alternative CharSequence-specific
  methods.

- Several new methods such as startsWith, endsWith etc.

0.3 -> 0.4

IMPORTANT: the hash computation functions in 
MinimalPerfectHash have been changed. Please regenerate your maps.

MinimalPerfectHash has been reimplemented to use CharSequence, so
it is more general. Moreover, we have a new SignedMinimalPerfectHash
that can be used to avoid false positives.

- New replace methods in mutable strings.

- Now we try to return a reference to this in all mutable string methods.

- Various fixes to documentation.

0.2 -> 0.3
- Introduced new class MutableString.

0.1 -> 0.2
- By mistake writeLongDelta() was really called writeDelta().
