# Release notes for PyTables Pro 2.0 series

Author: | Francesc Alted i Abad
Contact: | faltet@pytables.com
Author: | Ivan Vilata i Balaguer
Contact: | ivan@selidor.net

## Changes from 2.0.3 to 2.0.4

- Selections in tables works now in threaded environments. The problem was in the Numexpr package – the solution has been reported to the upstream authors too. Fixes #164.
- PyTables had problems importing native HDF5 files with gaps in nested compound types. This has been solved. Fixes #173.
- In order to prevent a bug existing in HDF5 1.6 series, the `EArray.truncate()` method refused to accept a 0 as parameter (i.e. truncate an existing EArray to have zero rows did not work). As this has been fixed in the recent HDF5 1.8 series, this limitation has been removed (but only if the user has one of these installed). Fixes #171.
- Small fixes for allowing the test suite to pass when using the new NumPy 1.1. However, it remains a small issue with the way the new NumPy represents complex numbers. I’m not fixing that in the PyTables suite, as there are chances that this can be fixed in NumPy itself (see ticket #841).

## Changes from 2.0.2.1 to 2.0.3

- Replaced the algorithm for computing chunksizes by another that is more general and useful for a larger range of expected dataset sizes. The outcome of the new calculation is the same than before for dataset sizes <= 100 GB. For datasets between 100 GB <= size < 10 TB, larger values are returned. For sizes >= 10 TB a maximum value of 1 MB is always returned.
- Added support for the latest 1.8.0 version of the HDF5 library. Fixes ticket #127.
- PyTables compiles now against latest versions of Pyrex (0.9.6.4). For the first time, the extensions do compile without warnings! Fixes #159.
- Numexpr module has been put in sync with the version in SciPy sandbox.
- Added a couple of warnings in User’s Guide so as to tell the user that it is not safe to use methods that can change the number of rows of a table in the middle of a row iterator. Fixes #153.
- Fixed a problem when updating multidimensional cells using the Row.update() method in the middle of table iterators . Fixes #149.
- Fixed a problem when using 64-bit indexes in 32-bit platforms. Solves ticket #148.
- Table.indexFilters is working now as documented. However, as per ticket #155, its use is now deprecated (will be removed in 2.1). Fixes #155.

## Changes from 2.0.2 to 2.0.2.1

- Optimization added for avoid to unnecessarily update index columns that have not been modified in table update operations. Fixes #139.

## Changes from 2.0.1 to 2.0.2

- Fixed a critical bug that returned wrong results when doing repetitive queries affecting the last row part of indices. Fixes #60 of the private Trac of Carabos.

- Added `__enter__()` and `__exit__()` methods to `File`; fixes #113. With this, and if using Python 2.5 you can do things like:

  > with tables.openFile(“test.h5”) as h5file:

  >   ...

- Carefully preserve type when converting NumPy scalar to numarray; fixes #125.

- Fixed a nasty bug that appeared when moving or renaming groups due to a bad interaction between `Group._g_updateChildrenLocation()` and the LRU cache. Solves #126.

- Return 0 when no rows are given to `Table.modifyRows()`; fixes #128.

- Added an informative message when the `nctoh5` utility is run without the NetCDF interface of ScientificPython bening installed.

- Now, a default representation of closed nodes is provided; fixes #129.

## Changes from 2.0 to 2.0.1

- The `coords` argument of `Table.readCoords()` was not checked for contiguousness, raising fatal errors when it was discontiguous. This has been fixed.

- There is an inconsistency in the way used to specify the atom shape in `Atom` constructors. When the shape is specified as `shape=()` it means a scalar atom and when it is specified as `shape=N` it means an atom with `shape=(N,)`. But when the shape is specified as `shape=1` (i.e. in the default case) then a scalar atom is obtained instead of an atom with `shape=(1,)`. This is inconsistent and not the behavior that NumPy exhibits.

  Changing this will require a migration path which includes deprecating the old behaviour if we want to make the change happen before a new major version. The proposed path is:

  > 1. In PyTables 2.0.1, we are changing the default value of the `shape` argument to `()`, and issue a `DeprecationWarning` when someone uses `shape=1` stating that, for the time being, it is equivalent to `()`, but in near future versions it will become equivalent to `(1,)`, and recommending the user to pass `shape=()` if a scalar is desired.
  > 2. In PyTables 2.1, we will remove the previous warning and take `shape=N` to mean `shape=(N,)` for any value of N.

  See ticket #96 for more info.

- The info about the `chunkshape` attribute of a leaf is now printed in the `__repr__()` of chunked leaves (all except `Array`).

- After some scrupulous benchmarking job, the size of the I/O buffer for `Table` objects has been reduced to the minimum that allows maximum performance. This represents more than 10x of reduction in size for that buffer, which will benefit those programs dealing with many tables simultaneously (#109).

- In the `ptrepack` utility, when `--complevel` and `--shuffle` were specified at the same time, the ‘shuffle’ filter was always set to ‘off’. This has been fixed (#104).

- An ugly bug related with the integrated Numexpr not being aware of all the variations of data arrangements in recarray objects has been fixed (#103). We should stress that the bug only affected the Numexpr version integrated in PyTables, and not the original one.

- When passing a record array to a table at creation time, its real length is now used instead of the default value for `expectedrows`. This allows for better performance (#97).

- Added some workarounds so that NumPy scalars can be successfully converted to numarray objects. Fixes #98.

- PyTables is now able to access table rows beyond 2**31 in 32-bit Python. The problem was a limitation of `xrange` and we have replaced it by a new `lrange` class written in Pyrex. Moreover, `lrange` has been made publicly accessible as a safe 64-bit replacement for `xrange` for 32-bit platforms users. Fixes #99.

- If a group and a table are created in a function, and the table is accessed through the group, the table can be flushed now. Fixes #94.

- It is now possible to directly assign a field in a nested record of a table using the natural naming notation (#93).

## Changes from 2.0rc2 to 2.0

- Added support for recognizing native HDF5 files with datasets compressed with szip compressor.
- Fixed a problem when asking for the string representation (str()) of closed files. Fixes ticket #79.
- Do not take LZO as available when its initialisation fails.
- Fixed a glitch in ptrepack utility. When the user wants a copy of a group, and a group is to be created in destination, the attributes of the original group are copied. If it is not to be created, the attributes will not be copied. I think this should be what the user would expect most of the times.
- Fixed the check for creating intermediate groups in ptrepack utility. Solves ticket #83.
- Before, when reading a dataset with an unknown CLASS id, a warning was issued and the dataset mapped to `UnImplemented`. This closed the door to have the opportunity to try to recognize the dataset and map it to a supported CLASS. Now, when a CLASS attribute is not recognized, an attempt to recognize its associated dataset is made. If it is recognized, the matching class is associated with the dataset. If it is not recognized, then a warning is issued and the dataset becomes mapped to `UnImplemented`.
- Always pass verbose and heavy values in the common test module to test(). Fixes ticket #85.
- Now, the `verbose` and `--heavy` flag passed to test_all.py are honored.
- All the DLL’s of dependencies are included now in Windows binaries. This should allow for better portability of the binaries.
- Fixed the description of Node._v_objectID that was misleading.

## Changes from 2.0rc1 to 2.0rc2

- The “Optimization tips” chapter of the User’s Guide has been completely updated to adapt to PyTables 2.0 series. In particular, new benchmarks on the much improved indexed queries have been included; you will see that PyTables indexing is competitive (and sometimes much faster) than that of traditional relational databases. With this, the manual should be fairly finished for 2.0 final release.
- Large refactoring done on the `Row` class. The most important change is that `Table.row` is now a single object. This allows to reuse the same `Row` instance even after `Table.flush()` calls, which can be convenient in many situations.
- I/O buffers unified in the `Row` class. That allows for bigger savings in memory space whenever the `Row` extension is used.
- Improved speed (up to a 70%) with unaligned column operations (a quite common scenario when dealing with `Table` objects) through the integrated Numexpr. In-kernel searches take advantage of this optimization.
- Added `VLUnicodeAtom` for storing variable-length Unicode strings in `VLArray` objects regardless of encoding. Closes ticket #51.
- Added support for `time` datatypes to be portable between big-endian and low-endian architectures. This feature is not currently supported natively by the HDF5 library, so the support for such conversion has been added in PyTables itself. Fixes #72.
- Added slice arguments to `Table.readWhere()` and `Table.getWhereList()`. Although API changes are frozen, this may still be seen as an inconsistency with other query methods. The patch is backwards-compatible anyway.
- Added missing overwrite argument to `File.renameNode()` and `Node._f_rename()`. Fixes ticket #66.
- Calling `tables.test()` no longer exits the interpreter session. Fixes ticket #67.
- Fix comparing strings where one is a prefix of the other in integrated Numexpr. Fixes ticket #76.
- Added a check for avoiding an ugly HDF5 message when copying a file over itself (for both `copyFile()` and `File.copyFile()`). Fixes ticket #73.
- Corrected the appendix E, were it was said that PyTables doesn’t support compounds of compounds (it does since version 1.2!).

## Changes from 2.0b2 to 2.0rc1

- The `lastrow` argument of `Table.flushRowsToIndex()` is no longer public. It was not documented, anyway. Fixes ticket #43.
- Added a `memlevel` argument to `Cols.createIndex()` which allows the user to control the amount of memory required for creating an index.
- Added `blocksizes` and `opts` arguments to `Cols.createIndex()`, which allow the user to control the sizes of index datasets, and to specify different optimization levels for each index dataset, respectively. These are very low-level options meant only for experienced users. Normal users should stick to the higher-level `memlevel` and `optlevel`.
- Query tests have been tuned to exhaustively check the new parametrization of indexes.
- A new algorithm has been implemented that better reduces the entropy of indexes.
- The API Reference section of the User’s Manual (and the matching docstrings) has been completely reviewed, expanded and corrected. This process has unveiled some errors and inconsistencies which have also been fixed.
- Fixed `VLArray.__getitem__()` to behave as expected in Python when using slices, instead of following the semantics of PyTables’ `read()` methods (e.g. reading just one element when no stop is provided). Fixes ticket #50.
- Removed implicit UTF-8 encoding from `VLArray` data using `vlstring` atoms. Now a variable-length string is stored as is, which lets users use any encoding of their choice, or none of them. A `vlunicode` atom will probably be added to the next release so as to fix ticket #51.
- Allow non-sequence objects to be passed to `VLArray.append()` when using an `object` atom. This was already possible in 1.x but stopped working when the old append syntax was dropped in 2.0. Fixes ticket #63.
- Changed `Cols.__len__()` to return the number of rows of the table or nested column (instead of the number of fields), like its counterparts in `Table` and `Column`.
- Python scalars cached in `AttributeSet` instances are now kept as NumPy objects instead of Python ones, because they do become NumPy objects when retrieved from disk. Fixes ticket #59.
- Avoid HDF5 error when appending an empty array to a `Table` (ticket #57) or `EArray` (ticket #49) dataset.
- Fix wrong implementation of the top-level `table.description._v_dflts` map, which was also including the pathnames of columns inside nested columns. Fixes ticket #45.
- Optimized the access to unaligned arrays in Numexpr between a 30% and a 70%.
- Fixed a die-hard bug that caused the loading of groups while closing a file. This only showed with certain usage patterns of the LRU cache (e.g. the one caused by `ManyNodesTestCase` in `test_indexes.py` under Pro).
- Avoid copious warnings about unused functions and variables when compiling Numexpr.
- Several fixes to Numexpr expressions with all constant values. Fixed tickets #53, #54, #55, #58. Reported bugs to mainstream developers.
- Solved an issue when trying to open one of the included test files in append mode on a system-wide installation by a normal user with no write privileges on it. The file isn’t being modified anyway, so the test is skipped then.
- Added a new benchmark to compare the I/O speed of `Array` and `EArray` objects with that of `cPickle`.
- The old `Row.__call__()` is no longer available as a public method. It was not documented, anyway. Fixes ticket #46.
- `Cols._f_close()` is no longer public. Fixes ticket #47.
- `Attributes._f_close()` is no longer public. Fixes ticket #52.
- The undocumented `Description.classdict` attribute has been completely removed. Fixes ticket #44.

## Changes from 2.0b1 to 2.0b2

- A very exhaustive overhauling of the User’s Manual is in process. The chapters 1 (Introduction), 2 (Installation), 3 (Tutorials) have been completed (and hopefully, the lines of code are easier to copy&paste now), while chapter 4 (API Reference) has been done up to (and including) the Table class. During this tedious (but critical in a library) overhauling work, we have tried hard to synchronize the text in the User’s Guide with that which appears on the docstrings.
- Removed the `recursive` argument in `Group._f_walkNodes()`. Using it with a false value was redundant with `Group._f_iterNodes()`. Fixes ticket #42.
- Removed the `coords` argument from `Table.read()`. It was undocumented and redundant with `Table.readCoordinates()`. Fixes ticket #41.
- Fixed the signature of `Group.__iter__()` (by removing its parameters).
- Added new `Table.coldescrs` and `Table.description._v_itemsize` attributes.
- Added a couple of new attributes for leaves:
  - `nrowsinbuf`: the number of rows that fit in the internal buffers.
  - `chunkshape`: the chunk size for chunked datasets.
- Fixed setuptools so that making an egg out of the PyTables 2 package is possible now.
- Added a new `tables.restrict_flavors()` function allowing to restrict available flavors to a given set. This can be useful e.g. if you want to force PyTables to get NumPy data out of an old, `numarray`-flavored PyTables file even if the `numarray` package is installed.
- Fixed a bug which caused filters of unavailable compression libraries to be loaded as using the default Zlib library, after issuing a warning. Added a new `FiltersWarning` and a `Filters.copy()`.

## Changes from 1.4.x to 2.0b1

### API additions

- `Column.createIndex()` has received a couple of new parameters: `optlevel` and `filters`. The first one sets the desired quality level of the index, while the second one allows the user to specify the filters for the index.
- `Table.indexprops` has been split into `Table.indexFilters` and `Table.autoIndex`. The later groups the functionality of the old `auto` and `reindex`.
- The new `Table.colpathnames` is a sequence which contains the full pathnames of all bottom-level columns in a table. This can be used to walk all `Column` objects in a table when used with `Table.colinstances`.
- The new `Table.colinstances` dictionary maps column pathnames to their associated `Column` or `Cols` object for simple or nested columns, respectively. This is similar to `Table.cols._f_col()`, but faster.
- `Row` has received a new `Row.fetch_all_fields()` method in order to return all the fields in the current row. This returns a NumPy void scalar for each call.
- New `tables.test(verbose=False, heavy=False)` high level function for interactively running the complete test suite from the Python console.
- Added a `tables.print_versions()` for easily getting the versions for all the software on which PyTables relies on.

### Backward-incompatible changes

- You can no longer mark a column for indexing in a `Col` declaration. The only way of creating an index for a column is to invoke the `createIndex()` method of the proper column object after the table has been created.
- Now the `Table.colnames` attribute is just a list of the names of top-level columns in a table. You can still get something similar to the old structure by using `Table.description._v_nestedNames`. See also the new `Table.colpathnames` attribute.
- The `File.objects`, `File.leaves` and `File.groups` dictionaries have been removed. If you still need this functionality, please use the `File.getNode()` and `File.walkNodes()` instead.
- `Table.removeIndex()` is no longer available; to remove an index on a column, one must use the `removeIndex()` method of the associated `Column` instance.
- `Column.dirty` is no longer available. If you want to check column index dirtiness, use `Column.index.dirty`.
- `complib` and `complevel` parameters have been removed from `File.createTable()`, `File.createEArray()`, `File.createCArray()` and `File.createVLArray()`. They were already deprecated in PyTables 1.x.
- The `shape` and `atom` parameters have been swapped in `File.createCArray()`. This has been done to be consistent with `Atom()` definitions (i.e. type comes before and shape after).

### Deprecated features

- `Node._v_rootgroup` has been removed. Please use `node._v_file.root` instead.
- The `Node._f_isOpen()` and `Leaf.isOpen()` methods have been removed. Please use the `Node._v_isopen` attribute instead (it is much faster).
- The `File.getAttrNode()`, `File.setAttrNode()` and `File.delAttrNode()` methods have been removed. Please use `File.getNodeAttr()`, `File.setNodeAttr()` and `File.delNodeAttr()` instead.
- `File.copyAttrs()` has been removed. Please use `File.copyNodeAttrs()` instead.
- The `table[colname]` idiom is no longer supported. You can use `table.cols._f_col(column)` for doing the same.

### API refinements

- `File.createEArray()` received a new `shape` parameter. This allows to not have to use the shape of the atom so as to set the shape of the underlying dataset on disk.
- All the leaf constructors have received a new `chunkshape` parameter that allows specifying the chunk sizes of datasets on disk.
- All `File.create*()` factories for `Leaf` nodes have received a new `byteorder` parameter that allows the user to specify the byteorder in which data will be written to disk (data in memory is now always handled in native order).

> Enjoy data!

> —The PyTables Team