X-Git-Url: https://git.tokkee.org/?p=sysdb.git;a=blobdiff_plain;f=ReleaseNotes;h=57b04e20460b9ff7240ead71fc23613b8eb79993;hp=6a987133027fd2ca57647de53566d6327a08c0ab;hb=0a3dd5b9b97e25156412a95bcecf25f8d75c72fc;hpb=4c23b7857aee9625cc008f169ffe88d2eef133e1 diff --git a/ReleaseNotes b/ReleaseNotes index 6a98713..57b04e2 100644 --- a/ReleaseNotes +++ b/ReleaseNotes @@ -6,6 +6,13 @@ and very detailed list of changes can be obtained from the Git logs for each release. + Compatibility Note: + ~~~~~~~~~~~~~~~~~~~ + All pre 1.0 development snapshots are not considered stable yet. That is, + all interfaces, including the plugin API, the network protocol, or the query + language, may change in an incompatible way at any time. All changes are + documented here. + About SysDB: ~~~~~~~~~~~~ “System DataBase” (SysDB) is a multi-backend system management and inventory @@ -17,7 +24,324 @@ Object names are canonicalized before they are added to the store to ensure a consistent view of your infrastructure. - + + +Version 0.8.0, Release Date: 2016-02-25 +--------------------------------------- + + This release focuses on a generic and much more powerful store (query) + interface and modular store implementations via plugins. The in-memory store + has been moved into a plugin on top of that. Various fixes and improvements, + mostly related to iterators, have been applied to the parser and query + language. + + * build system: + - Fix VPATH (out of tree) builds. + - Fixed a build issue (configure failure) when using RRDtool 1.5. + - Enable silent build rules by default (if supported). + - Fix build issues with GCC 5. + * parser: + - Add a new, standalone parser module based on a generic (public) AST + (abstract syntax tree). This replaces the (internal) parser of the + frontend module. + - Improved error reporting for syntactic and semantic errors by + propagating error messages in the parser and analyzer. + * core / store: + - Fully support array access and comparison (all element types). + - Service/metric parent host names are now available via the "hostname" + attribute (populated automatically). + - Rebase the store query interface on top of the new parser: store + implementations prepare actual queries from an AST, thus, allowing + multiple implementations of store readers. + - Querying the store (through the new interface) will emit data to an + arbitrary writer implementation allowing for more flexible post- + processing. + - Fixed a memory leak in the TIMESERIES implementation. + - data: Added support for boolean values; this is a backward incompatible + change which breaks the network encoding. + - Include nano-second fraction of datatime values (if any) in the string + format (including JSON query results). + - Add new callback type for store-readers (query plugins) based on the + new generic query interface. + - memstore: Refactor the in-memory store implementation into a separate + data-type. It implements the writer and query/reader APIs. + - The main, in-memory store is no longer enabled by default (see + store::memory plugin). Instead, all updates go through the plugin API + and are distributed to all registered writers. + - Handle hostname canonicalization in the generic plugin module instead + of requiring each writer plugin to do so. + - Do not canonicalize hostnames when fetching time-series. + - Determine an object's backends and update interval in the generic + plugin module and pass it on to writer plugins which are expected to + merge it with existing entries. + - Let the JSON formatter implement the writer API to make it more + flexible and uncouple it from the memstore module; let it support + arbitrary top-level objects (besides hosts). + - FETCH (in the core) now supports all object types and child objects as + well as fetching the base object without children / siblings. + * frontend: + - Added the SERVER_VERSION command; this allows clients to check for + supported servers. + - Fixed a memory leak in failed FETCH commands. + - Switch to the new, pluggable query interface: use registered plugins to + write to or query the SysDB store. + - Move protocol-specific (query) logic out of the core and into the + frontend (based on the new query API). + - A generic TIMESERIES implementation based on the query API replaces the + in-core implementation. + * query language: + - Check types of comparison operands and arithmetic expression and treat + mismatches as errors; previously those comparisons would fail silently. + - Added the 'NOT IN' operator. + - Support iterators (ANY/ALL) of arbitrary array expressions. + - Change child-object iterator syntax to 'ANY/ALL .name', that is, + require the field name and add support for accessing arbitrary fields. + - Added new field "value" for attribute values. + - Added new field "timeseries" (boolean) for metrics indicating whether + a time-series store is known to SysDB. + - Support attribute and backend iterators in FILTER expressions. + - Enforce that TIMESERIES end times are greater than start times. + - Added unary 'IS TRUE' / 'IS FALSE' operators. + - Iterator expressions may now reference a chain of sibling objects as + long as only one such reference is an iterator. For example, + `LOOKUP services MATCHING ANY host.backend =~ 'foo'`. + * testing: + - Use check "loop" tests; this will run all test-cases even if there are + failures. + - Create separate binaries for each test; this allows to run more in + parallel. + - Make it possible to run unit and integration tests separately; see + README. + + Stores: + * store::memory: Added a store-writer/store-reader plugin providing an + in-memory store replacing the previous, always-enabled store. It is based + on the 'memstore' module. + +Version 0.7.0, Release Date: 2015-02-21 +--------------------------------------- + + This release introduces various features related to running SysDB as an + agent. Objects may now also be submitted by arbitrary applications using the + network protocol. Two new plugins are available to collect local facter fact + values and to mirror all store operations to another instance. Various fixes + and improvements have been applied to the client program (sysdb). + + * build system: + - Added support for C++. + * sysdb: + - Fixed terminal resets on exit. + - Don't exit if the server closed the connection; instead reconnect + automatically on the next user input. + - Fixed string parsing: previously, quoted strings were not handled at + all and might have been terminated by an embedded semicolon. + - Also store failed commands in the history to enable history editing in + such cases. + - Improved program output: don't mix log messages with prompts after + sending a command; print log messages to the standard error channel. + - Fixed lookup of the .sysdb_history file for cases where the SysDB user + does not match the current operating system user. + * sysdbd: + - SSL options can be specified inside a block. + - Unload all plugins on exit to ensure a clean shutdown. + * client: + - Simplified RPC handling by providing a helper function. + - Added support for TCP connections using SSL/TLS and supporting, both, + IPv4 and IPv6. Remote peer verification is required for any such + connection. + - New command line flags -K, -C, and -A to specify the SSL private key, + client certificate, and CA certificates file names. + * core: + - Fully support arrays of all types. + - Fixed (implemented) reloading of time-series fetcher plugins. + - Added new callback type for "store-writers", making that part of the + store pluggable (queries are not supported). + * store: + - Added support for querying an object's siblings. + - Send all stored objects to store-writer plugins in addition to storing + them in the in-memory store. + * frontend: + - Fixed parsing of time values. + - Add support for storing hosts, services, metrics, and attributes using + the new 'STORE' command. + - Fixed handling of subsequent requests on the same connection by + avoiding unnecessary pauses of up to one second, thus, greatly speeding + up request handling. + - Added support for TCP connections supporting, both, IPv4 and IPv6. + SSL/TLS is enforced for any such connection. + - Added support for and require actual client authentication: + UNIX socket connections require peer authentication. + TCP connections require client SSL certificate authentication. + * query language: + - Added support for expressions accessing an object's siblings by using + the '.' syntax. + * JSON format: + - Fixed handling of special characters which are now being escaped. + - Include a flag indicating whether time-series data are available for a + metric. + * utils: + - os: This utility module provides some Operating System related helper + functions: access user information (current user, homedir), manage + directories (create, delete, resolve), I/O helpers (select, read, + write), networking helpers (DNS resolution). + - proto: Added support for marshaling and unmarshaling messages encoding + a datum of any type, hosts, services, metrics, and attributes. + - ssl: This new module provides convenience functions for SSL/TLS + servers, clients, and sessions using OpenSSL. + + Backends: + * facter: Added a plugin collecting local facter fact values using + libcfacter (C++). + + Stores: + * store::network: Added a store-writer plugin sending all data to another + SysDB instance using the client library and low-level STORE commands. + + Plugins: + * syslog: The log-level is now configurable using the "LogLevel" option. + +Version 0.6.0, Release Date: 2014-11-17 +--------------------------------------- + + This release focuses on a clearer query language. The underlying + architecture has been refactored to allow for more powerful and generic + expressions. + + * core: + - Added support for arrays. Elements may be of any other supported + data-type but only integers, decimals, and strings are fully supported. + * store: + - Refactored matchers: all matchers now operate on one or two expressions + or other matchers, replacing the old and less flexible conditional + matchers. + - Attribute values may now be used anywhere in an expression. + - On data-type mismatch when comparing attribute values, the values will + be cast to string (previously only done for regex matches). + - Added support for querying an object's backends (array of strings). + - Added support for iterating child objects and array values. + - All matchers now return false if any operand is NULL (except for IS + NULL). + - All operations now return NULL if any operand is NULL. + - Fixed behavior of not-operators (!=, !~) and let them not match if any + of the operands is NULL. + - Add support for looking up all object types (hosts, services, metrics). + - Refactored the JSON formatter to be more flexible and powerful. + - Record parent objects for each stored object. + * frontend: + - Added a separate parser mode for expressions. + - Added a new queryable field 'name' to be used in place of the rather + magical keyword 'host' when looking up hosts. + - Improved error reporting in the parser and the (newly introduced) + analyzer. + - Apply filters to hosts in the FETCH command. + * query language: + - All operators may now be used in all places if the involved data-types + support the respective operation. For example, regular expressions may + be used to match any value. Non-string values are cast to strings for + that purpose. IS NULL / IS NOT NULL may be applied to arbitrary + expressions. + - Introduced the 'IN' operator to check if a value or an array of values + is included in an array. + - Added support for arrays: [,,...] + - Added support for string and array concatenation using the '||' + operator. + - Attribute names have to be strings instead of identifiers now. + - Access object fields using '' rather than '.'. + - Let the LOOKUP and FETCH commands support services and metrics. + - Introduced 'FETCH service|metric . ' replaces + 'MATCHING service|metric|attribute ' expressions. + +Version 0.5.0, Release Date: 2014-10-06 +--------------------------------------- + + This release focuses on simplifications and more flexibility in the frontend + and the query language. Most of those changes are not backward-compatible + but easy client implementations. + + * sysdb: + - Include priority when printing log messages. + - Fixed handling of empty queries. + * store: + - When serializing the entire store (LIST command), return an array of + host objects (rather than wrapping it into another object). + - Fixed JSON format of time-series data. + - Fixed error checks of invalid compare expressions and reject them. + - Fixed lookups by back-end name. + - Added support for comparing attributes with different types by + comparing their string values. + * frontend: + - Clarified details about asynchronous messages and authentication + options and made client implementation more robust. + - Introduced a new message type encoding the JSON serialized response of + a query and its data type. + - Include the priority in log messages. + - Include object types in FETCH, LIST, and LOOKUP messages. + - Added support for including services and metrics in LIST responses and + skip hosts without the respective children. + * query language: + - Added support for services and metrics to LIST command. + - Changed syntax for attribute lookup to 'attributes[]'. + - Changed syntax for accessing queryable fields to '.'. + * utils: + - strbuf: Improved memory management. + +Version 0.4.0, Release Date: 2014-09-01 +--------------------------------------- + + This release features support for metrics and transparent access to a + backend's time-series data. Some backward-incompatible changes were applied + to some query commands for more consistency. + + * build system: + - Added checks for format strings and arguments. + - Changed configure option for collectd::unixsock to + --enable-collectd-unixsock. + * core: + - Added support for “metric” objects which may be assigned to a host. + - Introduced support for handling time-series data: added new data + structures describing time-series data and a new type of plugins called + “time-series fetchers” which are used to access time-series + information. + * store: + - Fixed a memory leak. + - Apply (very) simple optimizations to lookup expressions. + - Added support for metrics and their “data-stores” (describing how to + access the actual data referenced by a metric). + - Added support for querying time-series data through the newly + introduced “time-series fetcher” plugins. + * frontend: + - Fixed a bug that might have caused reading too much data from the + connection buffer in certain situations. + * query language: + - The ‘MATCHING’ clause is now optional in ‘LOOKUP’ queries. + - The ‘LIST’ and ‘FETCH’ commands support ‘FILTER’ clauses as well. + - ‘LIST’ / ‘FETCH’ were changed to ‘LIST hosts’ / ‘FETCH host’. + - Added the ‘TIMESERIES’ command to query arbitrary time-series which + fetches time-series data from a backend's data-store for a given period + of time. + - Added support for date/time values. + - Added support for single quotes in string values: two adjacent single + quotes may be used to specify a single quote. + * documentation: + - Documented the frontend protocol in frontend/proto.h. + - Documented changes to the query language and new / updated plugins. + - Documented all supported data types. + + Backends: + * collectd::unixsock: + - The plugin now uses metrics in place of services. + - New config options ‘TimeseriesBackend’ and ‘TimeseriesBaseURL’ to + specify access to time-series, currently limited to ‘rrdtool’ and + ‘rrdcached’ access. + + Plugins: + * timeseries::rrdtool: + - New plugin adding support to fetch time-series from local RRD files. + - Supports flushing of values in RRDCacheD. Version 0.3.0, Release Date: 2014-08-01 --------------------------------------- @@ -75,7 +399,7 @@ Version 0.3.0, Release Date: 2014-08-01 - Store plugin identifier (host, plugin, plugin_instance, type, type_instance) as service attributes. -Version 0.2.0, Release Date: 2014-07-01 +Version 0.2.0, Release Date: 2014-07-02 --------------------------------------- This is another development snapshot of SysDB. The main focus of this