1 SysDB Release Notes
2 =====================
4 The release notes contain noteworthy changes in each System DataBase (SysDB)
5 release, with a summary of the most important changes at the top. A complete
6 and very detailed list of changes can be obtained from the Git logs for each
7 release.
9 Compatibility Note:
10 ~~~~~~~~~~~~~~~~~~~
12 All pre 1.0 development snapshots are not considered stable yet. That is,
13 all interfaces, including the plugin API, the network protocol, or the query
14 language, may change in an incompatible way at any time. All changes are
15 documented here.
17 About SysDB:
18 ~~~~~~~~~~~~
19 “System DataBase” (SysDB) is a multi-backend system management and inventory
20 collection service. It stores system and inventory information about
21 hardware and software systems. This information is (continuously) collected
22 from various configurable backends (inventory services, monitoring services,
23 etc.) and stored in a graph-like hierarchy of generic objects. The store may
24 be queried through a generic interface independent of the active backends.
25 Object names are canonicalized before they are added to the store to ensure
26 a consistent view of your infrastructure.
28 <https://sysdb.io/>
30 Version 0.6.0, Release Date: 2014-11-17
31 ---------------------------------------
33 This release focuses on a clearer query language. The underlying
34 architecture has been refactored to allow for more powerful and generic
35 expressions.
37 * core:
38 - Added support for arrays. Elements may be of any other supported
39 data-type but only integers, decimals, and strings are fully supported.
40 * store:
41 - Refactored matchers: all matchers now operate on one or two expressions
42 or other matchers, replacing the old and less flexible conditional
43 matchers.
44 - Attribute values may now be used anywhere in an expression.
45 - On data-type mismatch when comparing attribute values, the values will
46 be cast to string (previously only done for regex matches).
47 - Added support for querying an object's backends (array of strings).
48 - Added support for iterating child objects and array values.
49 - All matchers now return false if any operand is NULL (except for IS
50 NULL).
51 - All operations now return NULL if any operand is NULL.
52 - Fixed behavior of not-operators (!=, !~) and let them not match if any
53 of the operands is NULL.
54 - Add support for looking up all object types (hosts, services, metrics).
55 - Refactored the JSON formatter to be more flexible and powerful.
56 - Record parent objects for each stored object.
57 * frontend:
58 - Added a separate parser mode for expressions.
59 - Added a new queryable field 'name' to be used in place of the rather
60 magical keyword 'host' when looking up hosts.
61 - Improved error reporting in the parser and the (newly introduced)
62 analyzer.
63 - Apply filters to hosts in the FETCH command.
64 * query language:
65 - All operators may now be used in all places if the involved data-types
66 support the respective operation. For example, regular expressions may
67 be used to match any value. Non-string values are cast to strings for
68 that purpose. IS NULL / IS NOT NULL may be applied to arbitrary
69 expressions.
70 - Introduced the 'IN' operator to check if a value or an array of values
71 is included in an array.
72 - Added support for arrays: [<elem1>,<elem2>,...]
73 - Added support for string and array concatenation using the '||'
74 operator.
75 - Attribute names have to be strings instead of identifiers now.
76 - Access object fields using '<field>' rather than '.<field>'.
77 - Let the LOOKUP and FETCH commands support services and metrics.
78 - Introduced 'FETCH service|metric <host>.<name' for fetching services
79 and metrics.
80 - Added support for 'ANY' and 'ALL' operators which apply a matcher to
81 each value emitted by an iterator (a host's services and metrics or
82 arrays). 'MATCHING ANY service|metric|attribute <cmp> <expr>' replaces
83 'MATCHING service|metric|attribute <cmp> <value>' expressions.
85 Version 0.5.0, Release Date: 2014-10-06
86 ---------------------------------------
88 This release focuses on simplifications and more flexibility in the frontend
89 and the query language. Most of those changes are not backward-compatible
90 but easy client implementations.
92 * sysdb:
93 - Include priority when printing log messages.
94 - Fixed handling of empty queries.
95 * store:
96 - When serializing the entire store (LIST command), return an array of
97 host objects (rather than wrapping it into another object).
98 - Fixed JSON format of time-series data.
99 - Fixed error checks of invalid compare expressions and reject them.
100 - Fixed lookups by back-end name.
101 - Added support for comparing attributes with different types by
102 comparing their string values.
103 * frontend:
104 - Clarified details about asynchronous messages and authentication
105 options and made client implementation more robust.
106 - Introduced a new message type encoding the JSON serialized response of
107 a query and its data type.
108 - Include the priority in log messages.
109 - Include object types in FETCH, LIST, and LOOKUP messages.
110 - Added support for including services and metrics in LIST responses and
111 skip hosts without the respective children.
112 * query language:
113 - Added support for services and metrics to LIST command.
114 - Changed syntax for attribute lookup to 'attributes[<name>]'.
115 - Changed syntax for accessing queryable fields to '.<field>'.
116 * utils:
117 - strbuf: Improved memory management.
119 Version 0.4.0, Release Date: 2014-09-01
120 ---------------------------------------
122 This release features support for metrics and transparent access to a
123 backend's time-series data. Some backward-incompatible changes were applied
124 to some query commands for more consistency.
126 * build system:
127 - Added checks for format strings and arguments.
128 - Changed configure option for collectd::unixsock to
129 --enable-collectd-unixsock.
130 * core:
131 - Added support for “metric” objects which may be assigned to a host.
132 - Introduced support for handling time-series data: added new data
133 structures describing time-series data and a new type of plugins called
134 “time-series fetchers” which are used to access time-series
135 information.
136 * store:
137 - Fixed a memory leak.
138 - Apply (very) simple optimizations to lookup expressions.
139 - Added support for metrics and their “data-stores” (describing how to
140 access the actual data referenced by a metric).
141 - Added support for querying time-series data through the newly
142 introduced “time-series fetcher” plugins.
143 * frontend:
144 - Fixed a bug that might have caused reading too much data from the
145 connection buffer in certain situations.
146 * query language:
147 - The ‘MATCHING’ clause is now optional in ‘LOOKUP’ queries.
148 - The ‘LIST’ and ‘FETCH’ commands support ‘FILTER’ clauses as well.
149 - ‘LIST’ / ‘FETCH’ were changed to ‘LIST hosts’ / ‘FETCH host’.
150 - Added the ‘TIMESERIES’ command to query arbitrary time-series which
151 fetches time-series data from a backend's data-store for a given period
152 of time.
153 - Added support for date/time values.
154 - Added support for single quotes in string values: two adjacent single
155 quotes may be used to specify a single quote.
156 * documentation:
157 - Documented the frontend protocol in frontend/proto.h.
158 - Documented changes to the query language and new / updated plugins.
159 - Documented all supported data types.
161 Backends:
162 * collectd::unixsock:
163 - The plugin now uses metrics in place of services.
164 - New config options ‘TimeseriesBackend’ and ‘TimeseriesBaseURL’ to
165 specify access to time-series, currently limited to ‘rrdtool’ and
166 ‘rrdcached’ access.
168 Plugins:
169 * timeseries::rrdtool:
170 - New plugin adding support to fetch time-series from local RRD files.
171 - Supports flushing of values in RRDCacheD.
173 Version 0.3.0, Release Date: 2014-08-01
174 ---------------------------------------
176 This release includes major enhancements and new features in the SysDB store
177 which is the core of the database. Most notably, complex expressions are now
178 supported when looking up information and filters may be used to preselect
179 the information to be returned. Some backward-incompatible changes were
180 introduced in the query language to avoid potentially confusing semantics.
182 * sysdb: Fixed non-interactive mode when using libedit.
183 * core:
184 - Fixed weird behavior when collector callbacks were registered without
185 any plugin context information (as it would, e.g., happen when using
186 LoadPlugin instead of LoadBackend in sysdbd).
187 * store:
188 - Introduced service attributes -- similar to host attributes but
189 assigned to a service object.
190 - Don't log “value too old” messages if an updated object uses the same
191 timestamp as the old object.
192 - Switched from linked-lists to AVL trees for storing all object types to
193 reduce lookup times from O(n) to O(log n).
194 - Added infrastructure for handling and evaluating arithmetic
195 expressions and filters (see also the changes to the query language
196 below).
197 - Export various fields describing base attributes of all stored object
198 types: last_update, age, interval, and backend. These fields may be
199 used when querying the store.
200 * frontend: Include new service attributes in JSON output.
201 * query language:
202 - Added support for ‘IS NULL’ attribute checks matching on non-existent
203 attributes.
204 - Changed ‘LOOKUP ... WHERE’ queries to ‘LOOKUP ... MATCHING’ to avoid
205 confusion with SQL semantics.
206 - Switched from ‘<obj>.name’ to ‘<obj>’ in MATCHING clauses, thus, no
207 longer shadowing attributes called “name”.
208 - Added support for arithmetic expressions in places that used simple
209 data values before. Arithmetic expressions support addition,
210 subtraction, multiplication, division, modulo, and concatenation
211 operations. The expressions may be based on constant values or
212 queryable fields of the stored objects (specified as ‘:last_update’,
213 ‘:age’, ‘:interval’, and ‘:backend’).
214 - Added ‘FILTER’ support to ‘LOOKUP’ queries. Filters are conditional
215 expressions which may optionally be used to limit the query and the
216 result to object (of any type) matching the filter condition.
217 - Added support to specify date and time values.
218 * utils:
219 - avltree: Added an AVL tree (self-balancing binary tree) implementation.
220 * documentation: Added documentation for the new query language features.
221 * testing: Run all unit tests through valgrind by default (if available).
223 Backends:
224 * collectd::unixsock:
225 - Added support for spaces in identifiers.
226 - Store plugin identifier (host, plugin, plugin_instance, type,
227 type_instance) as service attributes.
229 Version 0.2.0, Release Date: 2014-07-02
230 ---------------------------------------
232 This is another development snapshot of SysDB. The main focus of this
233 release is on stabilizing existing features and extending the query
234 infrastructure to support all attribute types and further operators.
236 * build system: Fixed linking of sysdbd to include *all* necessary object
237 files. This bug caused some plugins to be un-loadable.
238 * sysdbd: Fixed shutdown process in case of errors.
239 * sysdb: Fixed a bug causing some server replies to be ignored when exiting
240 from the client.
241 * core:
242 - Always log to the standard output stream if no user-provided logging
243 callbacks had been specified. Previously this was disabled after
244 enabling logging to clients in the frontend.
245 - Simplified plugin naming information and determine configuration
246 callback names automatically to ensure better consistency.
247 * store:
248 - Simplified internal data management.
249 - Store the list of backends along with each stored object which provide
250 the respective object.
251 * frontend:
252 - Simplified and improved the internal architecture matching / looking up
253 objects.
254 - Fixed parser to correctly reject invalid object types.
255 - Automatically create the base directory of a listening UNIX socket.
256 * query language: Added support for numeric constants and added operators
257 for less and greater than or equal attribute comparison.
258 * documentation: Added missing files to the distribution tarball.
260 Plugins:
261 * syslog: Don't log debug messages to syslog.
263 Version 0.1.0, Release Date: 2014-06-01
264 ---------------------------------------
266 This is the initial release of SysDB. It provides the following core
267 features:
269 * sysdb: A terminal-based, interactive client program. Supports
270 line-editing and history based on libreadline or libedit and uses an
271 asynchronous, event-driven architecture to handle user input and server
272 replies simultaneously.
273 * sysdbd: The database backend service configurable through a configuration
274 file. The daemon supports online reconfiguration.
275 * core: Flexible, multi-threaded core architecture providing a configurable
276 plugin infrastructure for querying external data-sources and extending
277 core functionality. The following plugin types are supported:
278 - data collection
279 - hostname canonicalization
280 - logging
281 * store: Central, in-memory object store managing host and service objects
282 and their attributes. Host names may be canonicalized through respective
283 plugins. Each object is automatically attributed with last update and
284 interval information using nano-seconds resolution. Attributes may have
285 various types (integer, decimal, string, date-time, binary).
286 * frontend: The frontend handling client connections uses a multi-threaded,
287 asynchronous, event-driven architecture capable of handling multiple
288 listen addresses.
289 * query language: The store may be queried using the ‘LIST’, ‘LOOKUP’, and
290 ‘FETCH’ commands returning JSON formatted objects.
291 * utils: A rich set of abstract data-types and utility functions for
292 internal use:
293 - generic channels for asynchronous, multi-threaded I/O
294 - DBI wrapper to ease common database access operations
295 - thread-aware error handling and reporting
296 - doubly linked-lists
297 - dynamic sized memory buffers supporting formatted strings and binary
298 data
299 - UNIX socket client implementation
300 * documentation: Full documentation for the tools, all plugins, and the
301 query language provided as manpages and HTML pages.
302 * testing: Unit and integration tests exist for most parts except plugins
303 covering 77% of all functions. Valgrind and Clang's address sanitzer are
304 used to detect memory leaks and programming errors.
306 The following plugins are shipped with this version of SysDB:
308 Backends:
309 * collectd::unixsock: Query collectd through its ‘unixsock’ interface
310 * mk-livestatus: Query arbitrary (monitoring) systems using the Check_MK
311 Livestatus interface.
312 * puppet::store-configs: Query Puppet through its “stored configuration”
313 database.
315 Plugins:
316 * cname::dns: Canonicalize hostnames by querying DNS information.
317 * syslog: Send log messages to the system log service.