Code

Aberrant Behavior Detection support. A brief overview added to rrdtool.pod.
authoroetiker <oetiker@a5681a0c-68f1-0310-ab6d-d61299d08faa>
Sun, 4 Mar 2001 13:01:56 +0000 (13:01 +0000)
committeroetiker <oetiker@a5681a0c-68f1-0310-ab6d-d61299d08faa>
Sun, 4 Mar 2001 13:01:56 +0000 (13:01 +0000)
Major updates to rrd_update.c, rrd_create.c. Minor update to other core files.
This is backwards compatible! But new files using the Aberrant stuff are not readable
by old rrdtool versions. See http://cricket.sourceforge.net/aberrant/rrd_hw.htm
-- Jake Brutlag <jakeb@corp.webtv.net>

git-svn-id: svn://svn.oetiker.ch/rrdtool/trunk/program@26 a5681a0c-68f1-0310-ab6d-d61299d08faa

20 files changed:
NEWS [new file with mode: 0644]
doc/rrdcreate.pod
doc/rrdgraph.pod
doc/rrdtool.pod
doc/rrdtune.pod
src/Makefile.am
src/fnv.h [new file with mode: 0644]
src/hash_32.c [new file with mode: 0644]
src/rrd_create.c
src/rrd_dump.c
src/rrd_format.h
src/rrd_graph.c
src/rrd_hw.c [new file with mode: 0644]
src/rrd_info.c
src/rrd_open.c
src/rrd_restore.c
src/rrd_tool.h
src/rrd_tune.c
src/rrd_update.c
src/rrdupdate.c [new file with mode: 0644]

diff --git a/NEWS b/NEWS
new file mode 100644 (file)
index 0000000..b9c3a64
--- /dev/null
+++ b/NEWS
@@ -0,0 +1,18 @@
+RRDTOOL NEWS
+============
+
+In this file I am noting the Major changes to rrdtool
+for details check the cvs ChangeLog
+
+2001/03/21 Tobias Oetiker <oetiker@ee.ethz.ch>
+  Added Aberrant Patch from Jake Brutlag <jakeb@corp.webtv.net>
+  From now one, new rrd files use version tag 0002. They can
+  NOT be read by the old 1.0.x rrdtools
+
+  Jack:
+  Aberrant Behavior Detection support. A brief overview added to
+  rrdtool.pod.  Major updates to rrd_update.c, rrd_create.c. Minor update to
+  other core files.  Updated documentation: rrdcreate.pod, rrdgraph.pod,
+  rrdtune.pod.  This is backwards compatible!  See
+  http://cricket.sourceforge.net/aberrant/rrd_hw.htm
+
index 8be05671cc3370901e7dd1533b6988bffe44c75d..14b6f1989c3efc85bf55c8deb6d45174b6d8f90e 100644 (file)
@@ -10,7 +10,7 @@ B<rrdtool> B<create> I<filename>
 S<[B<--start>|B<-b> I<start time>]> 
 S<[B<--step>|B<-s> I<step>]> 
 S<[B<DS:>I<ds-name>B<:>I<DST>B<:>I<heartbeat>B<:>I<min>B<:>I<max>]>
-S<[B<RRA:>I<CF>B<:>I<xff>B<:>I<steps>B<:>I<rows>]>
+S<[B<RRA:>I<CF>B<:>I<cf arguments>]>
 
 =head1 DESCRIPTION
 
@@ -106,30 +106,174 @@ I<If information on minimal/maximal expected values is available,
 always set the min and/or max properties. This will help RRDtool in
 doing a simple sanity check on the data supplied when running update.>
 
-=item B<RRA:>I<CF>B<:>I<xff>B<:>I<steps>B<:>I<rows>
-
+=item B<RRA:>I<CF>B<:>I<cf arguments>
+  
 The purpose of an B<RRD> is to store data in the round robin archives
-(B<RRA>). An archive consists of a number of data values from all the
-defined data-sources (B<DS>) and is defined with an B<RRA> line.
+(B<RRA>). An archive consists of a number of data values or statistics for 
+each of the defined data-sources (B<DS>) and is defined with an B<RRA> line.
 
 When data is entered into an B<RRD>, it is first fit into time slots of
 the length defined with the B<-s> option becoming a I<primary data point>.
 
-The data is also consolidated with the consolidation function (I<CF>)
-of the archive. The following consolidation functions are defined:
-B<AVERAGE>, B<MIN>, B<MAX>, B<LAST>.
+The data is also processed with the consolidation function (I<CF>)
+of the archive. There are several consolidation functions that consolidate
+primary data points via an aggregate function: B<AVERAGE>, B<MIN>, B<MAX>, B<LAST>.
+The format of B<RRA> line for these consolidation function is:
+
+B<RRA:>I<AVERAGE | MIN | MAX | LAST>B<:>I<xff>B<:>I<steps>B<:>I<rows>
 
 I<xff> The xfiles factor defines what part of a consolidation interval may
 be made up from I<*UNKNOWN*> data while the consolidated value is still
 regarded as known.
 
-I<steps> defines how many of these I<primary data points> are used to
-build a I<consolidated data point> which then goes into the archive.
+I<steps> defines how many of these I<primary data points> are used to build
+a I<consolidated data point> which then goes into the archive.
 
 I<rows> defines how many generations of data values are kept in an B<RRA>.
 
 =back
 
+=head1 Aberrant Behaviour detection with Holt-Winters forecasting
+
+by Jake Brutlag E<lt>jakeb@corp.webtv.netE<gt>
+
+In addition to the aggregate functions, there are a set of specialized
+functions that enable B<RRDtool> to provide data smoothing (via the
+Holt-Winters forecasting algorithm), confidence bands, and the flagging
+aberrant behavior in the data source time series:
+
+=over 4
+
+=item B<RRA:>I<HWPREDICT>B<:>I<rows>B<:>I<alpha>B<:>I<beta>B<:>I<seasonal period>B<:>I<rra num>
+
+=item B<RRA:>I<SEASONAL>B<:>I<seasonal period>B<:>I<gamma>B<:>I<rra num>
+
+=item B<RRA:>I<DEVSEASONAL>B<:>I<seasonal period>B<:>I<gamma>B<:>I<rra num>
+
+=item B<RRA:>I<DEVPREDICT>B<:>I<rows>B<:>I<rra num>
+
+=item B<RRA:>I<FAILURES>B<:>I<rows>B<:>I<threshold>B<:>I<window length>B<:>I<rra num>
+
+=back
+
+These B<RRAs> differ from the true consolidation functions in several ways.
+First, each of the B<RRA>s is updated once for every primary data point.
+Second, these B<RRAs> are interdependent. To generate real-time confidence
+bounds, then a matched set of HWPREDICT, SEASONAL, DEVSEASONAL, and
+DEVPREDICT must exist. Generating smoothed values of the primary data points
+requires both a HWPREDICT B<RRA> and SEASONAL B<RRA>. Aberrant behavior
+detection requires FAILURES, HWPREDICT, DEVSEASONAL, and SEASONAL.
+
+The actual predicted, or smoothed, values are stored in the HWPREDICT
+B<RRA>. The predicted deviations are store in DEVPREDICT (think a standard
+deviation which can be scaled to yield a confidence band). The FAILURES
+B<RRA> stores binary indicators. A 1 marks the indexed observation a
+failure; that is, the number of confidence bounds violations in the
+preceding window of observations met or exceeded a specified threshold. An
+example of using these B<RRAs> to graph confidence bounds and failures
+appears in L<rrdgraph>.
+
+The SEASONAL and DEVSEASONAL B<RRAs> store the seasonal coefficients for the
+Holt-Winters Forecasting algorithm and the seasonal deviations respectively.
+There is one entry per observation time point in the seasonal cycle. For
+example, if primary data points are generated every five minutes, and the
+seasonal cycle is 1 day, both SEASONAL and DEVSEASONAL with have 288 rows.
+
+In order to simplify the creation for the novice user, in addition to
+supporting explicit creation the HWPREDICT, SEASONAL, DEVPREDICT,
+DEVSEASONAL, and FAILURES B<RRAs>, the B<rrdtool> create command supports
+implicit creation of the other four when HWPREDICT is specified alone and
+the final argument I<rra num> is omitted.
+
+I<rows> specifies the length of the B<RRA> prior to wrap around. Remember
+that there is a one-to-one correspondence between primary data points and
+entries in these RRAs. For the HWPREDICT CF, I<rows> should be larger than
+the I<seasonal period>. If the DEVPREDICT B<RRA> is implicity created, the
+default number of rows is the same as the HWPREDICT I<rows> argument. If the
+FAILURES B<RRA> is implicitly created, I<rows> will be set to the I<seasonal
+period> argument of the HWPREDICT B<RRA>. Of course, the B<rrdtool>
+I<resize> command is available if these defaults are not sufficient and the
+create wishes to avoid explicit creations of the other specialized function
+B<RRAs>.
+
+I<seasonal period> specifies the number of primary data points in a seasonal
+cycle. If SEASONAL and DEVSEASONAL are implicitly created, this argument for
+those B<RRAs> is set automatically to the value specified by HWPREDICT. If
+they are explicity created, the creator should verify that all three
+I<seasonal period> arguments agree.
+
+I<alpha> is the adaptation parameter of the intercept (or baseline)
+coefficient in the Holt-Winters Forecasting algorithm. See L<rrdtool> for a
+description of this algorithm. I<alpha> must lie between 0 and 1. A value
+closer to 1 means that more recent observations carry greater weight in
+predicting the baseline component of the forecast. A value closer to 0 mean
+that past history carries greater weight in predicted the baseline
+component.
+
+I<beta> is the adaption parameter of the slope (or linear trend) coefficient
+in the Holt-Winters Forecating algorihtm. I<beta> must lie between 0 and 1
+and plays the same role as I<alpha> with respect to the predicted linear
+trend.
+
+I<gamma> is the adaption parameter of the seasonal coefficients in the
+Holt-Winters Forecasting algorithm (HWPREDICT) or the adaption parameter in
+the exponential smoothing update of the seasonal deviations. It must lie
+between 0 and 1. If the SEASONAL and DEVSEASONAL B<RRAs> are created
+implicitly, they will both have the same value for I<gamma>: the value
+specified for the HWPREDICT I<alpha> argument. Note that because there is
+one seasonal coefficient (or deviation) for each time point during the
+seasonal cycle, the adaption rate is much slower than the baseline. Each
+seasonal coefficient is only updated (or adapts) when the observed value
+occurs at the offset in the seasonal cycle corresponding to that
+coefficient.
+
+If SEASONAL and DEVSEASONAL B<RRAs> are created explicity, I<gamma> need not
+be the same for both. Note that I<gamma> can also be changed via the
+B<rrdtool> I<tune> command.
+
+I<rra num> provides the links between related B<RRAs>. If HWPREDICT is
+specified alone and the other B<RRAs> created implicitly, then there is no
+need to worry about this argument. If B<RRAs> are created explicitly, then
+pay careful attention to this argument. For each B<RRA> which includes this
+argument, there is a dependency between that B<RRA> and another B<RRA>. The
+I<rra num> argument is the 1-based index in the order of B<RRA> creation
+(that is, the order they appear in the I<create> command). The dependent
+B<RRA> for each B<RRA> requiring the I<rra num> argument is listed here:
+
+=over 4
+
+=item *
+
+HWPREDICT I<rra num> is the index of the SEASONAL B<RRA>.
+
+=item * 
+
+SEASONAL I<rra num> is the index of the HWPREDICT B<RRA>.
+
+=item * 
+
+DEVPREDICT I<rra num> is the index of the DEVSEASONAL B<RRA>.
+
+=item *
+
+DEVSEASONAL I<rra num> is the index of the HWPREDICT B<RRA>.
+
+=item * 
+
+FAILURES I<rra num> is the index of the DEVSEASONAL B<RRA>.
+
+=back
+
+I<threshold> is the minimum number of violations (observed values outside
+the confidence bounds) within a window that constitutes a failure. If the
+FAILURES B<RRA> is implicitly created, the default value is 7.
+
+I<window length> is the number of time points in the window. Specify an
+integer greater than or equal to the threshold and less than or equal to 28.
+The time interval this window represents depends on the interval between
+primary data points. If the FAILURES B<RRA> is implicity created, the
+default value is 9.
+
 =head1 The HEARTBEAT and the STEP
 
 Here is an explanation by Don Baarda on the inner workings of rrdtool.
@@ -232,6 +376,46 @@ every hour (12 * 300 seconds = 1 hour), for 100 days (2400 hours). The
 third and the fourth RRA's do the same with the for the maximum and
 average temperature, respectively.
 
+=head1 EXAMPLE 2
+
+C<rrdtool create monitor.rrd --step 300 
+DS:ifOutOctets:COUNTER:1800:0:4294967295
+RRA:AVERAGE:0.5:1:2016
+RRA:HWPREDICT:1440:0.1:0.0035:288>
+
+This example is a monitor of a router interface. The first B<RRA> tracks the
+traffic flow in octects; the second B<RRA> generates the specialized
+functions B<RRAs> for aberrant behavior detection. Note that the I<rra num>
+argument of HWPREDICT is missing, so the other B<RRAs> will be implicitly be
+created with default parameter values. In this example, the forecasting
+algorithm baseline adapts quickly; in fact the most recent one hour of
+observations (each at 5 minute intervals) account for 75% of the baseline
+prediction. The linear trend forecast adapts much more slowly. Observations
+made in during the last day (at 288 observations per day) account for only
+65% of the predicted linear trend. Note: these computations rely on an
+exponential smoothing formula described in a forthcoming LISA 2000 paper.
+
+The seasonal cycle is one day (288 data points at 300 second intervals), and
+the seasonal adaption paramter will be set to 0.1. The RRD file will store 5
+days (1440 data points) of forecasts and deviation predictions before wrap
+around. The file will store 1 day (a seasonal cycle) of 0-1 indicators in
+the FAILURES B<RRA>.
+
+The same RRD file and B<RRAs> are created with the following command, which explicitly
+creates all specialized function B<RRAs>.
+
+C<rrdtool create monitor.rrd --step 300 
+DS:ifOutOctets:COUNTER:1800:0:4294967295
+RRA:AVERAGE:0.5:1:2016
+RRA:HWPREDICT:1440:0.1:0.0035:288:3
+RRA:SEASONAL:288:0.1:2
+RRA:DEVPREDICT:1440:5
+RRA:DEVSEASONAL:288:0.1:2
+RRA:FAILURES:288:7:9:5>
+
+Of course, explicit creation need not replicate implicit create, a number of arguments 
+could be changed.
+
 =head1 AUTHOR
 
 Tobias Oetiker E<lt>oetiker@ee.ethz.chE<gt>
index 2e3a4f6470aa4a7d920c1aa23171253fc514ebf0..8e8038e98c1b56e1253d73907fd3a8b629fc1678 100644 (file)
@@ -41,6 +41,7 @@ S<[B<VRULE:>I<time>B<#>I<rrggbb>[B<:>I<legend>]]>
 S<[B<LINE>{B<1>|B<2>|B<3>}B<:>I<vname>[B<#>I<rrggbb>[B<:>I<legend>]]]>
 S<[B<AREA:>I<vname>[B<#>I<rrggbb>[B<:>I<legend>]]]>
 S<[B<STACK:>I<vname>[B<#>I<rrggbb>[B<:>I<legend>]]]>
+S<[B<TICK:>I<vname>B<#>I<rrggbb>[B<:>I<axis-fraction>[B<:>I<legend>]]]>
 
 =head1 DESCRIPTION
 
@@ -472,10 +473,18 @@ B<AREA> or B<LINE?> -- you need something to stack something onto in
 the first place ;) 
 
 Note, that when you STACK onto *UNKNOWN* data, rrdtool will not draw
-any graphics ... *UNKNOWN* is not zero ... if you want it to zero
+any graphics ... *UNKNOWN* is not zero ... if you want it to be zero
 then you might want to use a CDEF argument with IF and UN functions to
 turn *UNKNOWN* into zero ...
 
+=item B<TICK:>I<vname>B<#>I<rrggbb>[B<:>I<axis-fraction>[B<:>I<legend>]]
+
+Plot a tick mark (a vertical line) for each value of I<vname> that is
+non-zero and not *UNKNOWN*. The I<axis-fraction> argument specifies the
+length of the tick mark as a fraction of the y-axis; the default value
+is 0.1 (10% of the axis). Note that the color specification is not
+optional.
+
 =back
 
 =head1 NOTES on legend arguments
@@ -587,6 +596,61 @@ Note that this example assumes that your data is in the positive half of the y-a
 otherwhise you would would have to add NEGINF in order to extend the coverage
 of the rea to whole graph.
 
+=head1 EXAMPLE 4
+
+If the specialized function B<RRAs> exist for aberrant behavior detection, they
+can be used to generate the graph of a time series with confidence bands and
+failures.
+
+   rrdtool graph example.gif \
+          DEF:obs=monitor.rrd:ifOutOctets:AVERAGE \
+          DEF:pred=monitor.rrd:ifOutOctets:HWPREDICT \
+          DEF:dev=monitor.rrd:ifOutOctets:DEVPREDICT \
+          DEF:fail=monitor.rrd:ifOutOctets:FAILURES \
+          TICK:fail#ffffa0:1.0:"Failures\: Average bits out" \
+          CDEF:scaledobs=obs,8,* \
+          CDEF:upper=pred,dev,2,*,+ \
+          CDEF:lower=pred,dev,2,*,- \
+          CDEF:scaledupper=upper,8,* \
+          CDEF:scaledlower=lower,8,* \
+          LINE2:scaledobs#0000ff:"Average bits out" \
+          LINE1:scaledupper#ff0000:"Upper Confidence Bound: Average bits out" \
+          LINE1:scaledlower#ff0000:"Lower Confidence Bound: Average bits out"
+
+This example generates a graph of the data series in blue (LINE2 with the scaledobs
+virtual data source), confidence bounds in red (scaledupper and scaledlower virtual
+data sources), and potential failures (i.e. potential aberrant aberrant behavior)
+marked by vertical yellow lines (the fail data source).
+
+The raw data comes from an AVERAGE B<RRA>, the finest resolution of the observed
+time series (one consolidated data point per primary data point). The predicted
+(or smoothed) values are stored in the HWPREDICT B<RRA>. The predicted deviations
+(think standard deviation) values are stored in the DEVPREDICT B<RRA>. Finally,
+the FAILURES B<RRA> contains indicators, with 1 denoting a potential failure.
+
+All of the data is rescaled to bits (instead of Octets) by multiplying by 8.
+The confidence bounds are computed by an offset of 2 deviations both above
+and below the predicted values (the CDEFs upper and lower). Vertical lines
+indicated potential failures are graphed via the TICK graph element, which
+converts non-zero values in an B<RRA> into tick marks. Here an axis-fraction
+argument of 1.0 means the tick marks span the entire y-axis, and hence become
+vertical lines on the graph.
+
+The choice of 2 deviations (a scaling factor) matches the default used internally
+by the FAILURES B<RRA>. If the internal value is changed (see L<rrdtune>), this
+graphing command should be changed to be consistent.
+
+=head2 A note on data reduction:
+
+The B<rrdtool> I<graph> command is designed to plot data at a specified temporal
+resolution, regardless of the actually resolution of the data in the RRD file.
+This can present a problem for the specialized consolidation functions which
+maintain a one-to-one mapping between primary data points and consolidated
+data points. If a graph insists on viewing the contents of these B<RRAs> on a
+coarser temporal scale, the I<graph> command tries to do something intelligent,
+but the confidence bands and failures no longer have the same meaning and may
+be misleading.
+
 =head1 AUTHOR
 
 Tobias Oetiker E<lt>oetiker@ee.ethz.chE<gt>
index e4878636383492937e653fb79e19f7d4cf2ee067..fee6d356ba5b80d0c7ca70900d49c317bdcd96d4 100644 (file)
@@ -170,6 +170,63 @@ B<RRD>s. The graphing feature is fully configurable. Size, color and
 contents of the graph can be defined freely. Check L<rrdgraph>
 for more information on this.
 
+=item Aberrant Behavior Detection
+
+by Jake Brutlag E<lt>jakeb@corp.webtv.netE<gt>
+
+The  B<rrdtool> also provides the building blocks for near real-time
+aberrant behavior detection. These components include:
+
+=over 12
+
+=item *
+
+An algorithm for predicting the values time series one time step into the future.
+
+=item *
+
+A measure of deviation between the predicted values and the observed values.
+
+=item *
+
+A mechanism to decide if and when an observed value
+or sequence of observed values is I<too deviant> from the predicted value(s).
+
+=back
+
+Each of these components is briefly described:
+
+Holt-Winters Time Series Forecasting Algorithm is an online, or incremental, 
+algorithm that adaptively predicts future observations in a time series. It's 
+forecast is the sum of three components: a baseline (or intercept), a linear 
+trend over time (or slope), and a seasonal coefficient (a periodic effect, 
+such as a daily cycle). There is one seasonal coefficient for each time point 
+in the period (cycle). After a value is observed, each of these components is 
+updated via exponential smoothing. So the algorithm learns from past values 
+and uses them to predict the future. The rate of adaptation is governed by 
+3 parameters, alpha (intercept), beta (slope), and gamma (seasonal). The prediction 
+can also be viewed as a smoothed value for the time series.
+
+The measure of deviation is a seasonal weighted absolute deviation. The term 
+I<seasonal> means deviation is measured separately for each time point in the 
+seasonal cycle. As with Holt-Winters Forecasting, deviation is predicted using 
+the measure computed from past values (but only at that point in the seasonal cycle). 
+After the value is observed, the algorithm learns from the observed value via 
+exponential smoothing. Confidence bands for the observed time series are generated 
+by scaling the sequence of predicted deviation values (we usually think of the sequence 
+as a continuous line rather than a set of discrete points).
+
+Aberrant behavior (a potential failure) is reported whenever the number of 
+times the observed value violates the confidence bands meets or exceeds a 
+specified threshold within a specified temporal window (i.e. 5 violations 
+during the past 45 minutes with a value observed every 5 mintues).
+
+This functionality is embedded in a set of related B<RRAs>. In particular, a FAILURES
+B<RRA> logs potential failures. Presumably a front-end application to B<rrdtool> can
+utilize this B<RRA> to initiate real-time alerts if that is desired.
+
+You can find a detailed description of how to set this up in L<rrdcreate>.
+
 =back
 
 =head2 REMOTE CONTROL
index badd5d5f61f60bcadc4f206635b68e7959df7954..f96d8d1b01f4b0141cb6cc80c506fbd917e74624 100644 (file)
@@ -12,6 +12,13 @@ S<[B<--minimum>|B<-i> I<ds-name>:I<min>]>
 S<[B<--maximum>|B<-a> I<ds-name>:I<max>]>
 S<[B<--data-source-type>|B<-d> I<ds-name>:I<DST>]>
 S<[B<--data-source-rename>|B<-r> I<old-name>:I<new-name>]>
+S<[B<--deltapos> I<scale-value>]>
+S<[B<--deltaneg> I<scale value>]>
+S<[B<--failure-threshold> I<failure-threshold>]>
+S<[B<--window-length> I<window-length>]>
+S<[B<--alpha> I<adaption-parameter>]>
+S<[B<--beta> I<adaption-parameter>]>
+S<[B<--gamma> I<adaption-parameter>]>
 
 =head1 DESCRIPTION
 
@@ -21,11 +28,14 @@ All these tunable parameters together decide when data fed into an
 B<RRD> is to be regarded as invalid. Invalid data is entered into the 
 database as *UNKNOWN*.
 
-The main application of the B<tune> function is to relax the 
+One application of the B<tune> function is to relax the 
 validation rules on an B<RRD>. This allows to fill a new B<RRD> with
 data available in larger intervals than what you would normally want
 to permit.
 
+A second application of the B<tune> function is to set or alter parameters
+used by the specialized function B<RRAs> for aberrant behavior detection.
+
 =over 8
 
 =item I<filename>
@@ -51,13 +61,59 @@ Setting I<max> to 'U' will disable this limit.
 
 alter the type B<DST> of a data source.
 
-=item S<[B<--data-source-rename>|B<-r> I<old-name>:I<new-name>]>
+=item S<B<--data-source-rename>|B<-r> I<old-name>:I<new-name>>
 
 rename a data source
 
+=item S<B<--deltapos> I<scale-value>>
+
+Alter the deviation scaling factor for the upper bound of the confidence band
+used internally to calculate violations for the FAILURES B<RRA>. The default
+value is 2. Note that this parameter is not related to graphing confidence
+bounds, that scale factor is specified as a CDEV argument to generate a graph with
+confidence bounds. It need agree with the value used internally by the FAILURES
+B<RRA> (although common sense dictates it should).
+
+=item S<B<--deltaneg> I<scale-value>>
+
+Alter the deviation scaling factor for the lower bound of the confidence band
+used internally to calculate violations for the FAILURES B<RRA>. The default
+value is 2. As with B<--deltapos>, this argument is unrelated to the scale
+factor chosen when graphing confidence bounds.
+
+=item S<B<--failure-threshold> I<failure-threshold>>
+
+Alter the number of confidence bound violations that constitute a failure for
+purposes of the FAILURES B<RRA>. This must be an integer less than or equal to
+the window length of the FAILURES B<RRA>. This restriction is not verified by
+the tune option, so one can reset failure-threshold and window-length 
+simultaneously. Setting this option will reset the count of violations to 0.
+
+=item S<B<--window-length> I<window-length>>
+
+Alter the number of time points in the temporal window for determining failures.
+This must be an integer greater than or equal to the window length of the
+FAILURES B<RRA> and less than or equal to 28. Setting this option will reset the
+count of violations to 0.
+
+=item S<B<--alpha> I<adaption-parameter>>
+
+Alter the intercept adaptation parameter for the Holt-Winters forecasting algorithm.
+Must be between 0 and 1.
+
+=item S<B<--beta> I<adaption-parameter>>
+
+Alter the intercept adaptation parameter for the Holt-Winters forecasting algorithm.
+Must be between 0 and 1.
+
+=item S<B<--gamma> I<adaption-parameter>>
+
+Alter the seasonal coefficient and deviation adaptation parameters the SEASONAL and
+DEVSEAONAL B<RRAs>. Must be between 0 and 1.
+
 =back
 
-=head1 EXAMPLE
+=head1 EXAMPLE 1
 
 C<rrdtool tune data.rrd -h in:100000 -h out:100000 -h through:100000>
 
@@ -66,6 +122,14 @@ and 'through' to 10000 seconds which is a little over one day in data.rrd.
 This would allow to feed old data from mrtg-2.0 right into
 rrdtool without generating *UNKNOWN* entries.
 
+=head1 EXAMPLE 2
+
+C<rrdtool tune monitor.rrd --window-length 5 --failure-threshold 3>
+
+If the FAILURES B<RRA> is implicitly created, the default window-length is 9 and 
+the default failure-thresold is 7. This command now defines a failure as 3 or more
+violations in a temporal window of 5 time points.
+
 =head1 AUTHOR
 
 Tobias Oetiker <oetiker@ee.ethz.ch>
index 875c007d999f1a437a9071a87ae190e539e11327..33f679089a3d44b723865ed4130243f898c16ef4 100644 (file)
@@ -22,6 +22,8 @@ RRD_C_FILES =         \
        getopt1.c       \
        gifsize.c       \
        parsetime.c     \
+       hash_32.c       \
+        rrd_hw.c       \
        pngsize.c       \
        rrd_create.c    \
        rrd_diff.c      \
diff --git a/src/fnv.h b/src/fnv.h
new file mode 100644 (file)
index 0000000..205e5a8
--- /dev/null
+++ b/src/fnv.h
@@ -0,0 +1,102 @@
+/*
+ * fnv - Fowler/Noll/Vo- hash code
+ *
+ * @(#) $Revision$
+ * @(#) $Id$
+ * @(#) $Source$
+ *
+ ***
+ *
+ * Fowler/Noll/Vo- hash
+ *
+ * The basis of this hash algorithm was taken from an idea sent
+ * as reviewer comments to the IEEE POSIX P1003.2 committee by:
+ *
+ *      Phong Vo (http://www.research.att.com/info/kpv/)
+ *      Glenn Fowler (http://www.research.att.com/~gsf/)
+ *
+ * In a subsequent ballot round:
+ *
+ *      Landon Curt Noll (http://reality.sgi.com/chongo/)
+ *
+ * improved on their algorithm.  Some people tried this hash
+ * and found that it worked rather well.  In an EMail message
+ * to Landon, they named it the ``Fowler/Noll/Vo'' or FNV hash.
+ *
+ * FNV hashes are architected to be fast while maintaining a low
+ * collision rate. The FNV speed allows one to quickly hash lots
+ * of data while maintaining a reasonable collision rate.  See:
+ *
+ *      http://reality.sgi.com/chongo/tech/comp/fnv/
+ *
+ * for more details as well as other forms of the FNV hash.
+ *
+ ***
+ *
+ * NOTE: The FNV-0 historic hash is not recommended.  One should use
+ *      the FNV-1 hash instead.
+ *
+ * To use the 32 bit FNV-0 historic hash, pass FNV0_32_INIT as the
+ * Fnv32_t hashval argument to fnv_32_buf() or fnv_32_str().
+ *
+ * To use the 64 bit FNV-0 historic hash, pass FNV0_64_INIT as the
+ * Fnv64_t hashval argument to fnv_64_buf() or fnv_64_str().
+ *
+ * To use the recommended 32 bit FNV-1 hash, pass FNV1_32_INIT as the
+ * Fnv32_t hashval argument to fnv_32_buf() or fnv_32_str().
+ *
+ * To use the recommended 64 bit FNV-1 hash, pass FNV1_64_INIT as the
+ * Fnv64_t hashval argument to fnv_64_buf() or fnv_64_str().
+ *
+ ***
+ *
+ * Please do not copyright this code.  This code is in the public domain.
+ *
+ * LANDON CURT NOLL DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
+ * INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO
+ * EVENT SHALL LANDON CURT NOLL BE LIABLE FOR ANY SPECIAL, INDIRECT OR
+ * CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF
+ * USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+ * OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+ * PERFORMANCE OF THIS SOFTWARE.
+ *
+ * By:
+ *     chongo <Landon Curt Noll> /\oo/\
+ *     http://reality.sgi.com/chongo/
+ *     EMail: chongo_fnv at prime dot engr dot sgi dot com
+ *
+ * Share and Enjoy!    :-)
+ */
+
+#if !defined(__FNV_H__)
+#define __FNV_H__
+
+
+/*
+ * 32 bit FNV-0 hash type
+ */
+typedef unsigned long Fnv32_t;
+
+
+/* 
+ * 32 bit FNV-0 zero initial basis
+ *
+ * This historic hash is not recommended.  One should use
+ * the FNV-1 hash and inital basis instead.
+ */
+#define FNV0_32_INIT ((Fnv32_t)0)
+
+
+/*
+ * 32 bit FNV-1 non-zero initial basis
+ *
+ * The FNV-1 initial basis is the FNV-0 hash of the following 32 octets:
+ *
+ *              chongo <Landon Curt Noll> /\../\
+ *
+ * Note that the \'s above are not back-slashing escape characters.
+ * They are literal ASCII  backslash 0x5c characters.
+ */
+#define FNV1_32_INIT ((Fnv32_t)0x811c9dc5)
+
+#endif /* __FNV_H__ */
diff --git a/src/hash_32.c b/src/hash_32.c
new file mode 100644 (file)
index 0000000..3fda164
--- /dev/null
@@ -0,0 +1,152 @@
+/*
+ * hash_32 - 32 bit Fowler/Noll/Vo hash code
+ *
+ *
+ ***
+ *
+ * Fowler/Noll/Vo hash
+ *
+ * The basis of this hash algorithm was taken from an idea sent
+ * as reviewer comments to the IEEE POSIX P1003.2 committee by:
+ *
+ *      Phong Vo (http://www.research.att.com/info/kpv/)
+ *      Glenn Fowler (http://www.research.att.com/~gsf/)
+ *
+ * In a subsequent ballot round:
+ *
+ *      Landon Curt Noll (http://reality.sgi.com/chongo/)
+ *
+ * improved on their algorithm.  Some people tried this hash
+ * and found that it worked rather well.  In an EMail message
+ * to Landon, they named it the ``Fowler/Noll/Vo'' or FNV hash.
+ *
+ * FNV hashes are architected to be fast while maintaining a low
+ * collision rate. The FNV speed allows one to quickly hash lots
+ * of data while maintaining a reasonable collision rate.  See:
+ *
+ *      http://reality.sgi.com/chongo/tech/comp/fnv/
+ *
+ * for more details as well as other forms of the FNV hash.
+ ***
+ *
+ * NOTE: The FNV-0 historic hash is not recommended.  One should use
+ *      the FNV-1 hash instead.
+ *
+ * To use the 32 bit FNV-0 historic hash, pass FNV0_32_INIT as the
+ * Fnv32_t hashval argument to fnv_32_buf() or fnv_32_str().
+ *
+ * To use the recommended 32 bit FNV-1 hash, pass FNV1_32_INIT as the
+ * Fnv32_t hashval argument to fnv_32_buf() or fnv_32_str().
+ *
+ ***
+ *
+ * Please do not copyright this code.  This code is in the public domain.
+ *
+ * LANDON CURT NOLL DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
+ * INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO
+ * EVENT SHALL LANDON CURT NOLL BE LIABLE FOR ANY SPECIAL, INDIRECT OR
+ * CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF
+ * USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+ * OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+ * PERFORMANCE OF THIS SOFTWARE.
+ *
+ * By:
+ *     chongo <Landon Curt Noll> /\oo/\
+ *     http://reality.sgi.com/chongo/
+ *     EMail: chongo_fnv at prime dot engr dot sgi dot com
+ *
+ * Share and Enjoy!    :-)
+ */
+
+#include <stdlib.h>
+#include "fnv.h"
+
+
+/* 
+ * 32 bit magic FNV-0 and FNV-1 prime 
+ */
+#define FNV_32_PRIME ((Fnv32_t)0x01000193)     
+
+
+/*
+ * fnv_32_buf - perform a 32 bit Fowler/Noll/Vo hash on a buffer
+ *
+ * input:
+ *     buf     - start of buffer to hash
+ *     len     - length of buffer in octets
+ *     hval    - previous hash value or 0 if first call
+ *
+ * returns:
+ *     32 bit hash as a static hash type
+ *
+ * NOTE: To use the 32 bit FNV-0 historic hash, use FNV0_32_INIT as the hval
+ *      argument on the first call to either fnv_32_buf() or fnv_32_str().
+ *
+ * NOTE: To use the recommended 32 bit FNV-1 hash, use FNV1_32_INIT as the hval
+ *      argument on the first call to either fnv_32_buf() or fnv_32_str().
+ */
+Fnv32_t
+fnv_32_buf(void *buf, size_t len, Fnv32_t hval)
+{
+    unsigned char *bp = (unsigned char *)buf;  /* start of buffer */
+    unsigned char *be = bp + len;              /* beyond end of buffer */
+
+    /*
+     * FNV-1 hash each octet in the buffer
+     */
+    while (bp < be) {
+
+       /* multiply by the 32 bit FNV magic prime mod 2^64 */
+       hval *= FNV_32_PRIME;
+
+       /* xor the bottom with the current octet */
+       hval ^= (Fnv32_t)*bp++;
+    }
+
+    /* return our new hash value */
+    return hval;
+}
+
+
+/*
+ * fnv_32_str - perform a 32 bit Fowler/Noll/Vo hash on a string
+ *
+ * input:
+ *     str     - string to hash
+ *     hval    - previous hash value or 0 if first call
+ *
+ * returns:
+ *     32 bit hash as a static hash type
+ *
+ * NOTE: To use the 32 bit FNV-0 historic hash, use FNV0_32_INIT as the hval
+ *      argument on the first call to either fnv_32_buf() or fnv_32_str().
+ *
+ * NOTE: To use the recommended 32 bit FNV-1 hash, use FNV1_32_INIT as the hval
+ *      argument on the first call to either fnv_32_buf() or fnv_32_str().
+ */
+Fnv32_t
+fnv_32_str(char *str, Fnv32_t hval)
+{
+    unsigned char *s = (unsigned char *)str;   /* unsigned string */
+
+    /*
+     * FNV-1 hash each octet in the buffer
+     */
+    while (*s) {
+
+       /* multiply by the 32 bit FNV magic prime mod 2^64 */
+       hval *= FNV_32_PRIME;
+
+       /* xor the bottom with the current octet */
+       hval ^= (Fnv32_t)*s++;
+    }
+
+    /* return our new hash value */
+    return hval;
+}
+
+/* a wrapper function for fnv_32_str */
+unsigned long FnvHash(char *str)
+{
+  return fnv_32_str(str,FNV1_32_INIT);
+}
index f118da6fb11e9fdf0bfabd9c8cb7f6e2c589a484..d60e2e91a4208ed365beb5aa6e222fe84a867a07 100644 (file)
@@ -6,7 +6,10 @@
 
 #include "rrd_tool.h"
 
+/* prototype for FnvHash */
+unsigned long FnvHash(char *str);
 
+/* #define DEBUG */
 int
 rrd_create(int argc, char **argv) 
 {
@@ -15,7 +18,9 @@ rrd_create(int argc, char **argv)
     time_t             last_up;
     struct time_value last_up_tv;
     char *parsetime_error = NULL;
-
+    char *token;
+    unsigned short token_idx, error_flag, period=0;
+       unsigned long hashed_name;
     /* init last_up */
     last_up = time(NULL)-10;
     /* init rrd clean */
@@ -34,7 +39,8 @@ rrd_create(int argc, char **argv)
 
     /* set some defaults */
     strcpy(rrd.stat_head->cookie,RRD_COOKIE);
-    strcpy(rrd.stat_head->version,RRD_VERSION);
+       /* assume the will be version 1 compatible */
+    strcpy(rrd.stat_head->version,"0001");
     rrd.stat_head->float_cookie = FLOAT_COOKIE;
     rrd.stat_head->ds_cnt = 0; /* this will be adjusted later */
     rrd.stat_head->rra_cnt = 0; /* ditto */
@@ -103,7 +109,12 @@ rrd_create(int argc, char **argv)
        }
     }
     rrd.live_head->last_up = last_up;
-
+       
+       /* optind points to the first non-option command line arg,
+        * in this case, the file name. */
+       /* Compute the FNV hash value (used by SEASONAL and DEVSEASONAL
+        * arrays. */
+       hashed_name = FnvHash(argv[optind]);
     for(i=optind+1;i<argc;i++){
        char minstr[DS_NAM_SIZE], maxstr[DS_NAM_SIZE];  
        int ii;
@@ -162,36 +173,217 @@ rrd_create(int argc, char **argv)
        } else if (strncmp(argv[i],"RRA:",3)==0){
            size_t old_size = sizeof(rra_def_t)*(rrd.stat_head->rra_cnt);
            if((rrd.rra_def = rrd_realloc(rrd.rra_def,
-                                     old_size+sizeof(rra_def_t)))==NULL){
-               rrd_set_error("allocating rrd.rra_def");
-               rrd_free(&rrd);
-               return(-1);     
+                                     old_size+sizeof(rra_def_t)))==NULL)
+           {
+                  rrd_set_error("allocating rrd.rra_def");
+                  rrd_free(&rrd);
+                  return(-1);  
            }
            memset(&rrd.rra_def[rrd.stat_head->rra_cnt], 0, sizeof(rra_def_t));
-           if (sscanf(&argv[i][4],
-                      CF_NAM_FMT ":%lf:%lu:%lu",
-                      rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam,
-                      &rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_cdp_xff_val].u_val,
-                      &rrd.rra_def[rrd.stat_head->rra_cnt].pdp_cnt,
-                      &rrd.rra_def[rrd.stat_head->rra_cnt].row_cnt) == 4){
-               if(cf_conv(rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam) == -1){
-                   rrd_free(&rrd);
-                   return (-1);
+
+           token = strtok(&argv[i][4],":");
+           token_idx = error_flag = 0;
+           while (token != NULL)
+           {
+             switch(token_idx)
+             {
+             case 0:
+               if (sscanf(token,CF_NAM_FMT,
+                          rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam) != 1)
+                 rrd_set_error("Failed to parse CF name");
+               switch(cf_conv(rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam))
+               {
+               case CF_HWPREDICT:
+                 /* initialize some parameters */
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_hw_alpha].u_val = 0.1;
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_hw_beta].u_val = 1.0/288;
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt = 
+                   rrd.stat_head -> rra_cnt;
+                 /* need to mark the file version */
+          strcpy(rrd.stat_head->version,RRD_VERSION);
+                 break;
+               case CF_DEVSEASONAL:
+               case CF_SEASONAL:
+                 /* initialize some parameters */
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_seasonal_gamma].u_val = 0.1;
+                 /* fall through */
+               case CF_DEVPREDICT:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt = -1;
+                 break;
+               case CF_FAILURES:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_delta_pos].u_val = 2.0;
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_delta_neg].u_val = 2.0;
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_window_len].u_cnt = 3;
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_failure_threshold].u_cnt = 2;
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt = -1;
+                 break;
+               /* invalid consolidation function */
+               case -1:
+                 rrd_set_error("Unrecognized consolidation function %s",
+                        rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam);
+               default:
+                 break;
                }
-               if (rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_cdp_xff_val].u_val<0.0 ||
-                   rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_cdp_xff_val].u_val>=1.0) {
-                   rrd_set_error("the xff must always be >= 0 and < 1");
-                   rrd_free(&rrd);
-                   return (-1);
+               /* default: 1 pdp per cdp */ 
+               rrd.rra_def[rrd.stat_head->rra_cnt].pdp_cnt = 1;
+               break;
+             case 1:
+               switch(cf_conv(rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam))
+               {
+               case CF_HWPREDICT:
+               case CF_DEVSEASONAL:
+               case CF_SEASONAL:
+               case CF_DEVPREDICT:
+               case CF_FAILURES:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].row_cnt = atoi(token);
+                 break;
+               default:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_cdp_xff_val].u_val = atof(token);
+                 if (rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_cdp_xff_val].u_val<0.0 ||
+                     rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_cdp_xff_val].u_val>=1.0)
+                   rrd_set_error("Invalid xff: must be between 0 and 1");
+                 break;
                }
-               rrd.stat_head->rra_cnt++;                       
-           }
-           else {  
-               rrd_set_error("can't parse argument '%s'",argv[i]);
+               break;
+             case 2:
+               switch(cf_conv(rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam))
+               {
+               case CF_HWPREDICT:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_hw_alpha].u_val = atof(token);
+                 if (atof(token) <= 0.0 || atof(token) >= 1.0)
+                   rrd_set_error("Invalid alpha: must be between 0 and 1");
+                 break;
+               case CF_DEVSEASONAL:
+               case CF_SEASONAL:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_seasonal_gamma].u_val = 
+                       atof(token);
+                 if (atof(token) <= 0.0 || atof(token) >= 1.0)
+                   rrd_set_error("Invalid gamma: must be between 0 and 1");
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_seasonal_smooth_idx].u_cnt
+                       = hashed_name % rrd.rra_def[rrd.stat_head->rra_cnt].row_cnt; 
+                 break;
+           case CF_FAILURES:
+                 /* specifies the # of violations that constitutes the failure threshold */
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_failure_threshold].u_cnt =
+                       atoi(token);
+                 if (atoi(token) < 1 || atoi(token) > MAX_FAILURES_WINDOW_LEN)
+                       rrd_set_error("Failure threshold is out of range %d, %d",1,
+                         MAX_FAILURES_WINDOW_LEN);
+                 break;
+               case CF_DEVPREDICT:
+                 /* specifies the index (1-based) of CF_DEVSEASONAL array
+                  * associated with this CF_DEVPREDICT array. */
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt =
+                   atoi(token) - 1;
+                 break;
+               default:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].pdp_cnt = atoi(token);
+                 break;
+               }
+               break;
+             case 3:
+               switch(cf_conv(rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam))
+               {
+               case CF_HWPREDICT:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_hw_beta].u_val = atof(token);
+                 if (atof(token) < 0.0 || atof(token) > 1.0)
+                   rrd_set_error("Invalid beta: must be between 0 and 1");
+                 break;
+               case CF_DEVSEASONAL:
+               case CF_SEASONAL:
+                 /* specifies the index (1-based) of CF_HWPREDICT array
+                  * associated with this CF_DEVSEASONAL or CF_SEASONAL array. 
+                  * */
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt =
+                   atoi(token) - 1;
+                 break;
+               case CF_FAILURES:
+                 /* specifies the window length */
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_window_len].u_cnt =
+                       atoi(token);
+                 if (atoi(token) < 1 || atoi(token) > MAX_FAILURES_WINDOW_LEN)
+                       rrd_set_error("Window length is out of range %d, %d",1,
+                          MAX_FAILURES_WINDOW_LEN);
+                 /* verify that window length exceeds the failure threshold */
+                 if (rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_window_len].u_cnt <
+                         rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_failure_threshold].u_cnt)
+                   rrd_set_error("Window length is shorter than the failure threshold");
+                 break;
+               case CF_DEVPREDICT:
+                 /* shouldn't be any more arguments */
+                 rrd_set_error("Unexpected extra argument for consolidation function DEVPREDICT");
+                 break;
+               default:
+                 rrd.rra_def[rrd.stat_head->rra_cnt].row_cnt = atoi(token);
+                 break;
+               }
+               break;
+             case 4:
+               switch(cf_conv(rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam))
+               {
+               case CF_FAILURES:
+                 /* specifies the index (1-based) of CF_DEVSEASONAL array
+                  * associated with this CF_DEVFAILURES array. */
+                 rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt =
+                   atoi(token) - 1;
+                 break;
+               case CF_HWPREDICT:
+                 /* length of the associated CF_SEASONAL and CF_DEVSEASONAL arrays. */
+                 period = atoi(token);
+          if (period > rrd.rra_def[rrd.stat_head->rra_cnt].row_cnt)
+                   rrd_set_error("Length of seasonal cycle exceeds length of HW prediction array");
+                 break;
+               default:
+                 /* shouldn't be any more arguments */
+                 rrd_set_error("Unexpected extra argument for consolidation function %s",
+                               rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam);
+                  break;
+               }
+               break;
+             case 5:
+               /* If we are here, this must be a CF_HWPREDICT RRA.
+                * Specifies the index (1-based) of CF_SEASONAL array
+                * associated with this CF_HWPREDICT array. If this argument 
+                * is missing, then the CF_SEASONAL, CF_DEVSEASONAL, CF_DEVPREDICT,
+                * CF_FAILURES.
+                * arrays are created automatically. */
+               rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt =
+                 atoi(token) - 1;
+               break;
+             default:
+               /* should never get here */
+               rrd_set_error("Unknown error");
+               break;
+             } /* end switch */
+             if (rrd_test_error())
+             {
+               /* all errors are unrecoverable */
                rrd_free(&rrd);
-               return (-1);            
+               return (-1);
+             }
+             token = strtok(NULL,":");
+             token_idx++;
+           } /* end while */
+#ifdef DEBUG
+           fprintf(stderr,"Creating RRA CF: %s, dep idx %lu, current idx %lu\n",
+                   rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam,
+                   rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt, 
+                   rrd.stat_head -> rra_cnt);
+#endif
+           /* should we create CF_SEASONAL, CF_DEVSEASONAL, and CF_DEVPREDICT? */
+           if (cf_conv(rrd.rra_def[rrd.stat_head->rra_cnt].cf_nam) == CF_HWPREDICT
+               && rrd.rra_def[rrd.stat_head->rra_cnt].par[RRA_dependent_rra_idx].u_cnt 
+               == rrd.stat_head -> rra_cnt)
+           {
+#ifdef DEBUG
+             fprintf(stderr,"Creating HW contingent RRAs\n");
+#endif
+             if (create_hw_contingent_rras(&rrd,period,hashed_name) == -1) {
+               rrd_free(&rrd);
+               return -1;
+             }
            }
-
+           rrd.stat_head->rra_cnt++;                   
        } else {
            rrd_set_error("can't parse argument '%s'",argv[i]);
            rrd_free(&rrd);
@@ -214,6 +406,75 @@ rrd_create(int argc, char **argv)
     return rrd_create_fn(argv[optind],&rrd);
 }
 
+/* Create the CF_DEVPREDICT, CF_DEVSEASONAL, CF_SEASONAL, and CF_FAILURES RRAs
+ * associated with a CF_HWPREDICT RRA. */
+int
+create_hw_contingent_rras(rrd_t *rrd, unsigned short period, unsigned long hashed_name)
+{
+   size_t old_size;
+   rra_def_t* current_rra;
+   
+   /* save index to CF_HWPREDICT */
+   unsigned long hw_index = rrd -> stat_head -> rra_cnt;
+   /* advance the pointer */
+   (rrd -> stat_head -> rra_cnt)++;                    
+   /* allocate the memory for the 4 contingent RRAs */
+   old_size = sizeof(rra_def_t)*(rrd -> stat_head->rra_cnt);
+   if ((rrd -> rra_def = rrd_realloc(rrd -> rra_def,
+       old_size+4*sizeof(rra_def_t)))==NULL)
+   {
+     rrd_set_error("allocating rrd.rra_def");
+     return(-1);       
+   }
+   /* clear memory */
+   memset(&(rrd -> rra_def[rrd -> stat_head->rra_cnt]), 0, 4*sizeof(rra_def_t));
+
+   /* create the CF_SEASONAL RRA */
+   current_rra = &(rrd -> rra_def[rrd -> stat_head -> rra_cnt]);
+   strcpy(current_rra -> cf_nam,"SEASONAL");
+   current_rra -> row_cnt = period;
+   current_rra -> par[RRA_seasonal_smooth_idx].u_cnt = hashed_name % period;
+   current_rra -> pdp_cnt = 1;
+   current_rra -> par[RRA_seasonal_gamma].u_val = 
+     rrd -> rra_def[hw_index].par[RRA_hw_alpha].u_val;
+   current_rra -> par[RRA_dependent_rra_idx].u_cnt = hw_index; 
+   rrd -> rra_def[hw_index].par[RRA_dependent_rra_idx].u_cnt = rrd -> stat_head -> rra_cnt;
+
+   /* create the CF_DEVSEASONAL RRA */
+   (rrd -> stat_head -> rra_cnt)++; 
+   current_rra = &(rrd -> rra_def[rrd -> stat_head -> rra_cnt]);
+   strcpy(current_rra -> cf_nam,"DEVSEASONAL");
+   current_rra -> row_cnt = period;
+   current_rra -> par[RRA_seasonal_smooth_idx].u_cnt = hashed_name % period;
+   current_rra -> pdp_cnt = 1;
+   current_rra -> par[RRA_seasonal_gamma].u_val = 
+     rrd -> rra_def[hw_index].par[RRA_hw_alpha].u_val;
+   current_rra -> par[RRA_dependent_rra_idx].u_cnt = hw_index; 
+   
+   /* create the CF_DEVPREDICT RRA */
+   (rrd -> stat_head -> rra_cnt)++; 
+   current_rra = &(rrd -> rra_def[rrd -> stat_head -> rra_cnt]);
+   strcpy(current_rra -> cf_nam,"DEVPREDICT");
+   current_rra -> row_cnt = (rrd -> rra_def[hw_index]).row_cnt;
+   current_rra -> pdp_cnt = 1;
+   current_rra -> par[RRA_dependent_rra_idx].u_cnt 
+     = hw_index + 2; /* DEVSEASONAL */
+
+   /* create the CF_FAILURES RRA */
+   (rrd -> stat_head -> rra_cnt)++; 
+   current_rra = &(rrd -> rra_def[rrd -> stat_head -> rra_cnt]);
+   strcpy(current_rra -> cf_nam,"FAILURES");
+   current_rra -> row_cnt = period; 
+   current_rra -> pdp_cnt = 1;
+   current_rra -> par[RRA_delta_pos].u_val = 2.0;
+   current_rra -> par[RRA_delta_neg].u_val = 2.0;
+   current_rra -> par[RRA_failure_threshold].u_cnt = 7;
+   current_rra -> par[RRA_window_len].u_cnt = 9;
+   current_rra -> par[RRA_dependent_rra_idx].u_cnt = 
+        hw_index + 2; /* DEVSEASONAL */
+   return 0;
+}
+
 /* create and empty rrd file according to the specs given */
 
 int
@@ -266,21 +527,50 @@ rrd_create_fn(char *file_name, rrd_t *rrd)
        return(-1);
     }
 
-    /* can not be zero because we don't know nothing ... */
-    rrd->cdp_prep->scratch[CDP_val].u_val = DNAN;
-    for(i=0; i < rrd->stat_head->rra_cnt; i++) {
-
-       /* startup missing pdp count */
-       rrd->cdp_prep->scratch[CDP_unkn_pdp_cnt].u_cnt = 
-           ((rrd->live_head->last_up -
-            rrd->pdp_prep->scratch[PDP_unkn_sec_cnt].u_cnt)
-           % (rrd->stat_head->pdp_step 
-              * rrd->rra_def[i].pdp_cnt)) / rrd->stat_head->pdp_step;  
 
-
-       for(ii=0; ii < rrd->stat_head->ds_cnt; ii++) {
-           fwrite( rrd->cdp_prep,sizeof(cdp_prep_t),1,rrd_file);
-       }
+    for(i=0; i < rrd->stat_head->rra_cnt; i++) {
+       switch (cf_conv(rrd->rra_def[i].cf_nam))
+          {
+                 case CF_HWPREDICT:
+             rrd->cdp_prep->scratch[CDP_hw_intercept].u_val = DNAN;
+             rrd->cdp_prep->scratch[CDP_hw_last_intercept].u_val = DNAN;
+             rrd->cdp_prep->scratch[CDP_hw_slope].u_val = DNAN;
+             rrd->cdp_prep->scratch[CDP_hw_last_slope].u_val = DNAN;
+             rrd->cdp_prep->scratch[CDP_null_count].u_cnt = 1; 
+             rrd->cdp_prep->scratch[CDP_last_null_count].u_cnt = 1;
+                        break;
+                 case CF_SEASONAL:
+                 case CF_DEVSEASONAL:
+             rrd->cdp_prep->scratch[CDP_hw_seasonal].u_val = DNAN;
+             rrd->cdp_prep->scratch[CDP_hw_last_seasonal].u_val = DNAN;
+             rrd->cdp_prep->scratch[CDP_init_seasonal].u_cnt = 1; 
+                        break;
+                 case CF_FAILURES:
+             /* initialize violation history to 0 */
+                        for (ii = 0; ii < MAX_CDP_PAR_EN; ii++)
+                        {
+                               /* We can zero everything out, by setting u_val to the
+                                * NULL address. Each array entry in scratch is 8 bytes
+                                * (a double), but u_cnt only accessed 4 bytes (long) */
+                               rrd->cdp_prep->scratch[ii].u_val = 0.0;
+                        }
+                        break;
+                 default:
+             /* can not be zero because we don't know anything ... */
+             rrd->cdp_prep->scratch[CDP_val].u_val = DNAN;
+                /* startup missing pdp count */
+                rrd->cdp_prep->scratch[CDP_unkn_pdp_cnt].u_cnt = 
+                ((rrd->live_head->last_up -
+                rrd->pdp_prep->scratch[PDP_unkn_sec_cnt].u_cnt)
+                % (rrd->stat_head->pdp_step 
+                * rrd->rra_def[i].pdp_cnt)) / rrd->stat_head->pdp_step;        
+                    break;
+          }
+
+          for(ii=0; ii < rrd->stat_head->ds_cnt; ii++) 
+          {
+             fwrite( rrd->cdp_prep,sizeof(cdp_prep_t),1,rrd_file);
+          }
     }
 
     /* now, we must make sure that the rest of the rrd
@@ -292,13 +582,16 @@ rrd_create_fn(char *file_name, rrd_t *rrd)
        fclose(rrd_file);
        return(-1);
     }
-
-    rrd->rra_ptr->cur_row = 0;
-    for(i=0; i <rrd->stat_head->rra_cnt; i++)
-       fwrite( rrd->rra_ptr,
-               sizeof(rra_ptr_t), 1,rrd_file);
-
-
+       /* changed this initialization to be consistent with
+        * rrd_restore. With the old value (0), the first update
+        * would occur for cur_row = 1 because rrd_update increments
+        * the pointer a priori. */
+    for (i=0; i < rrd->stat_head->rra_cnt; i++)
+       {
+          rrd->rra_ptr->cur_row = rrd->rra_def[i].row_cnt - 1;
+          fwrite( rrd->rra_ptr, sizeof(rra_ptr_t),1,rrd_file);
+       }
 
     /* write the empty data area */
     for(i=0; 
index 95b5c52efd888c5e6591cd81f56f019c6e35fd89..a25fbe9e48d40f1de724c1ef4e2a9864231ac1a0 100644 (file)
@@ -5,8 +5,15 @@
  *****************************************************************************
  * $Id$
  * $Log$
- * Revision 1.1  2001/02/25 22:25:05  oetiker
- * Initial revision
+ * Revision 1.2  2001/03/04 13:01:55  oetiker
+ * Aberrant Behavior Detection support. A brief overview added to rrdtool.pod.
+ * Major updates to rrd_update.c, rrd_create.c. Minor update to other core files.
+ * This is backwards compatible! But new files using the Aberrant stuff are not readable
+ * by old rrdtool versions. See http://cricket.sourceforge.net/aberrant/rrd_hw.htm
+ * -- Jake Brutlag <jakeb@corp.webtv.net>
+ *
+ * Revision 1.1.1.1  2001/02/25 22:25:05  oetiker
+ * checkin
  *
  *****************************************************************************/
 
@@ -24,14 +31,15 @@ rrd_dump(int argc, char **argv)
     long         rra_base, rra_start, rra_next;
     FILE                  *in_file;
     rrd_t             rrd;
-
+    unival value;
 
     if(rrd_open(argv[1],&in_file,&rrd, RRD_READONLY)==-1){
        return(-1);
     }
+
     puts("<!-- Round Robin Database Dump -->");
     puts("<rrd>");
-    printf("\t<version> %s </version>\n",rrd.stat_head->version);
+    printf("\t<version> %s </version>\n",RRD_VERSION);
     printf("\t<step> %lu </step> <!-- Seconds -->\n",rrd.stat_head->pdp_step);
 #if HAVE_STRFTIME
     strftime(somestring,200,"%Y-%m-%d %H:%M:%S %Z",
@@ -83,22 +91,56 @@ rrd_dump(int argc, char **argv)
                       * sizeof(rrd_value_t));
        printf("\t<rra>\n");
        printf("\t\t<cf> %s </cf>\n",rrd.rra_def[i].cf_nam);
-       printf("\t\t<pdp_per_row> %lu </pdp_per_row> <!-- %lu seconds -->\n",
+       printf("\t\t<pdp_per_row> %lu </pdp_per_row> <!-- %lu seconds -->\n\n",
               rrd.rra_def[i].pdp_cnt, rrd.rra_def[i].pdp_cnt
               *rrd.stat_head->pdp_step);
-       printf("\t\t<xff> %0.10e </xff>\n\n",rrd.rra_def[i].par[RRA_cdp_xff_val].u_val);
+       /* added support for RRA parameters */
+       printf("\t\t<params>");
+       for (ii = 0; ii < MAX_RRA_PAR_EN; ii++)
+       {
+          value = rrd.rra_def[i].par[ii];
+          if (ii == RRA_dependent_rra_idx ||
+                  ii == RRA_seasonal_smooth_idx ||
+                  ii == RRA_failure_threshold)
+                 printf("<value> %lu </value>", value.u_cnt);
+          else {
+                 if (isnan(value.u_val)) {
+                     printf("<value> NaN </value>");
+                 } else {
+                     printf("<value> %0.10e </value>", value.u_val);
+                 }
+          }
+       }
+       printf("\t\t</params>\n");
        printf("\t\t<cdp_prep>\n");
        for(ii=0;ii<rrd.stat_head->ds_cnt;ii++){
-           double value = rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_val].u_val;
-           printf("\t\t\t<ds>");       
-           if (isnan(value)){
-             printf("<value> NaN </value>");
-           } else {
-             printf("<value> %0.10e </value>", value);
-           }
-           printf("  <unknown_datapoints> %lu </unknown_datapoints>",
-                   rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_unkn_pdp_cnt].u_cnt);
-          printf("</ds>\n");    
+#if 0
+               double value = rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_val].u_val;
+#endif 
+               printf("\t\t\t<ds>");
+               /* added support for exporting all CDP parameters */
+               for (iii=0; iii < MAX_CDP_PAR_EN ; iii++)
+               {
+                  value = rrd.cdp_prep[i*rrd.stat_head->ds_cnt + ii].scratch[iii];
+           /* handle integer values as a special case */
+                  if (cf_conv(rrd.rra_def[i].cf_nam) == CF_FAILURES ||
+                          iii == CDP_unkn_pdp_cnt || 
+                      iii == CDP_null_count ||
+                      iii == CDP_last_null_count)
+                    printf("<value> %lu </value>", value.u_cnt);
+                  else {       
+                    if (isnan(value.u_val)) {
+                      printf("<value> NaN </value>");
+                    } else {
+                      printf("<value> %0.10e </value>", value.u_val);
+                    }
+                  }
+               }
+#if 0 
+               printf("  <unknown_datapoints> %lu </unknown_datapoints>",
+                      rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_unkn_pdp_cnt].u_cnt);
+#endif
+        printf("</ds>\n");      
         }
        printf("\t\t</cdp_prep>\n");
 
index dde8bb9d1f1b5e53399689708db7d2de9a2f3773..3528362bd1a5908fa89033a388e1f6e5120f3280 100644 (file)
@@ -19,7 +19,7 @@
  *****************************************************************************/
 
 #define RRD_COOKIE    "RRD"
-#define RRD_VERSION   "0001"
+#define RRD_VERSION   "0002"
 #define FLOAT_COOKIE  8.642135E130
 
 #if defined(WIN32)
@@ -143,7 +143,7 @@ enum ds_param_en {   DS_mrhb_cnt=0,       /* minimum required heartbeat. A
                                           * least every ds_mrhb seconds,
                                           * otherwise it is regarded dead and
                                           * will be set to UNKNOWN */             
-                    DS_min_val,          /* the processed input of a ds must */
+                            DS_min_val,          /* the processed input of a ds must */
                      DS_max_val };        /* be between max_val and min_val
                                           * both can be set to UNKNOWN if you
                                           * do not care. Data outside the limits
@@ -168,11 +168,67 @@ typedef struct ds_def_t {
 enum cf_en           { CF_AVERAGE=0,     /* data consolidation functions */ 
                        CF_MINIMUM, 
                        CF_MAXIMUM,
-                       CF_LAST};
-
-enum rra_par_en {   RRA_cdp_xff_val=0};   /* what part of the consolidated 
-                                           datapoint may be unknown, while 
-                                           still a valid entry in goes into the rra */
+                       CF_LAST,
+                                          CF_HWPREDICT, 
+                                          /* An array of predictions using the seasonal 
+                                               * Holt-Winters algorithm. Requires an RRA of type
+                                               * CF_SEASONAL for this data source. */
+                                          CF_SEASONAL,
+                                          /* An array of seasonal effects. Requires an RRA of
+                                               * type CF_HWPREDICT for this data source. */
+                                          CF_DEVPREDICT,
+                                          /* An array of deviation predictions based upon
+                                               * smoothed seasonal deviations. Requires an RRA of
+                                               * type CF_DEVSEASONAL for this data source. */
+                                          CF_DEVSEASONAL,
+                                          /* An array of smoothed seasonal deviations. Requires
+                                               * an RRA of type CF_HWPREDICT for this data source.
+                                               * */
+                                          CF_FAILURES};
+                                          /* A binary array of failure indicators: 1 indicates
+                                               * that the number of violations in the prescribed
+                                               * window exceeded the prescribed threshold. */
+
+#define MAX_RRA_PAR_EN 10
+enum rra_par_en {   RRA_cdp_xff_val=0,  /* what part of the consolidated
+                     * datapoint must be known, to produce a
+                                        * valid entry in the rra */
+                                       RRA_hw_alpha,
+                                       /* exponential smoothing parameter for the intercept in
+                                        * the Holt-Winters prediction algorithm. */
+                                       RRA_hw_beta,
+                                       /* exponential smoothing parameter for the slope in
+                                        * the Holt-Winters prediction algorithm. */
+                                       RRA_dependent_rra_idx,
+                                       /* For CF_HWPREDICT: index of the RRA with the seasonal 
+                                        * effects of the Holt-Winters algorithm (of type
+                                        * CF_SEASONAL).
+                                        * For CF_DEVPREDICT: index of the RRA with the seasonal
+                                        * deviation predictions (of type CF_DEVSEASONAL).
+                                        * For CF_SEASONAL: index of the RRA with the Holt-Winters
+                                        * intercept and slope coefficient (of type CF_HWPREDICT).
+                                        * For CF_DEVSEASONAL: index of the RRA with the 
+                                        * Holt-Winters prediction (of type CF_HWPREDICT).
+                                        * For CF_FAILURES: index of the CF_DEVSEASONAL array.
+                                        * */
+                                       RRA_seasonal_smooth_idx,
+                                       /* For CF_SEASONAL and CF_DEVSEASONAL:
+                                        * an integer between 0 and row_count - 1 which
+                                        * is index in the seasonal cycle for applying
+                                        * the period smoother. */
+                                   RRA_failure_threshold,
+                                       /* For CF_FAILURES, number of violations within the last
+                                        * window required to mark a failure. */
+                    RRA_seasonal_gamma = RRA_hw_alpha,
+                                       /* exponential smoothing parameter for seasonal effects.
+                                        * */
+                    RRA_delta_pos = RRA_hw_alpha,
+                    RRA_delta_neg = RRA_hw_beta,
+                                       /* confidence bound scaling parameters for the
+                                        * the FAILURES RRA. */
+                    RRA_window_len = RRA_seasonal_smooth_idx};
+                                       /* For CF_FAILURES, the length of the window for measuring
+                                        * failures. */
                        
 #define CF_NAM_FMT    "%19[A-Z]"
 #define CF_NAM_SIZE   20
@@ -183,7 +239,7 @@ typedef struct rra_def_t {
     unsigned long    pdp_cnt;            /* how many primary data points are
                                          * required for a consolidated data
                                          * point?*/
-    unival           par[10];            /* index see rra_param_en */
+    unival           par[MAX_RRA_PAR_EN];            /* index see rra_param_en */
 
 } rra_def_t;
 
@@ -239,15 +295,56 @@ typedef struct pdp_prep_t{
 /****************************************************************************
  * POS 6: cdp_prep_t (* rra_cnt * ds_cnt )      data prep area for cdp values
  ****************************************************************************/
-enum cdp_par_en {  CDP_val=0,          /* the base_interval is always an
-                                         * average */
-                  CDP_unkn_pdp_cnt };       /* how many unknown pdp were
-                                                 * integrated. This and the cdp_xff
-                                           will decide if this is going to
-                                           be a UNKNOWN or a valid value */
+#define MAX_CDP_PAR_EN 10
+#define MAX_CDP_FAILURES_IDX 8 
+/* max CDP scratch entries avail to record violations for a FAILURES RRA */
+#define MAX_FAILURES_WINDOW_LEN 28
+enum cdp_par_en {  CDP_val=0,          
+                   /* the base_interval is always an
+                                       * average */
+                          CDP_unkn_pdp_cnt,       
+                                  /* how many unknown pdp were
+                           * integrated. This and the cdp_xff
+                                       * will decide if this is going to
+                                       * be a UNKNOWN or a valid value */
+                                  CDP_hw_intercept,
+                                  /* Current intercept coefficient for the Holt-Winters
+                                       * prediction algorithm. */
+                                  CDP_hw_last_intercept,
+                                  /* Last iteration intercept coefficient for the Holt-Winters
+                                       * prediction algorihtm. */
+                                  CDP_hw_slope,
+                                  /* Current slope coefficient for the Holt-Winters
+                                       * prediction algorithm. */
+                                  CDP_hw_last_slope,
+                                  /* Last iteration slope coeffient. */
+                                  CDP_null_count,
+                                  /* Number of sequential Unknown (DNAN) values + 1 preceding
+                                   * the current prediction.
+                                       * */
+                                  CDP_last_null_count,
+                                  /* Last iteration count of Unknown (DNAN) values. */
+                                  CDP_primary_val = 8,
+                                  /* optimization for bulk updates: the value of the first CDP
+                                       * value to be written in the bulk update. */
+                                  CDP_secondary_val = 9,
+                                  /* optimization for bulk updates: the value of subsequent
+                                       * CDP values to be written in the bulk update. */
+                   CDP_hw_seasonal = CDP_hw_intercept,
+                   /* Current seasonal coefficient for the Holt-Winters
+                    * prediction algorithm. This is stored in CDP prep to avoid
+                    * redundant seek operations. */
+                   CDP_hw_last_seasonal = CDP_hw_last_intercept,
+                   /* Last iteration seasonal coeffient. */
+                   CDP_seasonal_deviation = CDP_hw_intercept,
+                   CDP_last_seasonal_deviation = CDP_hw_last_intercept,
+                   CDP_init_seasonal = CDP_null_count};
+                   /* init_seasonal is a flag which when > 0, forces smoothing updates
+                    * to occur when rra_ptr.cur_row == 0 */
 
 typedef struct cdp_prep_t{
-    unival         scratch[10];          /* contents according to cdp_par_en *
+    unival         scratch[MAX_CDP_PAR_EN];          
+                                                                                /* contents according to cdp_par_en *
                                           * init state should be NAN */
 
 } cdp_prep_t;
@@ -283,7 +380,7 @@ typedef struct rrd_t {
  * Consolidated Data Points organized in Round Robin Archives.
  ****************************************************************************
  ****************************************************************************
+
  *RRA 0
  (0,0) .................... ( ds_cnt -1 , 0)
  .
index 136d3df47fe6587b6db9822f91c85a7c451effc1..f42acaf397b1980f5c966795f92b2a76bdc16140 100644 (file)
@@ -35,7 +35,7 @@ enum grc_en {GRC_CANVAS=0,GRC_BACK,GRC_SHADEA,GRC_SHADEB,
 
 
 enum gf_en {GF_PRINT=0,GF_GPRINT,GF_COMMENT,GF_HRULE,GF_VRULE,GF_LINE1,
-           GF_LINE2,GF_LINE3,GF_AREA,GF_STACK, GF_DEF, GF_CDEF };
+           GF_LINE2,GF_LINE3,GF_AREA,GF_STACK,GF_TICK,GF_DEF, GF_CDEF};
 
 enum op_en {OP_NUMBER=0,OP_VARIABLE,OP_INF,OP_PREV,OP_NEGINF,
            OP_UNKN,OP_NOW,OP_TIME,OP_LTIME,OP_ADD,OP_MOD,
@@ -346,6 +346,7 @@ enum gf_en gf_conv(char *string){
     conv_if(LINE3,GF_LINE3)
     conv_if(AREA,GF_AREA)
     conv_if(STACK,GF_STACK)
+       conv_if(TICK,GF_TICK)
     conv_if(DEF,GF_DEF)
     conv_if(CDEF,GF_CDEF)
     
@@ -687,12 +688,18 @@ reduce_data(
                if (isnan(newval)) newval = srcptr[i*(*ds_cnt)+col];
                else {
                    switch (cf) {
+                        case CF_HWPREDICT:
+                        case CF_DEVSEASONAL:
+                        case CF_DEVPREDICT:
+                        case CF_SEASONAL:
                        case CF_AVERAGE:
                            newval += srcptr[i*(*ds_cnt)+col];
                            break;
                        case CF_MINIMUM:
                            newval = min (newval,srcptr[i*(*ds_cnt)+col]);
                            break;
+                        case CF_FAILURES: 
+                        /* an interval contains a failure if any subintervals contained a failure */
                        case CF_MAXIMUM:
                            newval = max (newval,srcptr[i*(*ds_cnt)+col]);
                            break;
@@ -704,11 +711,16 @@ reduce_data(
            }
            if (validval == 0){newval = DNAN;} else{
                switch (cf) {
-                   case CF_AVERAGE:            
-                       newval /= validval;
+                    case CF_HWPREDICT:
+                    case CF_DEVSEASONAL:
+                    case CF_DEVPREDICT:
+                    case CF_SEASONAL:
+                    case CF_AVERAGE:                
+                        newval /= validval;
                        break;
                    case CF_MINIMUM:
-                   case CF_MAXIMUM:
+                    case CF_FAILURES:
+                   case CF_MAXIMUM:
                    case CF_LAST:
                        break;
                }
@@ -1389,6 +1401,7 @@ data_proc( image_desc_t *im ){
         (im->gdes[i].gf==GF_LINE2) ||
         (im->gdes[i].gf==GF_LINE3) ||
         (im->gdes[i].gf==GF_AREA) ||
+        (im->gdes[i].gf==GF_TICK) ||
         (im->gdes[i].gf==GF_STACK)){
        if((im->gdes[i].p_data = malloc((im->xsize +1)
                                        * sizeof(rrd_value_t)))==NULL){
@@ -1411,6 +1424,7 @@ data_proc( image_desc_t *im ){
            case GF_LINE2:
            case GF_LINE3:
            case GF_AREA:
+               case GF_TICK:
                paintval = 0.0;
            case GF_STACK:
                vidx = im->gdes[ii].vidx;
@@ -1430,7 +1444,8 @@ data_proc( image_desc_t *im ){
                if (! isnan(value)) {
                  paintval += value;
                  im->gdes[ii].p_data[i] = paintval;
-                 if (finite(paintval)){
+                 /* GF_TICK: the data values are not relevant for min and max */
+                 if (finite(paintval) && im->gdes[ii].gf != GF_TICK ){
                   if (isnan(minval) || paintval <  minval)
                     minval = paintval;
                   if (isnan(maxval) || paintval >  maxval)
@@ -1669,6 +1684,10 @@ print_calc(image_desc_t *im, char ***prdata)
                }
 
                switch (im->gdes[i].cf){
+               case CF_HWPREDICT:
+               case CF_DEVPREDICT:
+               case CF_DEVSEASONAL:
+               case CF_SEASONAL:
                case CF_AVERAGE:
                    validsteps++;
                    printval += im->gdes[vidx].data[ii];
@@ -1676,6 +1695,7 @@ print_calc(image_desc_t *im, char ***prdata)
                case CF_MINIMUM:
                    printval = min( printval, im->gdes[vidx].data[ii]);
                    break;
+           case CF_FAILURES:
                case CF_MAXIMUM:
                    printval = max( printval, im->gdes[vidx].data[ii]);
                    break;
@@ -1683,7 +1703,7 @@ print_calc(image_desc_t *im, char ***prdata)
                    printval = im->gdes[vidx].data[ii];
                }
            }
-           if (im->gdes[i].cf ==  CF_AVERAGE) {
+           if (im->gdes[i].cf ==  CF_AVERAGE || im -> gdes[i].cf > CF_LAST) {
                if (validsteps > 1) {
                    printval = (printval / validsteps);
                }
@@ -1738,6 +1758,7 @@ print_calc(image_desc_t *im, char ***prdata)
        case GF_LINE2:
        case GF_LINE3:
        case GF_AREA:
+       case GF_TICK:
        case GF_STACK:
        case GF_HRULE:
        case GF_VRULE:
@@ -2423,7 +2444,7 @@ MkLineBrush(image_desc_t *im,long cosel, enum gf_en typsel){
 
 int lazy_check(image_desc_t *im){
     FILE *fd = NULL;
-    int size = 1;
+       int size = 1;
     struct stat  gifstat;
     
     if (im->lazy == 0) return 0; /* no lazy option */
@@ -2438,11 +2459,11 @@ int lazy_check(image_desc_t *im){
       return 0; /* the file does not exist */
     switch (im->imgformat) {
     case IF_GIF:
-       size = GifSize(fd,&(im->xgif),&(im->ygif));
-       break;
+          size = GifSize(fd,&(im->xgif),&(im->ygif));
+          break;
     case IF_PNG:
-       size = PngSize(fd,&(im->xgif),&(im->ygif));
-       break;
+          size = PngSize(fd,&(im->xgif),&(im->ygif));
+          break;
     }
     fclose(fd);
     return size;
@@ -2592,7 +2613,22 @@ graph_paint(image_desc_t *im, char ***calcpr)
        case GF_COMMENT:
        case GF_HRULE:
        case GF_VRULE:
-         break;
+               break;
+       case GF_TICK:
+               for (ii = 0; ii < im->xsize; ii++)
+               {
+                  if (!isnan(im->gdes[i].p_data[ii]) && 
+                          im->gdes[i].p_data[ii] > 0.0)
+                  { 
+                         /* generate a tick */
+                         gdImageLine(gif, im -> xorigin + ii, 
+                                im -> yorigin - (im -> gdes[i].yrule * im -> ysize),
+                                im -> xorigin + ii, 
+                                im -> yorigin,
+                                im -> gdes[i].col.i);
+                  }
+               }
+               break;
        case GF_LINE1:
        case GF_LINE2:
        case GF_LINE3:
@@ -3181,6 +3217,48 @@ rrd_graph(int argc, char **argv, char ***prdata, int *xsize, int *ysize)
                return -1;
            }
            break;
+       case GF_TICK:
+           if((scancount=sscanf(
+               &argv[i][argstart],
+               "%29[^:#]#%2x%2x%2x:%lf:%n",
+               varname,
+               &col_red,
+               &col_green,
+               &col_blue,
+               &(im.gdes[im.gdes_c-1].yrule),
+               &strstart))>=1)
+               {
+               im.gdes[im.gdes_c-1].col.red = col_red;
+               im.gdes[im.gdes_c-1].col.green = col_green;
+               im.gdes[im.gdes_c-1].col.blue = col_blue;
+               if(strstart <= 0){
+                   im.gdes[im.gdes_c-1].legend[0] = '\0';
+               } else { 
+                   scan_for_col(&argv[i][argstart+strstart],FMT_LEG_LEN,im.gdes[im.gdes_c-1].legend);
+               }
+               if((im.gdes[im.gdes_c-1].vidx=find_var(&im,varname))==-1){
+                   im_free(&im);
+                   rrd_set_error("unknown variable '%s'",varname);
+                   return -1;
+               }
+               if (im.gdes[im.gdes_c-1].yrule <= 0.0 || im.gdes[im.gdes_c-1].yrule > 1.0)
+               {
+                   im_free(&im);
+                   rrd_set_error("Tick mark scaling factor out of range");
+                   return -1;
+               }
+               if (scancount < 4)
+                  im.gdes[im.gdes_c-1].col.red = -1;           
+           if (scancount < 5) 
+                  /* default tick marks: 10% of the y-axis */
+                  im.gdes[im.gdes_c-1].yrule = 0.1;
+
+               } else {
+                  im_free(&im);
+                  rrd_set_error("can't parse '%s'",&argv[i][argstart]);
+                  return -1;
+               } /* endif sscanf */
+               break;
        case GF_STACK:
            if(linepass == 0){
                im_free(&im);
diff --git a/src/rrd_hw.c b/src/rrd_hw.c
new file mode 100644 (file)
index 0000000..85b4fd7
--- /dev/null
@@ -0,0 +1,714 @@
+/*****************************************************************************
+ * RRDtool 1.0.21  Copyright Tobias Oetiker, 1997 - 2000
+ *****************************************************************************
+ * rrd_hw.c
+ *****************************************************************************
+ * Initial version by Jake Brutlag, WebTV Networks, 5/1/00
+ *****************************************************************************/
+
+#include "rrd_tool.h"
+
+/* #define DEBUG */
+
+unsigned long MyMod(signed long val, unsigned long mod);
+
+int update_hwpredict(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx, 
+                 unsigned long ds_idx, unsigned short CDP_scratch_idx);
+int update_seasonal(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx, 
+                                unsigned long ds_idx, unsigned short CDP_scratch_idx, 
+                                rrd_value_t *seasonal_coef);
+int update_devpredict(rrd_t *rrd, unsigned long cdp_idx, 
+                                 unsigned long rra_idx, unsigned long ds_idx, unsigned short CDP_scratch_idx);
+int update_devseasonal(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx, 
+                      unsigned long ds_idx, unsigned short CDP_scratch_idx, 
+                                  rrd_value_t *seasonal_dev);
+int update_failures(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx, 
+                               unsigned long ds_idx, unsigned short CDP_scratch_idx);
+
+int
+update_hwpredict(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx, 
+                 unsigned long ds_idx, unsigned short CDP_scratch_idx)
+{
+   rrd_value_t prediction, seasonal_coef;
+   unsigned long dependent_rra_idx, seasonal_cdp_idx;
+   unival *coefs = rrd -> cdp_prep[cdp_idx].scratch;
+   rra_def_t *current_rra = &(rrd -> rra_def[rra_idx]);
+
+   /* save coefficients from current prediction */
+   coefs[CDP_hw_last_intercept].u_val = coefs[CDP_hw_intercept].u_val;
+   coefs[CDP_hw_last_slope].u_val = coefs[CDP_hw_slope].u_val;
+   coefs[CDP_last_null_count].u_cnt = coefs[CDP_null_count].u_cnt;
+
+   /* retrieve the current seasonal coef */
+   dependent_rra_idx = current_rra -> par[RRA_dependent_rra_idx].u_cnt;
+   seasonal_cdp_idx = dependent_rra_idx*(rrd -> stat_head -> ds_cnt) + ds_idx;
+   if (dependent_rra_idx < rra_idx)
+         seasonal_coef = rrd -> cdp_prep[seasonal_cdp_idx].scratch[CDP_hw_last_seasonal].u_val;
+   else
+         seasonal_coef = rrd -> cdp_prep[seasonal_cdp_idx].scratch[CDP_hw_seasonal].u_val;
+   
+   /* compute the prediction */
+   if (isnan(coefs[CDP_hw_intercept].u_val) || isnan(coefs[CDP_hw_slope].u_val)
+      || isnan(seasonal_coef))
+   {
+     prediction = DNAN;
+      
+     /* bootstrap initialization of slope and intercept */
+     if (isnan(coefs[CDP_hw_intercept].u_val) &&
+        !isnan(coefs[CDP_scratch_idx].u_val)) 
+     {
+#ifdef DEBUG
+       fprintf(stderr,"Initialization of slope/intercept\n");
+#endif
+       coefs[CDP_hw_intercept].u_val = coefs[CDP_scratch_idx].u_val;
+       coefs[CDP_hw_last_intercept].u_val = coefs[CDP_scratch_idx].u_val;
+       /* initialize the slope to 0 */
+       coefs[CDP_hw_slope].u_val = 0.0;
+       coefs[CDP_hw_last_slope].u_val = 0.0;
+       /* initialize null count to 1 */
+       coefs[CDP_null_count].u_cnt = 1;
+       coefs[CDP_last_null_count].u_cnt = 1;
+     }
+     /* if seasonal coefficient is NA, then don't update intercept, slope */
+   } else {
+     prediction = coefs[CDP_hw_intercept].u_val + 
+       (coefs[CDP_hw_slope].u_val)*(coefs[CDP_null_count].u_cnt)
+       + seasonal_coef;
+#ifdef DEBUG
+     fprintf(stderr,"computed prediction: %f\n",prediction);
+#endif
+     if (isnan(coefs[CDP_scratch_idx].u_val))
+     {
+       /* NA value, no updates of intercept, slope;
+           * increment the null count */
+       (coefs[CDP_null_count].u_cnt)++;
+     } else {
+#ifdef DEBUG
+       fprintf(stderr,"Updating intercept, slope\n");
+#endif
+       /* update the intercept */
+       coefs[CDP_hw_intercept].u_val = (current_rra -> par[RRA_hw_alpha].u_val)*
+               (coefs[CDP_scratch_idx].u_val - seasonal_coef) +
+               (1 - current_rra -> par[RRA_hw_alpha].u_val)*(coefs[CDP_hw_intercept].u_val
+               + (coefs[CDP_hw_slope].u_val)*(coefs[CDP_null_count].u_cnt));
+       /* update the slope */
+       coefs[CDP_hw_slope].u_val = (current_rra -> par[RRA_hw_beta].u_val)*
+               (coefs[CDP_hw_intercept].u_val - coefs[CDP_hw_last_intercept].u_val) +
+               (1 - current_rra -> par[RRA_hw_beta].u_val)*(coefs[CDP_hw_slope].u_val);
+       /* reset the null count */
+       coefs[CDP_null_count].u_cnt = 1;
+     }
+   }
+
+   /* store the prediction for writing */
+   coefs[CDP_scratch_idx].u_val = prediction;
+   return 0;
+}
+
+int
+lookup_seasonal(rrd_t *rrd, unsigned long rra_idx, unsigned long rra_start,
+                               FILE *rrd_file, unsigned long offset, rrd_value_t **seasonal_coef)
+{
+   unsigned long pos_tmp;
+   /* rra_ptr[].cur_row points to the rra row to be written; this function
+       * reads cur_row + offset */
+   unsigned long row_idx = rrd -> rra_ptr[rra_idx].cur_row + offset;
+   /* handle wrap around */
+   if (row_idx >= rrd -> rra_def[rra_idx].row_cnt)
+     row_idx = row_idx % (rrd -> rra_def[rra_idx].row_cnt);
+
+   /* rra_start points to the appropriate rra block in the file */
+   /* compute the pointer to the appropriate location in the file */
+   pos_tmp = rra_start + (row_idx)*(rrd -> stat_head -> ds_cnt)*sizeof(rrd_value_t);
+
+   /* allocate memory if need be */
+   if (*seasonal_coef == NULL)
+         *seasonal_coef = 
+            (rrd_value_t *) malloc((rrd -> stat_head -> ds_cnt)*sizeof(rrd_value_t));
+   if (*seasonal_coef == NULL) {
+         rrd_set_error("memory allocation failure: seasonal coef");
+         return -1;
+   }
+
+   if (!fseek(rrd_file,pos_tmp,SEEK_SET))
+   {
+      if (fread(*seasonal_coef,sizeof(rrd_value_t),rrd->stat_head->ds_cnt,rrd_file)
+                 == rrd -> stat_head -> ds_cnt)
+         {
+                /* success! */
+         /* we can safely ignore the rule requiring a seek operation between read
+          * and write, because this read moves the file pointer to somewhere
+          * in the file other than the next write location.
+          * */
+                return 0;
+         } else {
+            rrd_set_error("read operation failed in lookup_seasonal(): %lu\n",pos_tmp);
+         }
+   } else {
+         rrd_set_error("seek operation failed in lookup_seasonal(): %lu\n",pos_tmp);
+   }
+   
+   return -1;
+}
+
+int
+update_seasonal(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx, 
+                                unsigned long ds_idx, unsigned short CDP_scratch_idx, rrd_value_t *seasonal_coef)
+{
+/* TODO: extract common if subblocks in the wake of I/O optimization */
+   rrd_value_t intercept, seasonal;
+   rra_def_t *current_rra = &(rrd -> rra_def[rra_idx]);
+   rra_def_t *hw_rra = &(rrd -> rra_def[current_rra -> par[RRA_dependent_rra_idx].u_cnt]);
+   /* obtain cdp_prep index for HWPREDICT */
+   unsigned long hw_cdp_idx = (current_rra -> par[RRA_dependent_rra_idx].u_cnt)
+      * (rrd -> stat_head -> ds_cnt) + ds_idx;
+   unival *coefs = rrd -> cdp_prep[hw_cdp_idx].scratch;
+
+   /* update seasonal coefficient in cdp prep areas */
+   seasonal = rrd -> cdp_prep[cdp_idx].scratch[CDP_hw_seasonal].u_val;
+   rrd -> cdp_prep[cdp_idx].scratch[CDP_hw_last_seasonal].u_val = seasonal;
+   rrd -> cdp_prep[cdp_idx].scratch[CDP_hw_seasonal].u_val =
+         seasonal_coef[ds_idx];
+
+   /* update seasonal value for disk */
+   if (current_rra -> par[RRA_dependent_rra_idx].u_cnt < rra_idx)
+         /* associated HWPREDICT has already been updated */
+         /* check for possible NA values */
+      if (isnan(rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val))
+         {
+                /* no update, store the old value unchanged,
+                 * doesn't matter if it is NA */
+            rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val = seasonal;
+         } else if (isnan(coefs[CDP_hw_last_intercept].u_val) 
+                    || isnan(coefs[CDP_hw_last_slope].u_val))
+         {
+                /* this should never happen, as HWPREDICT was already updated */
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val= DNAN;
+         } else if (isnan(seasonal))
+         {
+                /* initialization: intercept is not currently being updated */
+#ifdef DEBUG
+                fprintf(stderr,"Initialization of seasonal coef %lu\n",
+                       rrd -> rra_ptr[rra_idx].cur_row);
+#endif
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val 
+                       -= coefs[CDP_hw_last_intercept].u_val; 
+         } else {
+                intercept = coefs[CDP_hw_intercept].u_val;
+#ifdef DEBUG
+                fprintf(stderr,
+                       "Updating seasonal, params: gamma %f, new intercept %f, old seasonal %f\n",
+                       current_rra -> par[RRA_seasonal_gamma].u_val,
+                       intercept, seasonal);
+#endif
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val =
+                       (current_rra -> par[RRA_seasonal_gamma].u_val)*
+                       (rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val - intercept) +
+                       (1 - current_rra -> par[RRA_seasonal_gamma].u_val)*seasonal;
+         }
+   else {
+         /* SEASONAL array is updated first, which means the new intercept
+          * hasn't be computed; so we compute it here. */
+
+         /* check for possible NA values */
+      if (isnan(rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val))
+         {
+                /* no update, simple store the old value unchanged,
+                 * doesn't matter if it is NA */
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val =  seasonal;
+         } else if (isnan(coefs[CDP_hw_intercept].u_val) 
+                    || isnan(coefs[CDP_hw_slope].u_val))
+         {
+                /* Initialization of slope and intercept will occur.
+                 * force seasonal coefficient to 0. */
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val= 0.0;
+         } else if (isnan(seasonal))
+         {
+                /* initialization: intercept will not be updated
+                 * CDP_hw_intercept = CDP_hw_last_intercept; just need to 
+                 * subtract this baseline value. */
+#ifdef DEBUG
+                fprintf(stderr,"Initialization of seasonal coef %lu\n",
+                       rrd -> rra_ptr[rra_idx].cur_row);
+#endif
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val -= coefs[CDP_hw_intercept].u_val; 
+         } else {
+                /* Note that we must get CDP_scratch_idx from SEASONAL array, as CDP_scratch_idx
+                 * for HWPREDICT array will be DNAN. */
+            intercept = (hw_rra -> par[RRA_hw_alpha].u_val)*
+                   (rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val - seasonal)
+                   + (1 - hw_rra -> par[RRA_hw_alpha].u_val)*(coefs[CDP_hw_intercept].u_val
+                   + (coefs[CDP_hw_slope].u_val)*(coefs[CDP_null_count].u_cnt));
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val =
+                  (current_rra -> par[RRA_seasonal_gamma].u_val)*
+                  (rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val - intercept) +
+                  (1 - current_rra -> par[RRA_seasonal_gamma].u_val)*seasonal;
+         }
+   }
+#ifdef DEBUG
+   fprintf(stderr,"seasonal coefficient set= %f\n",
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val);
+#endif
+   return 0;
+}
+
+int
+update_devpredict(rrd_t *rrd, unsigned long cdp_idx, 
+                                 unsigned long rra_idx, unsigned long ds_idx, unsigned short CDP_scratch_idx)
+{
+   /* there really isn't any "update" here; the only reason this information
+    * is stored separately from DEVSEASONAL is to preserve deviation predictions
+    * for a longer duration than one seasonal cycle. */
+   unsigned long seasonal_cdp_idx = (rrd -> rra_def[rra_idx].par[RRA_dependent_rra_idx].u_cnt)
+      * (rrd -> stat_head -> ds_cnt) + ds_idx;
+
+   if (rrd -> rra_def[rra_idx].par[RRA_dependent_rra_idx].u_cnt < rra_idx)
+   {
+         /* associated DEVSEASONAL array already updated */
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val
+                = rrd -> cdp_prep[seasonal_cdp_idx].scratch[CDP_last_seasonal_deviation].u_val;
+   } else {
+         /* associated DEVSEASONAL not yet updated */
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val
+                = rrd -> cdp_prep[seasonal_cdp_idx].scratch[CDP_seasonal_deviation].u_val;
+   }
+   return 0;
+}
+
+int
+update_devseasonal(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx, 
+                      unsigned long ds_idx, unsigned short CDP_scratch_idx, 
+                                  rrd_value_t *seasonal_dev)
+{
+   rrd_value_t prediction = 0, seasonal_coef = DNAN;
+   rra_def_t *current_rra = &(rrd -> rra_def[rra_idx]);
+   /* obtain cdp_prep index for HWPREDICT */
+   unsigned long hw_rra_idx = current_rra -> par[RRA_dependent_rra_idx].u_cnt;
+   unsigned long hw_cdp_idx = hw_rra_idx * (rrd -> stat_head -> ds_cnt) + ds_idx;
+   unsigned long seasonal_cdp_idx;
+   unival *coefs = rrd -> cdp_prep[hw_cdp_idx].scratch;
+   rrd -> cdp_prep[cdp_idx].scratch[CDP_last_seasonal_deviation].u_val =
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_seasonal_deviation].u_val;
+   /* retrieve the next seasonal deviation value, could be NA */
+   rrd -> cdp_prep[cdp_idx].scratch[CDP_seasonal_deviation].u_val =
+         seasonal_dev[ds_idx];
+
+   /* retrieve the current seasonal_coef (not to be confused with the
+       * current seasonal deviation). Could make this more readable by introducing
+       * some wrapper functions. */
+   seasonal_cdp_idx = (rrd -> rra_def[hw_rra_idx].par[RRA_dependent_rra_idx].u_cnt)
+         *(rrd -> stat_head -> ds_cnt) + ds_idx;
+   if (rrd -> rra_def[hw_rra_idx].par[RRA_dependent_rra_idx].u_cnt < rra_idx)
+         /* SEASONAL array already updated */
+         seasonal_coef = rrd -> cdp_prep[seasonal_cdp_idx].scratch[CDP_hw_last_seasonal].u_val;
+   else
+         /* SEASONAL array not yet updated */
+         seasonal_coef = rrd -> cdp_prep[seasonal_cdp_idx].scratch[CDP_hw_seasonal].u_val;
+   
+   /* compute the abs value of the difference between the prediction and
+       * observed value */
+   if (hw_rra_idx < rra_idx)
+   {
+         /* associated HWPREDICT has already been updated */
+         if (isnan(coefs[CDP_hw_last_intercept].u_val) ||
+             isnan(coefs[CDP_hw_last_slope].u_val) ||
+             isnan(seasonal_coef))
+         {
+                /* one of the prediction values is uinitialized */
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val = DNAN;
+                return 0;
+         } else {
+            prediction = coefs[CDP_hw_last_intercept].u_val + 
+                  (coefs[CDP_hw_last_slope].u_val)*(coefs[CDP_last_null_count].u_cnt)
+                  + seasonal_coef;
+         }
+   } else {
+         /* associated HWPREDICT has NOT been updated */
+         if (isnan(coefs[CDP_hw_intercept].u_val) ||
+             isnan(coefs[CDP_hw_slope].u_val) ||
+             isnan(seasonal_coef))
+         {
+                /* one of the prediction values is uinitialized */
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val = DNAN;
+                return 0;
+         } else {
+            prediction = coefs[CDP_hw_intercept].u_val + 
+                  (coefs[CDP_hw_slope].u_val)*(coefs[CDP_null_count].u_cnt) 
+                  + seasonal_coef;
+         }
+   }
+
+   if (isnan(rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val))
+   {
+      /* no update, store existing value unchanged, doesn't
+          * matter if it is NA */
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val =
+                rrd -> cdp_prep[cdp_idx].scratch[CDP_last_seasonal_deviation].u_val;
+   } else if (isnan(rrd -> cdp_prep[cdp_idx].scratch[CDP_last_seasonal_deviation].u_val))
+   {
+         /* initialization */
+#ifdef DEBUG
+         fprintf(stderr,"Initialization of seasonal deviation\n");
+#endif
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val =
+            abs(prediction - rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val);
+   } else {
+         /* exponential smoothing update */
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val =
+           (rrd -> rra_def[rra_idx].par[RRA_seasonal_gamma].u_val)*
+           abs(prediction - rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val)
+           + (1 -  rrd -> rra_def[rra_idx].par[RRA_seasonal_gamma].u_val)*
+           (rrd -> cdp_prep[cdp_idx].scratch[CDP_last_seasonal_deviation].u_val);
+   }
+   return 0;
+}
+
+/* Check for a failure based on a threshold # of violations within the specified
+ * window. */
+int 
+update_failures(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx, 
+                               unsigned long ds_idx, unsigned short CDP_scratch_idx)
+{
+   /* detection of a violation depends on 3 RRAs:
+       * HWPREDICT, SEASONAL, and DEVSEASONAL */
+   rra_def_t *current_rra = &(rrd -> rra_def[rra_idx]);
+   unsigned long dev_rra_idx = current_rra -> par[RRA_dependent_rra_idx].u_cnt;
+   rra_def_t *dev_rra = &(rrd -> rra_def[dev_rra_idx]);
+   unsigned long hw_rra_idx = dev_rra -> par[RRA_dependent_rra_idx].u_cnt;
+   rra_def_t *hw_rra =  &(rrd -> rra_def[hw_rra_idx]);
+   unsigned long seasonal_rra_idx = hw_rra -> par[RRA_dependent_rra_idx].u_cnt;
+   unsigned long temp_cdp_idx;
+   rrd_value_t deviation = DNAN;
+   rrd_value_t seasonal_coef = DNAN;
+   rrd_value_t prediction = DNAN;
+   char violation = 0; 
+   unsigned short violation_cnt = 0, i;
+   char *violations_array;
+
+   /* usual checks to determine the order of the RRAs */
+   temp_cdp_idx = dev_rra_idx * (rrd -> stat_head -> ds_cnt) + ds_idx;
+   if (rra_idx < seasonal_rra_idx)
+   {
+         /* DEVSEASONAL not yet updated */
+         deviation = rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_seasonal_deviation].u_val;
+   } else {
+         /* DEVSEASONAL already updated */
+         deviation = rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_last_seasonal_deviation].u_val;
+   }
+   if (!isnan(deviation)) {
+
+   temp_cdp_idx = seasonal_rra_idx * (rrd -> stat_head -> ds_cnt) + ds_idx;
+   if (rra_idx < seasonal_rra_idx)
+   {
+         /* SEASONAL not yet updated */
+         seasonal_coef = rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_hw_seasonal].u_val;
+   } else {
+         /* SEASONAL already updated */
+         seasonal_coef = rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_hw_last_seasonal].u_val;
+   }
+   /* in this code block, we know seasonal coef is not DNAN, because deviation is not
+       * null */
+
+   temp_cdp_idx = hw_rra_idx * (rrd -> stat_head -> ds_cnt) + ds_idx;
+   if (rra_idx < hw_rra_idx)
+   {
+         /* HWPREDICT not yet updated */
+         prediction = rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_hw_intercept].u_val + 
+            (rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_hw_slope].u_val)
+                *(rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_null_count].u_cnt)
+                + seasonal_coef;
+   } else {
+         /* HWPREDICT already updated */
+         prediction = rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_hw_last_intercept].u_val + 
+            (rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_hw_last_slope].u_val)
+                *(rrd -> cdp_prep[temp_cdp_idx].scratch[CDP_last_null_count].u_cnt)
+                + seasonal_coef;
+   }
+
+   /* determine if the observed value is a violation */
+   if (!isnan(rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val))
+   {
+         if (rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val > prediction + 
+                (current_rra -> par[RRA_delta_pos].u_val)*deviation
+            || rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val < prediction - 
+                (current_rra -> par[RRA_delta_neg].u_val)*deviation)
+                violation = 1;
+   } else {
+         violation = 1; /* count DNAN values as violations */
+   }
+
+   }
+
+   /* determine if a failure has occurred and update the failure array */
+   violation_cnt = violation;
+   /* WARNING: this cast makes XML files non-portable across platforms,
+       * because an array of longs on disk is treated as an array of chars
+       * in memory. */
+   violations_array = (char *) ((void *) rrd -> cdp_prep[cdp_idx].scratch);
+   for (i = current_rra -> par[RRA_window_len].u_cnt; i > 1; i--)
+   {
+         /* shift */
+         violations_array[i-1] = violations_array[i-2]; 
+         violation_cnt += violations_array[i-1];
+   }
+   violations_array[0] = violation;
+
+   if (violation_cnt < current_rra -> par[RRA_failure_threshold].u_cnt)
+         /* not a failure */
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val = 0.0;
+   else
+         rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val = 1.0;
+
+   return (rrd-> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val);
+}
+
+/* For the specified CDP prep area and the FAILURES RRA,
+ * erase all history of past violations.
+ */
+void
+erase_violations(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx)
+{
+   unsigned short i;
+   char *violations_array;
+   /* check that rra_idx is a CF_FAILURES array */
+   if (cf_conv(rrd -> rra_def[rra_idx].cf_nam) != CF_FAILURES)
+   {
+#ifdef DEBUG
+         fprintf(stderr,"erase_violations called for non-FAILURES RRA: %s\n",
+            rrd -> rra_def[rra_idx].cf);
+#endif
+      return;
+   }
+
+#ifdef DEBUG
+   fprintf(stderr,"scratch buffer before erase:\n");
+   for (i = 0; i < MAX_CDP_PAR_EN; i++)
+   {
+         fprintf(stderr,"%lu ", rrd -> cdp_prep[cdp_idx].scratch[i].u_cnt);
+   }
+   fprintf(stderr,"\n");
+#endif
+
+   /* WARNING: this cast makes XML files non-portable across platforms,
+       * because an array of longs on disk is treated as an array of chars
+       * in memory. */
+   violations_array = (char *) ((void *) rrd -> cdp_prep[cdp_idx].scratch);
+   /* erase everything in the part of the CDP scratch array that will be
+       * used to store violations for the current window */
+   for (i = rrd -> rra_def[rra_idx].par[RRA_window_len].u_cnt; i > 0; i--)
+   {
+         violations_array[i-1] = 0;
+   }
+#ifdef DEBUG
+   fprintf(stderr,"scratch buffer after erase:\n");
+   for (i = 0; i < MAX_CDP_PAR_EN; i++)
+   {
+         fprintf(stderr,"%lu ", rrd -> cdp_prep[cdp_idx].scratch[i].u_cnt);
+   }
+   fprintf(stderr,"\n");
+#endif
+}
+
+/* Smooth a periodic array with a moving average: equal weights and
+ * length = 5% of the period. */
+int
+apply_smoother(rrd_t *rrd, unsigned long rra_idx, unsigned long rra_start,
+               FILE *rrd_file)
+{
+   unsigned long i, j, k;
+   unsigned long totalbytes;
+   rrd_value_t *rrd_values;
+   unsigned long row_length = rrd -> stat_head -> ds_cnt;
+   unsigned long row_count = rrd -> rra_def[rra_idx].row_cnt;
+   unsigned long offset;
+   FIFOqueue **buffers;
+   rrd_value_t *working_average;
+
+   offset = floor(0.025*row_count);
+   if (offset == 0) return 0; /* no smoothing */
+
+   /* allocate memory */
+   totalbytes = sizeof(rrd_value_t)*row_length*row_count;
+   rrd_values = (rrd_value_t *) malloc(totalbytes);
+   if (rrd_values == NULL)
+   {
+         rrd_set_error("apply smoother: memory allocation failure");
+         return -1;
+   }
+
+   /* rra_start is at the beginning of this rra */
+   if (fseek(rrd_file,rra_start,SEEK_SET))
+   {
+         rrd_set_error("seek to rra %d failed", rra_start);
+         free(rrd_values);
+         return -1;
+   }
+   fflush(rrd_file);
+   /* could read all data in a single block, but we need to
+    * check for NA values */
+   for (i = 0; i < row_count; ++i)
+   {
+         for (j = 0; j < row_length; ++j)
+         {
+                fread(&(rrd_values[i*row_length + j]),sizeof(rrd_value_t),1,rrd_file);
+                /* should check fread for errors... */
+                if (isnan(rrd_values[i*row_length + j])) {
+                       /* can't apply smoothing, still uninitialized values */
+#ifdef DEBUG
+                       fprintf(stderr,"apply_smoother: NA detected in seasonal array: %ld %ld\n",i,j);
+#endif
+                       free(rrd_values);
+                       return 0;
+                }
+         }
+   }
+
+   /* allocate queues, one for each data source */
+   buffers = (FIFOqueue **) malloc(sizeof(FIFOqueue *)*row_length);
+   for (i = 0; i < row_length; ++i)
+   {
+      queue_alloc(&(buffers[i]),2*offset + 1);
+   }
+   /* need working average initialized to 0 */
+   working_average = (rrd_value_t *) calloc(row_length,sizeof(rrd_value_t));
+
+   /* compute sums of the first 2*offset terms */ 
+   for (i = 0; i < 2*offset; ++i)
+   {
+         k = MyMod(i - offset,row_count);
+         for (j = 0; j < row_length; ++j)
+         {
+                queue_push(buffers[j],rrd_values[k*row_length + j]);
+                working_average[j] += rrd_values[k*row_length + j];
+         }
+   }
+
+   /* compute moving averages */
+   for (i = offset; i < row_count + offset; ++i)
+   {
+         for (j = 0; j < row_length; ++j)
+         {
+            k = MyMod(i,row_count);
+            /* add a term to the sum */
+            working_average[j] += rrd_values[k*row_length + j];
+            queue_push(buffers[j],rrd_values[k*row_length + j]);
+
+            /* reset k to be the center of the window */
+            k = MyMod(i - offset,row_count);
+            /* overwrite rdd_values entry, the old value is already
+             * saved in buffers */
+            rrd_values[k*row_length + j] = working_average[j]/(2*offset + 1);
+
+            /* remove a term from the sum */
+            working_average[j] -= queue_pop(buffers[j]);
+         }     
+   } 
+
+   for (i = 0; i < row_length; ++i)
+   {
+         queue_dealloc(buffers[i]);
+   }
+   free(buffers);
+   free(working_average);
+
+   /* flush updated values to disk */
+   fflush(rrd_file);
+   if (fseek(rrd_file,rra_start,SEEK_SET))
+   {
+         rrd_set_error("apply_smoother: seek to pos %d failed", rra_start);
+         free(rrd_values);
+         return -1;
+   }
+   /* write as a single block */
+   if (fwrite(rrd_values,sizeof(rrd_value_t),row_length*row_count,rrd_file)
+       != row_length*row_count)
+   {
+         rrd_set_error("apply_smoother: write failed to %lu",rra_start);
+         free(rrd_values);
+         return -1;
+   }
+
+   fflush(rrd_file);
+   free(rrd_values);
+   return 0;
+}
+
+int
+update_aberrant_CF(rrd_t *rrd, rrd_value_t pdp_val, enum cf_en current_cf, 
+                 unsigned long cdp_idx, unsigned long rra_idx, unsigned long ds_idx, 
+                 unsigned short CDP_scratch_idx, rrd_value_t *seasonal_coef)
+{
+   rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val = pdp_val;
+   switch (current_cf) {
+   case CF_AVERAGE:
+   default:
+        return 0;
+   case CF_HWPREDICT:
+     return update_hwpredict(rrd,cdp_idx,rra_idx,ds_idx,CDP_scratch_idx);
+   case CF_DEVPREDICT:
+        return update_devpredict(rrd,cdp_idx,rra_idx,ds_idx,CDP_scratch_idx);
+   case CF_SEASONAL:
+     return update_seasonal(rrd,cdp_idx,rra_idx,ds_idx,CDP_scratch_idx,seasonal_coef);
+   case CF_DEVSEASONAL:
+     return update_devseasonal(rrd,cdp_idx,rra_idx,ds_idx,CDP_scratch_idx,seasonal_coef);
+   case CF_FAILURES:
+     return update_failures(rrd,cdp_idx,rra_idx,ds_idx,CDP_scratch_idx);
+   }
+   return -1;
+}
+
+unsigned long MyMod(signed long val, unsigned long mod)
+{
+   unsigned long new_val;
+   if (val < 0)
+     new_val = ((unsigned long) abs(val)) % mod;
+   else
+     new_val = (val % mod);
+   
+   if (val < 0) 
+     return (mod - new_val);
+   else
+     return (new_val);
+}
+
+/* a standard fixed-capacity FIF0 queue implementation
+ * No overflow checking is performed. */
+int queue_alloc(FIFOqueue **q,int capacity)
+{
+   *q = (FIFOqueue *) malloc(sizeof(FIFOqueue));
+   if (*q == NULL) return -1;
+   (*q) -> queue = (rrd_value_t *) malloc(sizeof(rrd_value_t)*capacity);
+   if ((*q) -> queue == NULL)
+   {
+         free(*q);
+         return -1;
+   }
+   (*q) -> capacity = capacity;
+   (*q) -> head = capacity;
+   (*q) -> tail = 0;
+   return 0;
+}
+
+int queue_isempty(FIFOqueue *q)
+{
+   return (q -> head % q -> capacity == q -> tail);
+}
+
+void queue_push(FIFOqueue *q, rrd_value_t value)
+{
+   q -> queue[(q -> tail)++] = value;
+   q -> tail = q -> tail % q -> capacity;
+}
+
+rrd_value_t queue_pop(FIFOqueue *q)
+{
+   q -> head = q -> head % q -> capacity;
+   return q -> queue[(q -> head)++];
+}
+
+void queue_dealloc(FIFOqueue *q)
+{
+   free(q -> queue);
+   free(q);
+}
index 37c43db496f8e3a2b46acb6825877b5ae081f3e8..9f030e77b09b77e68156017c7a4a2b8d39d62cbf 100644 (file)
@@ -67,6 +67,7 @@ rrd_info(int argc, char **argv) {
     rrd_t        rrd;
     info_t       *data,*cd;
     infoval      info;
+       enum cf_en   current_cf;
 
     if(rrd_open(argv[1],&in_file,&rrd, RRD_READONLY)==-1){
        return(NULL);
@@ -113,6 +114,7 @@ rrd_info(int argc, char **argv) {
     for(i=0;i<rrd.stat_head->rra_cnt;i++){
        info.u_str=rrd.rra_def[i].cf_nam;
        cd=push(cd,sprintf_alloc("rra[%d].cf",         i),  RD_I_STR,   info);
+       current_cf = cf_conv(rrd.rra_def[i].cf_nam);
 
        info.u_cnt=rrd.rra_def[i].row_cnt;
        cd=push(cd,sprintf_alloc("rra[%d].rows",i),  RD_I_CNT,   info);
@@ -120,23 +122,69 @@ rrd_info(int argc, char **argv) {
        info.u_cnt=rrd.rra_def[i].pdp_cnt;
        cd=push(cd,sprintf_alloc("rra[%d].pdp_per_row",i),  RD_I_CNT,   info);
 
-        info.u_val=rrd.rra_def[i].par[RRA_cdp_xff_val].u_val;
-        cd=push(cd,sprintf_alloc("rra[%d].xff",i),  RD_I_VAL,   info);
-        
+       switch(current_cf)
+       {
+          case CF_HWPREDICT:
+                 info.u_val=rrd.rra_def[i].par[RRA_hw_alpha].u_val;
+                 cd=push(cd,sprintf_alloc("rra[%d].alpha",i),RD_I_VAL,info);
+                 info.u_val=rrd.rra_def[i].par[RRA_hw_beta].u_val;
+                 cd=push(cd,sprintf_alloc("rra[%d].beta",i),RD_I_VAL,info);
+                 break;
+          case CF_SEASONAL:
+          case CF_DEVSEASONAL:
+                 info.u_val=rrd.rra_def[i].par[RRA_seasonal_gamma].u_val;
+                 cd=push(cd,sprintf_alloc("rra[%d].gamma",i),RD_I_VAL,info);
+                 break;
+          case CF_FAILURES:
+                 info.u_val=rrd.rra_def[i].par[RRA_delta_pos].u_val;
+                 cd=push(cd,sprintf_alloc("rra[%d].delta_pos",i),RD_I_VAL,info);
+                 info.u_val=rrd.rra_def[i].par[RRA_delta_neg].u_val;
+                 cd=push(cd,sprintf_alloc("rra[%d].delta_neg",i),RD_I_VAL,info);
+                 info.u_cnt=rrd.rra_def[i].par[RRA_failure_threshold].u_cnt;
+                 cd=push(cd,sprintf_alloc("rra[%d].failure_threshold",i),RD_I_CNT,info);
+                 info.u_cnt=rrd.rra_def[i].par[RRA_window_len].u_cnt;
+                 cd=push(cd,sprintf_alloc("rra[%d].window_length",i),RD_I_CNT,info);
+                 break;
+          case CF_DEVPREDICT:
+                 break;
+          default:
+                 info.u_val=rrd.rra_def[i].par[RRA_cdp_xff_val].u_val;
+                 cd=push(cd,sprintf_alloc("rra[%d].xff",i),RD_I_VAL,info);
+                 break;
+       }
+
        for(ii=0;ii<rrd.stat_head->ds_cnt;ii++){
            info.u_val=rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_val].u_val;
            cd=push(cd,sprintf_alloc("rra[%d].cdp_prep[%d].value",i,ii), RD_I_VAL, info);
-
+        switch(current_cf)
+               {
+               case CF_HWPREDICT:
+           info.u_val=rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_hw_intercept].u_val;
+           cd=push(cd,sprintf_alloc("rra[%d].cdp_prep[%d].intercept",i,ii), RD_I_VAL, info);
+           info.u_val=rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_hw_slope].u_val;
+           cd=push(cd,sprintf_alloc("rra[%d].cdp_prep[%d].slope",i,ii), RD_I_VAL, info);
+           info.u_cnt=rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_null_count].u_cnt;
+           cd=push(cd,sprintf_alloc("rra[%d].cdp_prep[%d].NaN_count",i,ii), RD_I_CNT, info);
+                  break;
+               case CF_SEASONAL:
+           info.u_val=rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_hw_seasonal].u_val;
+           cd=push(cd,sprintf_alloc("rra[%d].cdp_prep[%d].seasonal",i,ii), RD_I_VAL, info);
+                  break;
+               case CF_DEVSEASONAL:
+           info.u_val=rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_seasonal_deviation].u_val;
+           cd=push(cd,sprintf_alloc("rra[%d].cdp_prep[%d].deviation",i,ii), RD_I_VAL, info);
+                  break;
+               case CF_DEVPREDICT:
+               case CF_FAILURES:
+                  break;
+               default:
            info.u_cnt=rrd.cdp_prep[i*rrd.stat_head->ds_cnt+ii].scratch[CDP_unkn_pdp_cnt].u_cnt;
            cd=push(cd,sprintf_alloc("rra[%d].cdp_prep[%d].unknown_datapoints",i,ii), RD_I_CNT, info);
+                  break;
         }
     }
-    rrd_free(&rrd);
+       }
+       rrd_free(&rrd);
     return(data);
 
 }
-
-
-
-
-
index 1a3496d444564d79ccfd6a9e4a82d88882a8db7b..7b546f5d1e0dff8bae10a279fee2750ea5d63322 100644 (file)
@@ -5,6 +5,13 @@
  *****************************************************************************
  * $Id$
  * $Log$
+ * Revision 1.3  2001/03/04 13:01:55  oetiker
+ * Aberrant Behavior Detection support. A brief overview added to rrdtool.pod.
+ * Major updates to rrd_update.c, rrd_create.c. Minor update to other core files.
+ * This is backwards compatible! But new files using the Aberrant stuff are not readable
+ * by old rrdtool versions. See http://cricket.sourceforge.net/aberrant/rrd_hw.htm
+ * -- Jake Brutlag <jakeb@corp.webtv.net>
+ *
  * Revision 1.2  2001/03/04 10:29:20  oetiker
  * fixed filedescriptor leak
  * -- Mike Franusich <mike@franusich.com>
@@ -45,6 +52,15 @@ rrd_open(char *file_name, FILE **in_file, rrd_t *rrd, int rdwr)
        rrd_set_error("opening '%s': %s",file_name, strerror(errno));
        return (-1);
     }
+/*
+       if (rdwr == RRD_READWRITE)
+       {
+          if (setvbuf((*in_file),NULL,_IONBF,2)) {
+                 rrd_set_error("failed to disable the stream buffer\n");
+                 return (-1);
+          }
+       }
+*/
     
 #define MYFREAD(MYVAR,MYVART,MYCNT) \
     if ((MYVAR = malloc(sizeof(MYVART) * MYCNT)) == NULL) {\
@@ -63,7 +79,7 @@ rrd_open(char *file_name, FILE **in_file, rrd_t *rrd, int rdwr)
            fclose(*in_file);
            return(-1);}
 
-       if (strncmp(rrd->stat_head->version,RRD_VERSION,5) != 0){
+        if (atoi(rrd->stat_head->version) > atoi(RRD_VERSION)){
            rrd_set_error("can't handle RRD file version %s",
                        rrd->stat_head->version);
            free(rrd->stat_head);
index 4e2b6c311a7d02a11f25b776a012064a6a8ece04..f1d1fad92cc647ce05c525384cf22581fdd97b1e 100644 (file)
@@ -99,7 +99,7 @@ int read_tag(char **buf, char *tag, char *format, void *value){
 int xml2rrd(char* buf, rrd_t* rrd, char rc){
   /* pass 1 identify number of RRAs  */
   char *ptr,*ptr2,*ptr3; /* walks thought the buffer */
-  long rows=0,mempool=0,i=0;
+  long rows=0,mempool=0,i=0,ii;
   xml_lc(buf); /* lets lowercase all active parts of the xml */
   ptr=buf;
   ptr2=buf;
@@ -115,6 +115,14 @@ int xml2rrd(char* buf, rrd_t* rrd, char rc){
 
   strcpy(rrd->stat_head->cookie,RRD_COOKIE);
   read_tag(&ptr,"version","%4[0-9]",rrd->stat_head->version);
+  /* added primitive version checking */
+  if (atoi(rrd -> stat_head -> version) < 2)
+  {
+    rrd_set_error("Incompatible file version, detected version %s, required version %s\n",
+                 rrd -> stat_head -> version, RRD_VERSION);
+    free(rrd -> stat_head);
+    return -1;
+  }
   rrd->stat_head->float_cookie = FLOAT_COOKIE;
   rrd->stat_head->ds_cnt = 0;
   rrd->stat_head->rra_cnt = 0;
@@ -187,21 +195,47 @@ int xml2rrd(char* buf, rrd_t* rrd, char rc){
       if(cf_conv(rrd->rra_def[rrd->stat_head->rra_cnt-1].cf_nam) == -1) return -1;
 
       read_tag(&ptr2,"pdp_per_row","%lu",&(rrd->rra_def[rrd->stat_head->rra_cnt-1].pdp_cnt));
-      read_tag(&ptr2,"xff","%lf",&(rrd->rra_def[rrd->stat_head->rra_cnt-1].par[RRA_cdp_xff_val].u_val));
-      if(rrd->rra_def[rrd->stat_head->rra_cnt-1].par[RRA_cdp_xff_val].u_val > 1 ||
-         rrd->rra_def[rrd->stat_head->rra_cnt-1].par[RRA_cdp_xff_val].u_val < 0)
-          return -1;
-      
+      /* add support to read RRA parameters */
+      eat_tag(&ptr2, "params");
+      for (i = 0; i < MAX_RRA_PAR_EN; i++)
+      {
+       if (i == RRA_dependent_rra_idx || 
+               i == RRA_seasonal_smooth_idx ||
+               i == RRA_failure_threshold)
+         read_tag(&ptr2, "value","%u",
+                  &(rrd->rra_def[rrd->stat_head->rra_cnt-1].par[i].u_cnt));
+       else
+         read_tag(&ptr2, "value","%lf",
+                  &(rrd->rra_def[rrd->stat_head->rra_cnt-1].par[i].u_val));
+      }
+      eat_tag(&ptr2, "/params");
       eat_tag(&ptr2,"cdp_prep");
-      for(i=0;i<rrd->stat_head->ds_cnt;i++){
-          eat_tag(&ptr2,"ds");
-          read_tag(&ptr2,"value","%lf",&(rrd->cdp_prep[rrd->stat_head->ds_cnt*
-                                                      (rrd->stat_head->rra_cnt-1)
-                                                      +i].scratch[CDP_val].u_val));
-          read_tag(&ptr2,"unknown_datapoints","%lu",&(rrd->cdp_prep[rrd->stat_head->ds_cnt
-                                                     *(rrd->stat_head->rra_cnt-1)
-                                                     +i].scratch[CDP_unkn_pdp_cnt].u_cnt));
-          eat_tag(&ptr2,"/ds");
+      for(i=0;i<rrd->stat_head->ds_cnt;i++)
+      {
+       eat_tag(&ptr2,"ds");
+       /* add suport to read CDP parameters */
+       for (ii = 0; ii < MAX_CDP_PAR_EN; ii++)
+       {
+      /* handle integer values as a special case */
+      if (cf_conv(rrd->rra_def[rrd->stat_head->rra_cnt-1].cf_nam) == CF_FAILURES ||
+                 ii == CDP_unkn_pdp_cnt || 
+                 ii == CDP_null_count ||
+             ii == CDP_last_null_count)
+         {
+           read_tag(&ptr2,"value","%lu",
+                    &(rrd->cdp_prep[rrd->stat_head->ds_cnt*(rrd->stat_head->rra_cnt-1)
+                    +i].scratch[ii].u_cnt));
+         } else {
+           read_tag(&ptr2,"value","%lf",&(rrd->cdp_prep[rrd->stat_head->ds_cnt*
+                    (rrd->stat_head->rra_cnt-1) +i].scratch[ii].u_val));
+         }
+
+#if 0
+         read_tag(&ptr2,"unknown_datapoints","%lu",&(rrd->cdp_prep[rrd->stat_head->ds_cnt
+                  *(rrd->stat_head->rra_cnt-1) +i].scratch[CDP_unkn_pdp_cnt].u_cnt));
+#endif
+       } /* end for */
+       eat_tag(&ptr2,"/ds");
       }
       eat_tag(&ptr2,"/cdp_prep");
       rrd->rra_def[rrd->stat_head->rra_cnt-1].row_cnt=0;
@@ -257,6 +291,9 @@ int xml2rrd(char* buf, rrd_t* rrd, char rc){
   }
 
   for(i=0; i <rrd->stat_head->rra_cnt; i++) {
+         /* last row in the xml file is the most recent; as
+          * rrd_update increments the current row pointer, set cur_row
+          * here to the last row. */
       rrd->rra_ptr[i].cur_row = rrd->rra_def[i].row_cnt-1;
   }
   if (ptr==NULL)
@@ -379,3 +416,20 @@ rrd_restore(int argc, char **argv)
     rrd_free(&rrd);    
     return 0;
 }
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
index 789f072cd46bb38da31df48648776850498f629c..279566512bede984765e72d1804936029c0f9ada 100644 (file)
@@ -5,8 +5,15 @@
  *****************************************************************************
  * $Id$
  * $Log$
- * Revision 1.1  2001/02/25 22:25:06  oetiker
- * Initial revision
+ * Revision 1.2  2001/03/04 13:01:55  oetiker
+ * Aberrant Behavior Detection support. A brief overview added to rrdtool.pod.
+ * Major updates to rrd_update.c, rrd_create.c. Minor update to other core files.
+ * This is backwards compatible! But new files using the Aberrant stuff are not readable
+ * by old rrdtool versions. See http://cricket.sourceforge.net/aberrant/rrd_hw.htm
+ * -- Jake Brutlag <jakeb@corp.webtv.net>
+ *
+ * Revision 1.1.1.1  2001/02/25 22:25:06  oetiker
+ * checkin
  *
  *****************************************************************************/
 #ifdef  __cplusplus
@@ -153,11 +160,36 @@ enum dst_en dst_conv(char *string);
 long ds_match(rrd_t *rrd,char *ds_nam);
 double rrd_diff(char *a, char *b);
 
+/* functions added for aberrant behavior detection.
+ * implemented for the most part in rrd_hw.c */
+int update_aberrant_CF(rrd_t *rrd, rrd_value_t pdp_val, enum cf_en current_cf,
+   unsigned long cdp_idx, unsigned long rra_idx, unsigned long ds_idx,
+   unsigned short CDP_scratch_idx, rrd_value_t *seasonal_coef);
+int create_hw_contingent_rras(rrd_t *rrd, unsigned short period, 
+   unsigned long hashed_name);
+int lookup_seasonal(rrd_t *rrd, unsigned long rra_idx, unsigned long rra_start,
+   FILE *rrd_file, unsigned long offset, rrd_value_t **seasonal_coef);
+void erase_violations(rrd_t *rrd, unsigned long cdp_idx, unsigned long rra_idx);
+int apply_smoother(rrd_t *rrd, unsigned long rra_idx, unsigned long rra_start,
+   FILE *rrd_file);
+
+/* a standard fixed-capacity FIFO queue implementation */
+typedef struct FIFOqueue {
+   rrd_value_t *queue;
+   int capacity, head, tail;
+} FIFOqueue;
+
+int queue_alloc(FIFOqueue **q,int capacity);
+void queue_dealloc(FIFOqueue *q);
+void queue_push(FIFOqueue *q, rrd_value_t value);
+int queue_isempty(FIFOqueue *q);
+rrd_value_t queue_pop(FIFOqueue *q);
+
+#define BURNIN_CYCLES 3
+
 #endif
 
 
 #ifdef  __cplusplus
 }
 #endif
-
-
index d04a725daa04d4f955a6c3d060fe945b5f7a315e..b45e5db2465c849937bd902515e88711d507f886 100644 (file)
@@ -5,13 +5,24 @@
  *****************************************************************************
  * $Id$
  * $Log$
- * Revision 1.1  2001/02/25 22:25:06  oetiker
- * Initial revision
+ * Revision 1.2  2001/03/04 13:01:55  oetiker
+ * Aberrant Behavior Detection support. A brief overview added to rrdtool.pod.
+ * Major updates to rrd_update.c, rrd_create.c. Minor update to other core files.
+ * This is backwards compatible! But new files using the Aberrant stuff are not readable
+ * by old rrdtool versions. See http://cricket.sourceforge.net/aberrant/rrd_hw.htm
+ * -- Jake Brutlag <jakeb@corp.webtv.net>
+ *
+ * Revision 1.1.1.1  2001/02/25 22:25:06  oetiker
+ * checkin
  *
  *****************************************************************************/
 
 #include "rrd_tool.h"
 
+int set_hwarg(rrd_t *rrd,enum cf_en cf,enum rra_par_en rra_par,char *arg);
+int set_deltaarg(rrd_t *rrd,enum rra_par_en rra_par,char *arg);
+int set_windowarg(rrd_t *rrd,enum rra_par_en,char *arg);
+
 int
 rrd_tune(int argc, char **argv)    
 {   
@@ -41,11 +52,19 @@ rrd_tune(int argc, char **argv)
            {"maximum",          required_argument, 0, 'a'},
            {"data-source-type", required_argument, 0, 'd'},
            {"data-source-rename", required_argument, 0, 'r'},
+           /* added parameter tuning options for aberrant behavior detection */
+               {"deltapos",required_argument,0,'p'},
+               {"deltaneg",required_argument,0,'n'},
+               {"window-length",required_argument,0,'w'},
+               {"failure-threshold",required_argument,0,'f'},
+           {"alpha",required_argument,0,'x'},
+           {"beta",required_argument,0,'y'},
+           {"gamma",required_argument,0,'z'},
            {0,0,0,0}
        };
        int option_index = 0;
        int opt;
-       opt = getopt_long(argc, argv, "h:i:a:d:r:", 
+       opt = getopt_long(argc, argv, "h:i:a:d:r:p:n:w:f:x:y:z:", 
                          long_options, &option_index);
        if (opt == EOF)
            break;
@@ -145,6 +164,48 @@ rrd_tune(int argc, char **argv)
            strncpy(rrd.ds_def[ds].ds_nam,ds_new,DS_NAM_SIZE-1);
            rrd.ds_def[ds].ds_nam[DS_NAM_SIZE-1]='\0';
            break;
+    case 'p':
+               if (set_deltaarg(&rrd,RRA_delta_pos,optarg)) {
+                  rrd_free(&rrd);
+                  return -1;
+               }
+               break;
+       case 'n':
+               if (set_deltaarg(&rrd,RRA_delta_neg,optarg)) {
+                  rrd_free(&rrd);
+                  return -1;
+               }
+               break;
+       case 'f':
+               if (set_windowarg(&rrd,RRA_failure_threshold,optarg)) {
+                  rrd_free(&rrd);
+                  return -1;
+               }
+               break;
+       case 'w':
+               if (set_windowarg(&rrd,RRA_window_len,optarg)) {
+                  rrd_free(&rrd);
+                  return -1;
+               }
+               break;
+       case 'x':
+               if (set_hwarg(&rrd,CF_HWPREDICT,RRA_hw_alpha,optarg)) {
+                  rrd_free(&rrd);
+                  return -1;
+               }
+               break;
+       case 'y':
+               if (set_hwarg(&rrd,CF_HWPREDICT,RRA_hw_beta,optarg)) {
+                  rrd_free(&rrd);
+                  return -1;
+               }
+               break;
+       case 'z':
+               if (set_hwarg(&rrd,CF_SEASONAL,RRA_seasonal_gamma,optarg)) {
+                  rrd_free(&rrd);
+                  return -1;
+               }
+               break;
        case '?':
             if (optopt != 0)
                 rrd_set_error("unknown option '%c'", optopt);
@@ -162,6 +223,9 @@ rrd_tune(int argc, char **argv)
               sizeof(stat_head_t),1, rrd_file);
        fwrite(rrd.ds_def,
               sizeof(ds_def_t), rrd.stat_head->ds_cnt, rrd_file);
+       /* need to write rra_defs for RRA parameter changes */
+       fwrite(rrd.rra_def, sizeof(rra_def_t), rrd.stat_head->rra_cnt,
+                  rrd_file);
     } else {
        int i;
        for(i=0;i< rrd.stat_head->ds_cnt;i++)
@@ -177,3 +241,114 @@ rrd_tune(int argc, char **argv)
     return 0;
 }
 
+int set_hwarg(rrd_t *rrd,enum cf_en cf,enum rra_par_en rra_par,char *arg)
+{
+   double param;
+   unsigned long i;
+   signed short rra_idx = -1, devseasonal_idx = -1;
+   /* read the value */
+   param = atof(arg);
+   if (param <= 0.0 || param >= 1.0)
+   {
+         rrd_set_error("Holt-Winters parameter must be between 0 and 1");
+         return -1;
+   }
+   /* does the appropriate RRA exist?  */
+   for (i =  0; i < rrd -> stat_head -> rra_cnt; ++i)
+   {
+         if (cf_conv(rrd -> rra_def[i].cf_nam) == cf)
+         {
+                rra_idx = i;
+         }
+         /* find the DEVSEASONAL index separately, as it is optional */
+         if (cf_conv(rrd -> rra_def[i].cf_nam) == CF_DEVSEASONAL)
+         {
+                devseasonal_idx = i;
+         }
+   }
+   if (rra_idx == -1) 
+   {
+         rrd_set_error("Holt-Winters RRA does not exist in this RRD");
+         return -1;
+   }
+   
+   /* set the value */
+   rrd -> rra_def[rra_idx].par[rra_par].u_val = param;
+   if (devseasonal_idx > -1)
+         rrd -> rra_def[devseasonal_idx].par[rra_par].u_val = param;
+   return 0;
+}
+
+int set_deltaarg(rrd_t *rrd,enum rra_par_en rra_par,char *arg)
+{
+   rrd_value_t param;
+   unsigned long i;
+   signed short rra_idx = -1;
+
+   param = atof(arg);
+   if (param < 0.1)
+   {
+         rrd_set_error("Parameter specified is too small");
+         return -1;
+   }
+   /* does the appropriate RRA exist?  */
+   for (i = 0; i < rrd -> stat_head -> rra_cnt; ++i)
+   {
+         if (cf_conv(rrd -> rra_def[i].cf_nam) == CF_FAILURES) 
+         {
+                rra_idx = i;
+                break;
+         }
+   }
+   if (rra_idx == -1) 
+   {
+         rrd_set_error("Failures RRA does not exist in this RRD");
+         return -1;
+   }
+
+   /* set the value */
+   rrd -> rra_def[rra_idx].par[rra_par].u_val = param;
+   return 0;
+}
+
+int set_windowarg(rrd_t *rrd,enum rra_par_en rra_par,char *arg)
+{
+   unsigned long param;
+   unsigned long i, cdp_idx;
+   signed short rra_idx = -1;
+   /* read the value */
+   param = atoi(arg);
+   /* there are 4 chars to a long, reserve on CDP entry for future
+       * use. */
+   if (param < 1 || param > MAX_FAILURES_WINDOW_LEN)
+   {
+         rrd_set_error("Parameter must be between %d and %d",
+                1, MAX_CDP_PAR_EN - 1);
+         return -1;
+   }
+   /* does the appropriate RRA exist?  */
+   for (i = 0; i < rrd -> stat_head -> rra_cnt; ++i)
+   {
+         if (cf_conv(rrd -> rra_def[i].cf_nam) == CF_FAILURES) 
+         {
+                rra_idx = i;
+                break;
+         }
+   }
+   if (rra_idx == -1) 
+   {
+         rrd_set_error("Failures RRA does not exist in this RRD");
+         return -1;
+   }
+   
+   /* set the value */
+   rrd -> rra_def[rra_idx].par[rra_par].u_cnt = param;
+
+   /* erase existing violations */
+   for (i = 0; i < rrd -> stat_head -> ds_cnt; i++)
+   {
+         cdp_idx = rra_idx * (rrd -> stat_head -> ds_cnt) + i;
+         erase_violations(rrd,cdp_idx,rra_idx);
+   }
+   return 0;
+}
index 1e5a83e07c694c283ddd552ab3b53bd9e247e44b..08ecb493fd28936614c2952a07fef9fd5484d7ad 100644 (file)
@@ -5,6 +5,13 @@
  *****************************************************************************
  * $Id$
  * $Log$
+ * Revision 1.3  2001/03/04 13:01:55  oetiker
+ * Aberrant Behavior Detection support. A brief overview added to rrdtool.pod.
+ * Major updates to rrd_update.c, rrd_create.c. Minor update to other core files.
+ * This is backwards compatible! But new files using the Aberrant stuff are not readable
+ * by old rrdtool versions. See http://cricket.sourceforge.net/aberrant/rrd_hw.htm
+ * -- Jake Brutlag <jakeb@corp.webtv.net>
+ *
  * Revision 1.2  2001/03/04 11:14:25  oetiker
  * added at-style-time@value:value syntax to rrd_update
  * --  Dave Bodenstab <imdave@mcs.net>
  #include <io.h>
 #endif
 
-
 /* Prototypes */
 int LockRRD(FILE *rrd_file);
-
+void write_RRA_row (rrd_t *rrd, unsigned long rra_idx, unsigned long *rra_current,
+    unsigned short CDP_scratch_idx, FILE *rrd_file);
 /*#define DEBUG */
 
+#define IFDNAN(X,Y) (isnan(X) ? (Y) : (X));
+
 
 #ifdef STANDALONE
 int 
@@ -56,7 +66,8 @@ rrd_update(int argc, char **argv)
 {
 
     int              arg_i = 2;
-    long             i,ii,iii;
+    short            j;
+    long             i,ii,iii=1;
 
     unsigned long    rra_begin;          /* byte pointer to the rra
                                          * area in the rrd file.  this
@@ -81,8 +92,6 @@ rrd_update(int argc, char **argv)
                                          * was last updated */
     unsigned long    occu_pdp_age;       /* how long ago was the last
                                          * pdp_step time */
-    unsigned long    pdp_st;             /* helper for cdp_prep 
-                                         * processing */
     rrd_value_t      *pdp_new;           /* prepare the incoming data
                                          * to be added the the
                                          * existing entry */
@@ -98,9 +107,23 @@ rrd_update(int argc, char **argv)
     rrd_t            rrd;
     time_t           current_time = time(NULL);
     char             **updvals;
-    int              wrote_to_file = 0;
-    char             *template = NULL;          
-
+    int              schedule_smooth = 0;
+    char             *template = NULL;   
+       rrd_value_t      *seasonal_coef = NULL, *last_seasonal_coef = NULL;
+                                        /* a vector of future Holt-Winters seasonal coefs */
+    unsigned long    elapsed_pdp_st;
+                                        /* number of elapsed PDP steps since last update */
+    unsigned long    *rra_step_cnt = NULL;
+                                        /* number of rows to be updated in an RRA for a data
+                                         * value. */
+    unsigned long    start_pdp_offset;
+                                        /* number of PDP steps since the last update that
+                                         * are assigned to the first CDP to be generated
+                                         * since the last update. */
+    unsigned short   scratch_idx;
+                                        /* index into the CDP scratch array */
+    enum cf_en       current_cf;
+                                        /* numeric id of the current consolidation function */
 
     while (1) {
        static struct option long_options[] =
@@ -123,7 +146,7 @@ rrd_update(int argc, char **argv)
 
        case '?':
            rrd_set_error("unknown option '%s'",argv[optind-1]);
-            rrd_free(&rrd);
+            rrd_free(&rrd);            
            return(-1);
        }
     }
@@ -158,7 +181,7 @@ rrd_update(int argc, char **argv)
       rrd_free(&rrd);
       fclose(rrd_file);
       return(-1);   
-    }
+    } 
 
     if((updvals = malloc( sizeof(char*) * (rrd.stat_head->ds_cnt+1)))==NULL){
        rrd_set_error("allocating updvals pointer array");
@@ -248,7 +271,7 @@ rrd_update(int argc, char **argv)
        enum {atstyle, normal} timesyntax;
        struct time_value ds_tv;
         if (stepper == NULL){
-                rrd_set_error("faild duplication argv entry");
+                rrd_set_error("failed duplication argv entry");
                 free(updvals);
                 free(pdp_temp);  
                 free(tmpl_idx);
@@ -329,7 +352,7 @@ rrd_update(int argc, char **argv)
        }
        
        
-       /* seek to the beginning of the rrd's */
+       /* seek to the beginning of the rra's */
        if (rra_current != rra_begin) {
            if(fseek(rrd_file, rra_begin, SEEK_SET) != 0) {
                rrd_set_error("seek error in rrd");
@@ -532,183 +555,439 @@ rrd_update(int argc, char **argv)
 #endif
            }
 
-
-           /* now we have to integrate this data into the cdp_prep areas */
-           /* going through the round robin archives */
-           for(i = 0;
-               i < rrd.stat_head->rra_cnt;
-               i++){
-               enum cf_en current_cf = cf_conv(rrd.rra_def[i].cf_nam);
-               /* going through all pdp_st moments which have occurred 
-                * since the last run */
-               for(pdp_st  = proc_pdp_st+rrd.stat_head->pdp_step; 
-                   pdp_st <= occu_pdp_st; 
-                   pdp_st += rrd.stat_head->pdp_step){
-
+               /* compute the number of elapsed pdp_st moments */
+               elapsed_pdp_st = (occu_pdp_st - proc_pdp_st) / rrd.stat_head -> pdp_step;
 #ifdef DEBUG
-                   fprintf(stderr,"RRA %lu STEP %lu\n",i,pdp_st);
+               fprintf(stderr,"elapsed PDP steps: %lu\n", elapsed_pdp_st);
 #endif
+               if (rra_step_cnt == NULL)
+               {
+                  rra_step_cnt = (unsigned long *) 
+                         malloc((rrd.stat_head->rra_cnt)* sizeof(unsigned long));
+               }
 
-                   if((pdp_st %
-                       (rrd.rra_def[i].pdp_cnt*rrd.stat_head->pdp_step)) == 0){
+           for(i = 0, rra_start = rra_begin;
+               i < rrd.stat_head->rra_cnt;
+           rra_start += rrd.rra_def[i].row_cnt * rrd.stat_head -> ds_cnt * sizeof(rrd_value_t),
+               i++)
+               {
+               current_cf = cf_conv(rrd.rra_def[i].cf_nam);
+               start_pdp_offset = rrd.rra_def[i].pdp_cnt -
+                  (proc_pdp_st / rrd.stat_head -> pdp_step) % rrd.rra_def[i].pdp_cnt;
+        if (start_pdp_offset <= elapsed_pdp_st) {
+           rra_step_cnt[i] = (elapsed_pdp_st - start_pdp_offset) / 
+                     rrd.rra_def[i].pdp_cnt + 1;
+           } else {
+                  rra_step_cnt[i] = 0;
+               }
 
-                       /* later on the cdp_prep values will be transferred to
-                        * the rra.  we want to be in the right place. */
-                       rrd.rra_ptr[i].cur_row++;
-                       if (rrd.rra_ptr[i].cur_row >= rrd.rra_def[i].row_cnt)
-                           /* oops ... we have to wrap the beast ... */
-                           rrd.rra_ptr[i].cur_row=0;                   
+               if (current_cf == CF_SEASONAL || current_cf == CF_DEVSEASONAL) 
+               {
+                  /* If this is a bulk update, we need to skip ahead in the seasonal
+                       * arrays so that they will be correct for the next observed value;
+                       * note that for the bulk update itself, no update will occur to
+                       * DEVSEASONAL or SEASONAL; futhermore, HWPREDICT and DEVPREDICT will
+                       * be set to DNAN. */
+           if (rra_step_cnt[i] > 2) 
+                  {
+                         /* skip update by resetting rra_step_cnt[i],
+                          * note that this is not data source specific; this is due
+                          * to the bulk update, not a DNAN value for the specific data
+                          * source. */
+                         rra_step_cnt[i] = 0;
+              lookup_seasonal(&rrd,i,rra_start,rrd_file,elapsed_pdp_st, 
+                            &last_seasonal_coef);
+                     lookup_seasonal(&rrd,i,rra_start,rrd_file,elapsed_pdp_st + 1,
+                            &seasonal_coef);
+                  }
+               
+                 /* periodically run a smoother for seasonal effects */
+                 /* Need to use first cdp parameter buffer to track
+                  * burnin (burnin requires a specific smoothing schedule).
+                  * The CDP_init_seasonal parameter is really an RRA level,
+                  * not a data source within RRA level parameter, but the rra_def
+                  * is read only for rrd_update (not flushed to disk). */
+                 iii = i*(rrd.stat_head -> ds_cnt);
+                 if (rrd.cdp_prep[iii].scratch[CDP_init_seasonal].u_cnt 
+                         <= BURNIN_CYCLES)
+                 {
+                    if (rrd.rra_ptr[i].cur_row + elapsed_pdp_st 
+                                > rrd.rra_def[i].row_cnt - 1) {
+                          /* mark off one of the burnin cycles */
+                          ++(rrd.cdp_prep[iii].scratch[CDP_init_seasonal].u_cnt);
+                      schedule_smooth = 1;
+                        }  
+                 } else {
+                        /* someone has no doubt invented a trick to deal with this
+                         * wrap around, but at least this code is clear. */
+                        if (rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt >
+                            rrd.rra_ptr[i].cur_row)
+                        {
+                                /* here elapsed_pdp_st = rra_step_cnt[i] because of 1-1
+                                 * mapping between PDP and CDP */
+                                if (rrd.rra_ptr[i].cur_row + elapsed_pdp_st
+                                       >= rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt)
+                                {
 #ifdef DEBUG
-                       fprintf(stderr,"  -- RRA Preseek %ld\n",ftell(rrd_file));
+                                       fprintf(stderr,
+                                       "schedule_smooth 1: cur_row %lu, elapsed_pdp_st %lu, smooth idx %lu\n",
+                    rrd.rra_ptr[i].cur_row, elapsed_pdp_st, 
+                                       rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt);
 #endif
-                       /* determine if a seek is even needed. */
-                       rra_pos_tmp = rra_start +
-                               rrd.stat_head->ds_cnt*rrd.rra_ptr[i].cur_row*sizeof(rrd_value_t);
-                       if(rra_pos_tmp != rra_current) {
-                           if(fseek(rrd_file, rra_pos_tmp, SEEK_SET) != 0){
-                               rrd_set_error("seek error in rrd");
-                               break;
-                           }
-                           rra_current = rra_pos_tmp;
-                       }
+                                       schedule_smooth = 1;
+                                }
+             } else {
+                                /* can't rely on negative numbers because we are working with
+                                 * unsigned values */
+                                /* Don't need modulus here. If we've wrapped more than once, only
+                                 * one smooth is executed at the end. */
+                                if (rrd.rra_ptr[i].cur_row + elapsed_pdp_st >= rrd.rra_def[i].row_cnt
+                                       && rrd.rra_ptr[i].cur_row + elapsed_pdp_st - rrd.rra_def[i].row_cnt
+                                       >= rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt)
+                                {
 #ifdef DEBUG
-                       fprintf(stderr,"  -- RRA Postseek %ld\n",ftell(rrd_file));
+                                       fprintf(stderr,
+                                       "schedule_smooth 2: cur_row %lu, elapsed_pdp_st %lu, smooth idx %lu\n",
+                    rrd.rra_ptr[i].cur_row, elapsed_pdp_st, 
+                                       rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt);
 #endif
-                   }
+                                       schedule_smooth = 1;
+                                }
+                        }
+                 }
+
+             rra_current = ftell(rrd_file); 
+               } /* if cf is DEVSEASONAL or SEASONAL */
 
+        if (rrd_test_error()) break;
+
+                   /* update CDP_PREP areas */
+                   /* loop over data soures within each RRA */
                    for(ii = 0;
                        ii < rrd.stat_head->ds_cnt;
-                       ii++){
+                       ii++)
+                       {
+                       
+                       /* iii indexes the CDP prep area for this data source within the RRA */
                        iii=i*rrd.stat_head->ds_cnt+ii;
-                   
-                       /* the contents of cdp_prep[].scratch[CDP_val].u_val depends
-                        * on the consolidation function ! */
-                   
-                       if (isnan(pdp_temp[ii])){    /* pdp is unknown */
-                           rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt++;
+
+                       if (rrd.rra_def[i].pdp_cnt > 1) {
+                         
+                          if (rra_step_cnt[i] > 0) {
+                          /* If we are in this block, as least 1 CDP value will be written to
+                               * disk, this is the CDP_primary_val entry. If more than 1 value needs
+                               * to be written, then the "fill in" value is the CDP_secondary_val
+                               * entry. */
+                                 if (isnan(pdp_temp[ii]))
+                  {
+                                        rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt += start_pdp_offset;
+                                        rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val = DNAN;
+                                 } else {
+                                        /* CDP_secondary value is the RRA "fill in" value for intermediary
+                                         * CDP data entries. No matter the CF, the value is the same because
+                                         * the average, max, min, and last of a list of identical values is
+                                         * the same, namely, the value itself. */
+                                        rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val = pdp_temp[ii];
+                                 }
+                     
+                                 if (rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt
+                                     > rrd.rra_def[i].pdp_cnt*
+                                     rrd.rra_def[i].par[RRA_cdp_xff_val].u_val)
+                                 {
+                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = DNAN;
+                                        /* initialize carry over */
+                                        if (current_cf == CF_AVERAGE) {
+                                                  if (isnan(pdp_temp[ii])) { 
+                                                         rrd.cdp_prep[iii].scratch[CDP_val].u_val = DNAN;
+                                                  } else {
+                                                         rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii] *
+                                                                ((elapsed_pdp_st - start_pdp_offset) % rrd.rra_def[i].pdp_cnt);
+                                                  }
+                                        } else {
+                                               rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                        }
+                                 } else {
+                                        rrd_value_t cum_val, cur_val; 
+                                    switch (current_cf) {
+                                               case CF_AVERAGE:
+                                                 cum_val = IFDNAN(rrd.cdp_prep[iii].scratch[CDP_val].u_val, 0.0);
+                                                 cur_val = IFDNAN(pdp_temp[ii],0.0);
+                          rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val =
+                                              (cum_val + cur_val) /
+                                          (rrd.rra_def[i].pdp_cnt
+                                              -rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt);
+                                                  /* initialize carry over value */
+                                                  if (isnan(pdp_temp[ii])) { 
+                                                         rrd.cdp_prep[iii].scratch[CDP_val].u_val = DNAN;
+                                                  } else {
+                                                         rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii] *
+                                                                ((elapsed_pdp_st - start_pdp_offset) % rrd.rra_def[i].pdp_cnt);
+                                                  }
+                                                  break;
+                                               case CF_MAXIMUM:
+                                                 cum_val = IFDNAN(rrd.cdp_prep[iii].scratch[CDP_val].u_val, -DINF);
+                                                 cur_val = IFDNAN(pdp_temp[ii],-DINF);
 #ifdef DEBUG
-                           fprintf(stderr,"  ** UNKNOWN ADD %lu\n",
-                                   rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt);
+                                                 if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val) &&
+                                                         isnan(pdp_temp[ii])) {
+                                                    fprintf(stderr,
+                                                               "RRA %lu, DS %lu, both CDP_val and pdp_temp are DNAN!",
+                                                               i,ii);
+                                                        exit(-1);
+                                                 }
 #endif
-                       } else {
-                           if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val)){
-                               /* cdp_prep is unknown when it does not
-                                * yet contain data. It can not be zero for
-                                * things like mim and max consolidation
-                                * functions */
+                                                 if (cur_val > cum_val)
+                                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = cur_val;
+                                                 else
+                                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = cum_val;
+                                                 /* initialize carry over value */
+                                                 rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                                 break;
+                                               case CF_MINIMUM:
+                                                 cum_val = IFDNAN(rrd.cdp_prep[iii].scratch[CDP_val].u_val, DINF);
+                                                 cur_val = IFDNAN(pdp_temp[ii],DINF);
 #ifdef DEBUG
-                               fprintf(stderr,"  ** INIT CDP %e\n", pdp_temp[ii]);
+                                                 if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val) &&
+                                                         isnan(pdp_temp[ii])) {
+                                                    fprintf(stderr,
+                                                               "RRA %lu, DS %lu, both CDP_val and pdp_temp are DNAN!",
+                                                               i,ii);
+                                                        exit(-1);
+                                                 }
 #endif
-                               rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
-                           }
-                           else {
-                               switch (current_cf){
-                                   case CF_AVERAGE:                            
-                                       rrd.cdp_prep[iii].scratch[CDP_val].u_val+=pdp_temp[ii];
+                                                 if (cur_val < cum_val)
+                                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = cur_val;
+                                                 else
+                                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = cum_val;
+                                                 /* initialize carry over value */
+                                                 rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                                 break;
+                                               case CF_LAST:
+                                               default:
+                                                  rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = pdp_temp[ii];
+                                                  /* initialize carry over value */
+                                                  rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                               break;
+                                        }
+                                 } /* endif meets xff value requirement for a valid value */
+                                 /* initialize carry over CDP_unkn_pdp_cnt, this must after CDP_primary_val
+                                  * is set because CDP_unkn_pdp_cnt is required to compute that value. */
+                                 if (isnan(pdp_temp[ii]))
+                                        rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt = 
+                                               (elapsed_pdp_st - start_pdp_offset) % rrd.rra_def[i].pdp_cnt;
+                                 else
+                                        rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt = 0;
+               } else  /* rra_step_cnt[i]  == 0 */
+                          {
 #ifdef DEBUG
-                                       fprintf(stderr,"  ** AVERAGE %e\n", 
-                                               rrd.cdp_prep[iii].scratch[CDP_val].u_val);
+                                 if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val)) {
+                                 fprintf(stderr,"schedule CDP_val update, RRA %lu DS %lu, DNAN\n",
+                                        i,ii);
+                                 } else {
+                                 fprintf(stderr,"schedule CDP_val update, RRA %lu DS %lu, %10.2f\n",
+                                        i,ii,rrd.cdp_prep[iii].scratch[CDP_val].u_val);
+                                 }
 #endif
-                                       break;    
-                                   case CF_MINIMUM:
-                                       if (pdp_temp[ii] < rrd.cdp_prep[iii].scratch[CDP_val].u_val)
+                                 if (isnan(pdp_temp[ii])) {
+                                rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt += elapsed_pdp_st;
+                                 } else if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val))
+                                 {
+                                        if (current_cf == CF_AVERAGE) {
+                                           rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii] *
+                                                  elapsed_pdp_st;
+                                        } else {
                                            rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                        }
 #ifdef DEBUG
-                                       fprintf(stderr,"  ** MINIMUM %e\n", 
-                                               rrd.cdp_prep[iii].scratch[CDP_val].u_val);
+                                        fprintf(stderr,"Initialize CDP_val for RRA %lu DS %lu: %10.2f\n",
+                                           i,ii,rrd.cdp_prep[iii].scratch[CDP_val].u_val);
 #endif
-                                       break;
-                                   case CF_MAXIMUM:
-                                       if (pdp_temp[ii] > rrd.cdp_prep[iii].scratch[CDP_val].u_val)
-                                           rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                 } else {
+                                        switch (current_cf) {
+                                        case CF_AVERAGE:
+                                           rrd.cdp_prep[iii].scratch[CDP_val].u_val += pdp_temp[ii] *
+                                                  elapsed_pdp_st;
+                                               break;
+                                        case CF_MINIMUM:
+                                               if (pdp_temp[ii] < rrd.cdp_prep[iii].scratch[CDP_val].u_val)
+                                                  rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                               break; 
+                                        case CF_MAXIMUM:
+                                               if (pdp_temp[ii] > rrd.cdp_prep[iii].scratch[CDP_val].u_val)
+                                                  rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                               break; 
+                                        case CF_LAST:
+                                        default:
+                                               rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                               break;
+                                        }
+                                 }
+                          }
+                       } else { /* rrd.rra_def[i].pdp_cnt == 1 */
+                          if (elapsed_pdp_st > 2)
+                          {
+                                  switch (current_cf) {
+                                  case CF_AVERAGE:
+                                  default:
+                                 rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val=pdp_temp[ii];
+                                 rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val=pdp_temp[ii];
+                                         break;
+                   case CF_SEASONAL:
+                                  case CF_DEVSEASONAL:
+                                         /* need to update cached seasonal values, so they are consistent
+                                          * with the bulk update */
+                      /* WARNING: code relies on the fact that CDP_hw_last_seasonal and
+                                          * CDP_last_deviation are the same. */
+                      rrd.cdp_prep[iii].scratch[CDP_hw_last_seasonal].u_val =
+                                                last_seasonal_coef[ii];
+                                         rrd.cdp_prep[iii].scratch[CDP_hw_seasonal].u_val =
+                                                seasonal_coef[ii];
+                                         break;
+                   case CF_HWPREDICT:
+                                         /* need to update the null_count and last_null_count.
+                                          * even do this for non-DNAN pdp_temp because the
+                                          * algorithm is not learning from batch updates. */
+                                         rrd.cdp_prep[iii].scratch[CDP_null_count].u_cnt += 
+                                                elapsed_pdp_st;
+                                         rrd.cdp_prep[iii].scratch[CDP_last_null_count].u_cnt += 
+                                                elapsed_pdp_st - 1;
+                                         /* fall through */
+                                  case CF_DEVPREDICT:
+                                 rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = DNAN;
+                                 rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val = DNAN;
+                                         break;
+                   case CF_FAILURES:
+                                         /* do not count missed bulk values as failures */
+                                 rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = 0;
+                                 rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val = 0;
+                                         /* need to reset violations buffer.
+                                          * could do this more carefully, but for now, just
+                                          * assume a bulk update wipes away all violations. */
+                      erase_violations(&rrd, iii, i);
+                                         break;
+                                  }
+                          } 
+                       } /* endif rrd.rra_def[i].pdp_cnt == 1 */
+
+                       if (rrd_test_error()) break;
+
+                       } /* endif data sources loop */
+        } /* end RRA Loop */
+
+               /* this loop is only entered if elapsed_pdp_st < 3 */
+               for (j = elapsed_pdp_st, scratch_idx = CDP_primary_val; 
+                        j > 0 && j < 3; j--, scratch_idx = CDP_secondary_val)
+               {
+              for(i = 0, rra_start = rra_begin;
+                  i < rrd.stat_head->rra_cnt;
+              rra_start += rrd.rra_def[i].row_cnt * rrd.stat_head -> ds_cnt * sizeof(rrd_value_t),
+                  i++)
+                  {
+                         if (rrd.rra_def[i].pdp_cnt > 1) continue;
+
+                 current_cf = cf_conv(rrd.rra_def[i].cf_nam);
+                         if (current_cf == CF_SEASONAL || current_cf == CF_DEVSEASONAL)
+                         {
+                        lookup_seasonal(&rrd,i,rra_start,rrd_file,
+                                   elapsed_pdp_st + (scratch_idx == CDP_primary_val ? 1 : 2),
+                               &seasonal_coef);
+                         }
+                         if (rrd_test_error()) break;
+                     /* loop over data soures within each RRA */
+                     for(ii = 0;
+                         ii < rrd.stat_head->ds_cnt;
+                         ii++)
+                         {
+                            update_aberrant_CF(&rrd,pdp_temp[ii],current_cf,
+                                       i*(rrd.stat_head->ds_cnt) + ii,i,ii,
+                                   scratch_idx, seasonal_coef);
+                         }
+           } /* end RRA Loop */
+                  if (rrd_test_error()) break;
+           } /* end elapsed_pdp_st loop */
+
+               if (rrd_test_error()) break;
+
+               /* Ready to write to disk */
+               /* Move sequentially through the file, writing one RRA at a time.
+                * Note this architecture divorces the computation of CDP with
+                * flushing updated RRA entries to disk. */
+           for(i = 0, rra_start = rra_begin;
+               i < rrd.stat_head->rra_cnt;
+           rra_start += rrd.rra_def[i].row_cnt * rrd.stat_head -> ds_cnt * sizeof(rrd_value_t),
+               i++) {
+               /* is there anything to write for this RRA? If not, continue. */
+        if (rra_step_cnt[i] == 0) continue;
+
+               /* write the first row */
 #ifdef DEBUG
-                                       fprintf(stderr,"  ** MAXIMUM %e\n", 
-                                               rrd.cdp_prep[iii].scratch[CDP_val].u_val);
+        fprintf(stderr,"  -- RRA Preseek %ld\n",ftell(rrd_file));
 #endif
-                                       break;
-                                   case CF_LAST:
-                                       rrd.cdp_prep[iii].scratch[CDP_val].u_val=pdp_temp[ii];
+           rrd.rra_ptr[i].cur_row++;
+           if (rrd.rra_ptr[i].cur_row >= rrd.rra_def[i].row_cnt)
+                  rrd.rra_ptr[i].cur_row = 0; /* wrap around */
+               /* positition on the first row */
+               rra_pos_tmp = rra_start +
+                  (rrd.stat_head->ds_cnt)*(rrd.rra_ptr[i].cur_row)*sizeof(rrd_value_t);
+               if(rra_pos_tmp != rra_current) {
+                  if(fseek(rrd_file, rra_pos_tmp, SEEK_SET) != 0){
+                     rrd_set_error("seek error in rrd");
+                     break;
+                  }
+                  rra_current = rra_pos_tmp;
+               }
 #ifdef DEBUG
-                                       fprintf(stderr,"  ** LAST %e\n", 
-                                               rrd.cdp_prep[iii].scratch[CDP_val].u_val);
+           fprintf(stderr,"  -- RRA Postseek %ld\n",ftell(rrd_file));
 #endif
-                                       break;    
-                                   default:
-                                       rrd_set_error("Unknown cf %s",
-                                                     rrd.rra_def[i].cf_nam);
-                                       break;
-                               }
-                           }
-                       }
-
-
-                       /* is the data in the cdp_prep ready to go into
-                        * its rra ? */
-                       if((pdp_st % 
-                           (rrd.rra_def[i].pdp_cnt*rrd.stat_head->pdp_step)) == 0){
-
-                           /* prepare cdp_pref for its transition to the rra. */
-                           if (rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt 
-                               > rrd.rra_def[i].pdp_cnt*
-                               rrd.rra_def[i].par[RRA_cdp_xff_val].u_val)
-                               /* to much of the cdp_prep is unknown ... */
-                               rrd.cdp_prep[iii].scratch[CDP_val].u_val = DNAN;
-                           else if (current_cf == CF_AVERAGE){
-                               /* for a real average we have to divide
-                                * the sum we built earlier on. While ignoring
-                                * the unknown pdps */
-                               rrd.cdp_prep[iii].scratch[CDP_val].u_val 
-                                       /= (rrd.rra_def[i].pdp_cnt
-                                           -rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt);
-                           }
-                           /* we can write straight away, because we are
-                            * already in the right place ... */
-
+               scratch_idx = CDP_primary_val;
+               write_RRA_row(&rrd, i, &rra_current, scratch_idx, rrd_file);
+               if (rrd_test_error()) break;
+
+               /* write other rows of the bulk update, if any */
+               scratch_idx = CDP_secondary_val;
+               for ( ; rra_step_cnt[i] > 1; 
+                    rra_step_cnt[i]--, rrd.rra_ptr[i].cur_row++)
+               {
+                  if (rrd.rra_ptr[i].cur_row == rrd.rra_def[i].row_cnt)
+                  {
 #ifdef DEBUG
-                           fprintf(stderr,"  -- RRA WRITE VALUE %e, at %ld\n",
-                                   rrd.cdp_prep[iii].scratch[CDP_val].u_val,ftell(rrd_file));
+              fprintf(stderr,"Wraparound for RRA %s, %lu updates left\n",
+                         rrd.rra_def[i].cf_nam, rra_step_cnt[i] - 1);
 #endif
-
-                           if(fwrite(&(rrd.cdp_prep[iii].scratch[CDP_val].u_val),
-                                     sizeof(rrd_value_t),1,rrd_file) != 1){
-                               rrd_set_error("writing rrd");
-                               break;
-                           }
-                           rra_current += sizeof(rrd_value_t);
-                           wrote_to_file = 1;
-
+                         /* wrap */
+                         rrd.rra_ptr[i].cur_row = 0;
+                         /* seek back to beginning of current rra */
+                     if (fseek(rrd_file, rra_start, SEEK_SET) != 0)
+                         {
+                        rrd_set_error("seek error in rrd");
+                        break;
+                         }
 #ifdef DEBUG
-                           fprintf(stderr,"  -- RRA WROTE new at %ld\n",ftell(rrd_file));
+                 fprintf(stderr,"  -- Wraparound Postseek %ld\n",ftell(rrd_file));
 #endif
-
-                           /* make cdp_prep ready for the next run */
-                           rrd.cdp_prep[iii].scratch[CDP_val].u_val = DNAN;
-                           rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt = 0;
-                       }
-                   }
-                   /* break out of this loop if error_string has been set */
-                   if (rrd_test_error())
-                       break;
+                         rra_current = rra_start;
+                  }
+                  write_RRA_row(&rrd, i, &rra_current, scratch_idx, rrd_file);
                }
-               /* break out of this loop if error_string has been set */
+               
                if (rrd_test_error())
-                   break;
-               /* to be able to position correctly in the next rra w move
-                * the rra_start pointer on to the next rra */
-               rra_start += rrd.rra_def[i].row_cnt
-                       *rrd.stat_head->ds_cnt*sizeof(rrd_value_t);
+                 break;
+               } /* RRA LOOP */
 
-           }
            /* break out of the argument parsing loop if error_string is set */
            if (rrd_test_error()){
-               free(step_start);
-               break;
-           }
-       }
+                  free(step_start);
+                  break;
+           } 
+           
+       } /* endif a pdp_st has occurred */ 
        rrd.live_head->last_up = current_time;
        free(step_start);
-    }
+    } /* function argument loop */
 
+    if (seasonal_coef != NULL) free(seasonal_coef);
+    if (last_seasonal_coef != NULL) free(last_seasonal_coef);
+       if (rra_step_cnt != NULL) free(rra_step_cnt);
 
     /* if we got here and if there is an error and if the file has not been
      * written to, then close things up and return. */
@@ -803,6 +1082,36 @@ rrd_update(int argc, char **argv)
        return(-1);
     }
 
+    /* calling the smoothing code here guarantees at most
+        * one smoothing operation per rrd_update call. Unfortunately,
+        * it is possible with bulk updates, or a long-delayed update
+        * for smoothing to occur off-schedule. This really isn't
+        * critical except during the burning cycles. */
+       if (schedule_smooth)
+       {
+#ifndef WIN32
+         rrd_file = fopen(argv[optind],"r+");
+#else
+         rrd_file = fopen(argv[optind],"rb+");
+#endif
+         rra_start = rra_begin;
+         for (i = 0; i < rrd.stat_head -> rra_cnt; ++i)
+         {
+           if (cf_conv(rrd.rra_def[i].cf_nam) == CF_DEVSEASONAL ||
+               cf_conv(rrd.rra_def[i].cf_nam) == CF_SEASONAL)
+           {
+#ifdef DEBUG
+             fprintf(stderr,"Running smoother for rra %ld\n",i);
+#endif
+             apply_smoother(&rrd,i,rra_start,rrd_file);
+             if (rrd_test_error())
+               break;
+           }
+           rra_start += rrd.rra_def[i].row_cnt
+             *rrd.stat_head->ds_cnt*sizeof(rrd_value_t);
+         }
+         fclose(rrd_file);
+       }
     rrd_free(&rrd);
     free(updvals);
     free(tmpl_idx);
@@ -847,3 +1156,30 @@ LockRRD(FILE *rrdfile)
 
     return(stat);
 }
+
+
+void
+write_RRA_row (rrd_t *rrd, unsigned long rra_idx, unsigned long *rra_current,
+   unsigned short CDP_scratch_idx, FILE *rrd_file)
+{
+   unsigned long ds_idx, cdp_idx;
+
+   for (ds_idx = 0; ds_idx < rrd -> stat_head -> ds_cnt; ds_idx++)
+   {
+      /* compute the cdp index */
+      cdp_idx =rra_idx * (rrd -> stat_head->ds_cnt) + ds_idx;
+#ifdef DEBUG
+         fprintf(stderr,"  -- RRA WRITE VALUE %e, at %ld CF:%s\n",
+            rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val,ftell(rrd_file),
+            rrd -> rra_def[rra_idx].cf_nam);
+#endif
+
+         if(fwrite(&(rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val),
+                sizeof(rrd_value_t),1,rrd_file) != 1)
+         { 
+            rrd_set_error("writing rrd");
+                return;
+         }
+         *rra_current += sizeof(rrd_value_t);
+       }
+}
diff --git a/src/rrdupdate.c b/src/rrdupdate.c
new file mode 100644 (file)
index 0000000..cbc99a6
--- /dev/null
@@ -0,0 +1,1184 @@
+/*****************************************************************************
+ * RRDtool 1.0.33  Copyright Tobias Oetiker, 1997 - 2000
+ *****************************************************************************
+ * rrd_update.c  RRD Update Function
+ *****************************************************************************
+ * $Id$
+ * $Log$
+ * Revision 1.3  2001/03/04 13:01:56  oetiker
+ * Aberrant Behavior Detection support. A brief overview added to rrdtool.pod.
+ * Major updates to rrd_update.c, rrd_create.c. Minor update to other core files.
+ * This is backwards compatible! But new files using the Aberrant stuff are not readable
+ * by old rrdtool versions. See http://cricket.sourceforge.net/aberrant/rrd_hw.htm
+ * -- Jake Brutlag <jakeb@corp.webtv.net>
+ *
+ * Revision 1.2  2001/03/04 11:14:25  oetiker
+ * added at-style-time@value:value syntax to rrd_update
+ * --  Dave Bodenstab <imdave@mcs.net>
+ * Revision 1.1  2001/02/25 22:25:06  oetiker
+ * Initial revision
+ *
+ *****************************************************************************/
+
+#include "rrd_tool.h"
+#include <sys/types.h>
+#include <fcntl.h>
+
+#ifdef WIN32
+ #include <sys/locking.h>
+ #include <sys/stat.h>
+ #include <io.h>
+#endif
+
+/* Prototypes */
+int LockRRD(FILE *rrd_file);
+void write_RRA_row (rrd_t *rrd, unsigned long rra_idx, unsigned long *rra_current,
+    unsigned short CDP_scratch_idx, FILE *rrd_file);
+/*#define DEBUG */
+
+#define IFDNAN(X,Y) (isnan(X) ? (Y) : (X));
+
+
+#ifdef STANDALONE
+int 
+main(int argc, char **argv){
+        rrd_update(argc,argv);
+        if (rrd_test_error()) {
+                printf("RRDtool 1.0.33  Copyright 1997-2000 by Tobias Oetiker <tobi@oetiker.ch>\n\n"
+                        "Usage: rrdupdate filename\n"
+                        "\t\t\t[--template|-t ds-name:ds-name:...]\n"
+                        "\t\t\ttime|N:value[:value...]\n\n"
+                        "\t\t\tat-time@value[:value...]\n\n"
+                        "\t\t\t[ time:value[:value...] ..]\n\n");
+                                   
+                printf("ERROR: %s\n",rrd_get_error());
+                rrd_clear_error();                                                            
+                return 1;
+        }
+        return 0;
+}
+#endif
+
+int
+rrd_update(int argc, char **argv)
+{
+
+    int              arg_i = 2;
+    short            j;
+    long             i,ii,iii=1;
+
+    unsigned long    rra_begin;          /* byte pointer to the rra
+                                         * area in the rrd file.  this
+                                         * pointer never changes value */
+    unsigned long    rra_start;          /* byte pointer to the rra
+                                         * area in the rrd file.  this
+                                         * pointer changes as each rrd is
+                                         * processed. */
+    unsigned long    rra_current;        /* byte pointer to the current write
+                                         * spot in the rrd file. */
+    unsigned long    rra_pos_tmp;        /* temporary byte pointer. */
+    unsigned long    interval,
+       pre_int,post_int;                /* interval between this and
+                                         * the last run */
+    unsigned long    proc_pdp_st;        /* which pdp_st was the last
+                                         * to be processed */
+    unsigned long    occu_pdp_st;        /* when was the pdp_st
+                                         * before the last update
+                                         * time */
+    unsigned long    proc_pdp_age;       /* how old was the data in
+                                         * the pdp prep area when it
+                                         * was last updated */
+    unsigned long    occu_pdp_age;       /* how long ago was the last
+                                         * pdp_step time */
+    rrd_value_t      *pdp_new;           /* prepare the incoming data
+                                         * to be added the the
+                                         * existing entry */
+    rrd_value_t      *pdp_temp;          /* prepare the pdp values 
+                                         * to be added the the
+                                         * cdp values */
+
+    long             *tmpl_idx;          /* index representing the settings
+                                           transported by the template index */
+    long             tmpl_cnt = 2;       /* time and data */
+
+    FILE             *rrd_file;
+    rrd_t            rrd;
+    time_t           current_time = time(NULL);
+    char             **updvals;
+    int              schedule_smooth = 0;
+    char             *template = NULL;   
+       rrd_value_t      *seasonal_coef = NULL, *last_seasonal_coef = NULL;
+                                        /* a vector of future Holt-Winters seasonal coefs */
+    unsigned long    elapsed_pdp_st;
+                                        /* number of elapsed PDP steps since last update */
+    unsigned long    *rra_step_cnt = NULL;
+                                        /* number of rows to be updated in an RRA for a data
+                                         * value. */
+    unsigned long    start_pdp_offset;
+                                        /* number of PDP steps since the last update that
+                                         * are assigned to the first CDP to be generated
+                                         * since the last update. */
+    unsigned short   scratch_idx;
+                                        /* index into the CDP scratch array */
+    enum cf_en       current_cf;
+                                        /* numeric id of the current consolidation function */
+
+    while (1) {
+       static struct option long_options[] =
+       {
+           {"template",      required_argument, 0, 't'},
+           {0,0,0,0}
+       };
+       int option_index = 0;
+       int opt;
+       opt = getopt_long(argc, argv, "t:", 
+                         long_options, &option_index);
+       
+       if (opt == EOF)
+         break;
+       
+       switch(opt) {
+       case 't':
+           template = optarg;
+           break;
+
+       case '?':
+           rrd_set_error("unknown option '%s'",argv[optind-1]);
+            rrd_free(&rrd);            
+           return(-1);
+       }
+    }
+
+    /* need at least 2 arguments: filename, data. */
+    if (argc-optind < 2) {
+       rrd_set_error("Not enough arguments");
+       return -1;
+    }
+
+    if(rrd_open(argv[optind],&rrd_file,&rrd, RRD_READWRITE)==-1){
+       return -1;
+    }
+    rra_current = rra_start = rra_begin = ftell(rrd_file);
+    /* This is defined in the ANSI C standard, section 7.9.5.3:
+
+        When a file is opened with udpate mode ('+' as the second
+        or third character in the ... list of mode argument
+        variables), both input and ouptut may be performed on the
+        associated stream.  However, ...  input may not be directly
+        followed by output without an intervening call to a file
+        positioning function, unless the input oepration encounters
+        end-of-file. */
+    fseek(rrd_file, 0, SEEK_CUR);
+
+    
+    /* get exclusive lock to whole file.
+     * lock gets removed when we close the file.
+     */
+    if (LockRRD(rrd_file) != 0) {
+      rrd_set_error("could not lock RRD");
+      rrd_free(&rrd);
+      fclose(rrd_file);
+      return(-1);   
+    } 
+
+    if((updvals = malloc( sizeof(char*) * (rrd.stat_head->ds_cnt+1)))==NULL){
+       rrd_set_error("allocating updvals pointer array");
+       rrd_free(&rrd);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    if ((pdp_temp = malloc(sizeof(rrd_value_t)
+                          *rrd.stat_head->ds_cnt))==NULL){
+       rrd_set_error("allocating pdp_temp ...");
+       free(updvals);
+       rrd_free(&rrd);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    if ((tmpl_idx = malloc(sizeof(unsigned long)
+                          *(rrd.stat_head->ds_cnt+1)))==NULL){
+       rrd_set_error("allocating tmpl_idx ...");
+       free(pdp_temp);
+       free(updvals);
+       rrd_free(&rrd);
+        fclose(rrd_file);
+       return(-1);
+    }
+    /* initialize template redirector */
+    /* default config
+       tmpl_idx[0] -> 0; (time)
+       tmpl_idx[1] -> 1; (DS 0)
+       tmpl_idx[2] -> 2; (DS 1)
+       tmpl_idx[3] -> 3; (DS 2)
+       ... */
+    for (i=0;i<=rrd.stat_head->ds_cnt;i++) tmpl_idx[i]=i;
+    tmpl_cnt=rrd.stat_head->ds_cnt+1;
+    if (template) {
+       char *dsname;
+       int tmpl_len;
+       dsname = template;
+       tmpl_cnt = 1; /* the first entry is the time */
+       tmpl_len = strlen(template);
+       for(i=0;i<=tmpl_len ;i++) {
+           if (template[i] == ':' || template[i] == '\0') {
+               template[i] = '\0';
+               if (tmpl_cnt>rrd.stat_head->ds_cnt){
+                   rrd_set_error("Template contains more DS definitions than RRD");
+                   free(updvals); free(pdp_temp);
+                   free(tmpl_idx); rrd_free(&rrd);
+                   fclose(rrd_file); return(-1);
+               }
+               if ((tmpl_idx[tmpl_cnt++] = ds_match(&rrd,dsname)) == -1){
+                   rrd_set_error("unknown DS name '%s'",dsname);
+                   free(updvals); free(pdp_temp);
+                   free(tmpl_idx); rrd_free(&rrd);
+                   fclose(rrd_file); return(-1);
+               } else {
+                 /* the first element is always the time */
+                 tmpl_idx[tmpl_cnt-1]++; 
+                 /* go to the next entry on the template */
+                 dsname = &template[i+1];
+                  /* fix the damage we did before */
+                  if (i<tmpl_len) {
+                     template[i]=':';
+                  } 
+
+               }
+           }       
+       }
+    }
+    if ((pdp_new = malloc(sizeof(rrd_value_t)
+                         *rrd.stat_head->ds_cnt))==NULL){
+       rrd_set_error("allocating pdp_new ...");
+       free(updvals);
+       free(pdp_temp);
+       free(tmpl_idx);
+       rrd_free(&rrd);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    /* loop through the arguments. */
+    for(arg_i=optind+1; arg_i<argc;arg_i++) {
+       char *stepper = malloc((strlen(argv[arg_i])+1)*sizeof(char));
+        char *step_start = stepper;
+       char *p;
+       char *parsetime_error = NULL;
+       enum {atstyle, normal} timesyntax;
+       struct time_value ds_tv;
+        if (stepper == NULL){
+                rrd_set_error("failed duplication argv entry");
+                free(updvals);
+                free(pdp_temp);  
+                free(tmpl_idx);
+                rrd_free(&rrd);
+                fclose(rrd_file);
+                return(-1);
+         }
+       /* initialize all ds input to unknown except the first one
+           which has always got to be set */
+       for(ii=1;ii<=rrd.stat_head->ds_cnt;ii++) updvals[ii] = "U";
+       strcpy(stepper,argv[arg_i]);
+       updvals[0]=stepper;
+       /* separate all ds elements; first must be examined separately
+          due to alternate time syntax */
+       if ((p=strchr(stepper,'@'))!=NULL) {
+           timesyntax = atstyle;
+           *p = '\0';
+           stepper = p+1;
+       } else if ((p=strchr(stepper,':'))!=NULL) {
+           timesyntax = normal;
+           *p = '\0';
+           stepper = p+1;
+       } else {
+           rrd_set_error("expected timestamp not found in data source from %s:...",
+                         argv[arg_i]);
+           free(step_start);
+           break;
+       }
+       ii=1;
+       updvals[tmpl_idx[ii]] = stepper;
+       while (*stepper) {
+           if (*stepper == ':') {
+               *stepper = '\0';
+               ii++;
+               if (ii<tmpl_cnt){                   
+                   updvals[tmpl_idx[ii]] = stepper+1;
+               }
+           }
+           stepper++;
+       }
+
+       if (ii != tmpl_cnt-1) {
+           rrd_set_error("expected %lu data source readings (got %lu) from %s:...",
+                         tmpl_cnt-1, ii, argv[arg_i]);
+           free(step_start);
+           break;
+       }
+       
+        /* get the time from the reading ... handle N */
+       if (timesyntax == atstyle) {
+            if ((parsetime_error = parsetime(updvals[0], &ds_tv))) {
+                rrd_set_error("ds time: %s: %s", updvals[0], parsetime_error );
+               free(step_start);
+               break;
+           }
+           if (ds_tv.type == RELATIVE_TO_END_TIME ||
+               ds_tv.type == RELATIVE_TO_START_TIME) {
+               rrd_set_error("specifying time relative to the 'start' "
+                              "or 'end' makes no sense here: %s",
+                             updvals[0]);
+               free(step_start);
+               break;
+           }
+
+           current_time = mktime(&ds_tv.tm) + ds_tv.offset;
+       } else if (strcmp(updvals[0],"N")==0){
+           current_time = time(NULL);
+       } else {
+           current_time = atol(updvals[0]);
+       }
+       
+       if(current_time <= rrd.live_head->last_up){
+           rrd_set_error("illegal attempt to update using time %ld when "
+                         "last update time is %ld (minimum one second step)",
+                         current_time, rrd.live_head->last_up);
+           free(step_start);
+           break;
+       }
+       
+       
+       /* seek to the beginning of the rra's */
+       if (rra_current != rra_begin) {
+           if(fseek(rrd_file, rra_begin, SEEK_SET) != 0) {
+               rrd_set_error("seek error in rrd");
+               free(step_start);
+               break;
+           }
+           rra_current = rra_begin;
+       }
+       rra_start = rra_begin;
+
+       /* when was the current pdp started */
+       proc_pdp_age = rrd.live_head->last_up % rrd.stat_head->pdp_step;
+       proc_pdp_st = rrd.live_head->last_up - proc_pdp_age;
+
+       /* when did the last pdp_st occur */
+       occu_pdp_age = current_time % rrd.stat_head->pdp_step;
+       occu_pdp_st = current_time - occu_pdp_age;
+       interval = current_time - rrd.live_head->last_up;
+    
+       if (occu_pdp_st > proc_pdp_st){
+           /* OK we passed the pdp_st moment*/
+           pre_int =  occu_pdp_st - rrd.live_head->last_up; /* how much of the input data
+                                                             * occurred before the latest
+                                                             * pdp_st moment*/
+           post_int = occu_pdp_age;                         /* how much after it */
+       } else {
+           pre_int = interval;
+           post_int = 0;
+       }
+
+#ifdef DEBUG
+       printf(
+              "proc_pdp_age %lu\t"
+              "proc_pdp_st %lu\t" 
+              "occu_pfp_age %lu\t" 
+              "occu_pdp_st %lu\t"
+              "int %lu\t"
+              "pre_int %lu\t"
+              "post_int %lu\n", proc_pdp_age, proc_pdp_st, 
+               occu_pdp_age, occu_pdp_st,
+              interval, pre_int, post_int);
+#endif
+    
+       /* process the data sources and update the pdp_prep 
+        * area accordingly */
+       for(i=0;i<rrd.stat_head->ds_cnt;i++){
+           enum dst_en dst_idx;
+           dst_idx= dst_conv(rrd.ds_def[i].dst);
+           if((updvals[i+1][0] != 'U') &&
+              rrd.ds_def[i].par[DS_mrhb_cnt].u_cnt >= interval) {
+              double rate = DNAN;
+              /* the data source type defines how to process the data */
+               /* pdp_temp contains rate * time ... eg the bytes
+                * transferred during the interval. Doing it this way saves
+                * a lot of math operations */
+               
+
+               switch(dst_idx){
+               case DST_COUNTER:
+               case DST_DERIVE:
+                   if(rrd.pdp_prep[i].last_ds[0] != 'U'){
+                      pdp_new[i]= rrd_diff(updvals[i+1],rrd.pdp_prep[i].last_ds);
+                      if(dst_idx == DST_COUNTER) {
+                         /* simple overflow catcher sugestet by andres kroonmaa */
+                         /* this will fail terribly for non 32 or 64 bit counters ... */
+                         /* are there any others in SNMP land ? */
+                         if (pdp_new[i] < (double)0.0 ) 
+                           pdp_new[i] += (double)4294967296.0 ;  /* 2^32 */
+                         if (pdp_new[i] < (double)0.0 ) 
+                           pdp_new[i] += (double)18446744069414584320.0; /* 2^64-2^32 */;
+                      }
+                      rate = pdp_new[i] / interval;
+                   }
+                  else {
+                    pdp_new[i]= DNAN;          
+                  }
+                  break;
+               case DST_ABSOLUTE:
+                   pdp_new[i]= atof(updvals[i+1]);
+                   rate = pdp_new[i] / interval;                 
+                   break;
+               case DST_GAUGE:
+                   pdp_new[i] = atof(updvals[i+1]) * interval;
+                   rate = pdp_new[i] / interval;                  
+                   break;
+               default:
+                   rrd_set_error("rrd contains unknown DS type : '%s'",
+                                 rrd.ds_def[i].dst);
+                   break;
+               }
+               /* break out of this for loop if the error string is set */
+               if (rrd_test_error()){
+                   break;
+               }
+              /* make sure pdp_temp is neither too large or too small
+               * if any of these occur it becomes unknown ...
+               * sorry folks ... */
+              if ( ! isnan(rate) && 
+                   (( ! isnan(rrd.ds_def[i].par[DS_max_val].u_val) &&
+                        rate > rrd.ds_def[i].par[DS_max_val].u_val ) ||     
+                   ( ! isnan(rrd.ds_def[i].par[DS_min_val].u_val) &&
+                       rate < rrd.ds_def[i].par[DS_min_val].u_val ))){
+                 pdp_new[i] = DNAN;
+              }               
+           } else {
+               /* no news is news all the same */
+               pdp_new[i] = DNAN;
+           }
+           
+           /* make a copy of the command line argument for the next run */
+#ifdef DEBUG
+           fprintf(stderr,
+                   "prep ds[%lu]\t"
+                   "last_arg '%s'\t"
+                   "this_arg '%s'\t"
+                   "pdp_new %10.2f\n",
+                   i,
+                   rrd.pdp_prep[i].last_ds,
+                   updvals[i+1], pdp_new[i]);
+#endif
+           if(dst_idx == DST_COUNTER || dst_idx == DST_DERIVE){
+               strncpy(rrd.pdp_prep[i].last_ds,
+                       updvals[i+1],LAST_DS_LEN-1);
+               rrd.pdp_prep[i].last_ds[LAST_DS_LEN-1]='\0';
+           }
+       }
+       /* break out of the argument parsing loop if the error_string is set */
+       if (rrd_test_error()){
+           free(step_start);
+           break;
+       }
+       /* has a pdp_st moment occurred since the last run ? */
+
+       if (proc_pdp_st == occu_pdp_st){
+           /* no we have not passed a pdp_st moment. therefore update is simple */
+
+           for(i=0;i<rrd.stat_head->ds_cnt;i++){
+               if(isnan(pdp_new[i]))
+                   rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt += interval;
+               else
+                   rrd.pdp_prep[i].scratch[PDP_val].u_val+= pdp_new[i];
+#ifdef DEBUG
+               fprintf(stderr,
+                       "NO PDP  ds[%lu]\t"
+                       "value %10.2f\t"
+                       "unkn_sec %5lu\n",
+                       i,
+                       rrd.pdp_prep[i].scratch[PDP_val].u_val,
+                       rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt);
+#endif
+           }   
+       } else {
+           /* an pdp_st has occurred. */
+
+           /* in pdp_prep[].scratch[PDP_val].u_val we have collected rate*seconds which 
+            * occurred up to the last run.        
+           pdp_new[] contains rate*seconds from the latest run.
+           pdp_temp[] will contain the rate for cdp */
+
+
+           for(i=0;i<rrd.stat_head->ds_cnt;i++){
+               /* update pdp_prep to the current pdp_st */
+               if(isnan(pdp_new[i]))
+                   rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt += pre_int;
+               else
+                   rrd.pdp_prep[i].scratch[PDP_val].u_val += 
+                       pdp_new[i]/(double)interval*(double)pre_int;
+
+               /* if too much of the pdp_prep is unknown we dump it */
+               if ((rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt 
+                    > rrd.ds_def[i].par[DS_mrhb_cnt].u_cnt) ||
+                   (occu_pdp_st-proc_pdp_st <= 
+                    rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt)) {
+                   pdp_temp[i] = DNAN;
+               } else {
+                   pdp_temp[i] = rrd.pdp_prep[i].scratch[PDP_val].u_val
+                       / (double)( occu_pdp_st
+                                  - proc_pdp_st
+                                  - rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt);
+               }
+               /* make pdp_prep ready for the next run */
+               if(isnan(pdp_new[i])){
+                   rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt = post_int;
+                   rrd.pdp_prep[i].scratch[PDP_val].u_val = 0.0;
+               } else {
+                   rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt = 0;
+                   rrd.pdp_prep[i].scratch[PDP_val].u_val = 
+                       pdp_new[i]/(double)interval*(double)post_int;
+               }
+
+#ifdef DEBUG
+               fprintf(stderr,
+                       "PDP UPD ds[%lu]\t"
+                       "pdp_temp %10.2f\t"
+                       "new_prep %10.2f\t"
+                       "new_unkn_sec %5lu\n",
+                       i, pdp_temp[i],
+                       rrd.pdp_prep[i].scratch[PDP_val].u_val,
+                       rrd.pdp_prep[i].scratch[PDP_unkn_sec_cnt].u_cnt);
+#endif
+           }
+
+               /* compute the number of elapsed pdp_st moments */
+               elapsed_pdp_st = (occu_pdp_st - proc_pdp_st) / rrd.stat_head -> pdp_step;
+#ifdef DEBUG
+               fprintf(stderr,"elapsed PDP steps: %lu\n", elapsed_pdp_st);
+#endif
+               if (rra_step_cnt == NULL)
+               {
+                  rra_step_cnt = (unsigned long *) 
+                         malloc((rrd.stat_head->rra_cnt)* sizeof(unsigned long));
+               }
+
+           for(i = 0, rra_start = rra_begin;
+               i < rrd.stat_head->rra_cnt;
+           rra_start += rrd.rra_def[i].row_cnt * rrd.stat_head -> ds_cnt * sizeof(rrd_value_t),
+               i++)
+               {
+               current_cf = cf_conv(rrd.rra_def[i].cf_nam);
+               start_pdp_offset = rrd.rra_def[i].pdp_cnt -
+                  (proc_pdp_st / rrd.stat_head -> pdp_step) % rrd.rra_def[i].pdp_cnt;
+        if (start_pdp_offset <= elapsed_pdp_st) {
+           rra_step_cnt[i] = (elapsed_pdp_st - start_pdp_offset) / 
+                     rrd.rra_def[i].pdp_cnt + 1;
+           } else {
+                  rra_step_cnt[i] = 0;
+               }
+
+               if (current_cf == CF_SEASONAL || current_cf == CF_DEVSEASONAL) 
+               {
+                  /* If this is a bulk update, we need to skip ahead in the seasonal
+                       * arrays so that they will be correct for the next observed value;
+                       * note that for the bulk update itself, no update will occur to
+                       * DEVSEASONAL or SEASONAL; futhermore, HWPREDICT and DEVPREDICT will
+                       * be set to DNAN. */
+           if (rra_step_cnt[i] > 2) 
+                  {
+                         /* skip update by resetting rra_step_cnt[i],
+                          * note that this is not data source specific; this is due
+                          * to the bulk update, not a DNAN value for the specific data
+                          * source. */
+                         rra_step_cnt[i] = 0;
+              lookup_seasonal(&rrd,i,rra_start,rrd_file,elapsed_pdp_st, 
+                            &last_seasonal_coef);
+                     lookup_seasonal(&rrd,i,rra_start,rrd_file,elapsed_pdp_st + 1,
+                            &seasonal_coef);
+                  }
+               
+                 /* periodically run a smoother for seasonal effects */
+                 /* Need to use first cdp parameter buffer to track
+                  * burnin (burnin requires a specific smoothing schedule).
+                  * The CDP_init_seasonal parameter is really an RRA level,
+                  * not a data source within RRA level parameter, but the rra_def
+                  * is read only for rrd_update (not flushed to disk). */
+                 iii = i*(rrd.stat_head -> ds_cnt);
+                 if (rrd.cdp_prep[iii].scratch[CDP_init_seasonal].u_cnt 
+                         <= BURNIN_CYCLES)
+                 {
+                    if (rrd.rra_ptr[i].cur_row + elapsed_pdp_st 
+                                > rrd.rra_def[i].row_cnt - 1) {
+                          /* mark off one of the burnin cycles */
+                          ++(rrd.cdp_prep[iii].scratch[CDP_init_seasonal].u_cnt);
+                      schedule_smooth = 1;
+                        }  
+                 } else {
+                        /* someone has no doubt invented a trick to deal with this
+                         * wrap around, but at least this code is clear. */
+                        if (rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt >
+                            rrd.rra_ptr[i].cur_row)
+                        {
+                                /* here elapsed_pdp_st = rra_step_cnt[i] because of 1-1
+                                 * mapping between PDP and CDP */
+                                if (rrd.rra_ptr[i].cur_row + elapsed_pdp_st
+                                       >= rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt)
+                                {
+#ifdef DEBUG
+                                       fprintf(stderr,
+                                       "schedule_smooth 1: cur_row %lu, elapsed_pdp_st %lu, smooth idx %lu\n",
+                    rrd.rra_ptr[i].cur_row, elapsed_pdp_st, 
+                                       rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt);
+#endif
+                                       schedule_smooth = 1;
+                                }
+             } else {
+                                /* can't rely on negative numbers because we are working with
+                                 * unsigned values */
+                                /* Don't need modulus here. If we've wrapped more than once, only
+                                 * one smooth is executed at the end. */
+                                if (rrd.rra_ptr[i].cur_row + elapsed_pdp_st >= rrd.rra_def[i].row_cnt
+                                       && rrd.rra_ptr[i].cur_row + elapsed_pdp_st - rrd.rra_def[i].row_cnt
+                                       >= rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt)
+                                {
+#ifdef DEBUG
+                                       fprintf(stderr,
+                                       "schedule_smooth 2: cur_row %lu, elapsed_pdp_st %lu, smooth idx %lu\n",
+                    rrd.rra_ptr[i].cur_row, elapsed_pdp_st, 
+                                       rrd.rra_def[i].par[RRA_seasonal_smooth_idx].u_cnt);
+#endif
+                                       schedule_smooth = 1;
+                                }
+                        }
+                 }
+
+             rra_current = ftell(rrd_file); 
+               } /* if cf is DEVSEASONAL or SEASONAL */
+
+        if (rrd_test_error()) break;
+
+                   /* update CDP_PREP areas */
+                   /* loop over data soures within each RRA */
+                   for(ii = 0;
+                       ii < rrd.stat_head->ds_cnt;
+                       ii++)
+                       {
+                       
+                       /* iii indexes the CDP prep area for this data source within the RRA */
+                       iii=i*rrd.stat_head->ds_cnt+ii;
+
+                       if (rrd.rra_def[i].pdp_cnt > 1) {
+                         
+                          if (rra_step_cnt[i] > 0) {
+                          /* If we are in this block, as least 1 CDP value will be written to
+                               * disk, this is the CDP_primary_val entry. If more than 1 value needs
+                               * to be written, then the "fill in" value is the CDP_secondary_val
+                               * entry. */
+                                 if (isnan(pdp_temp[ii]))
+                  {
+                                        rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt += start_pdp_offset;
+                                        rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val = DNAN;
+                                 } else {
+                                        /* CDP_secondary value is the RRA "fill in" value for intermediary
+                                         * CDP data entries. No matter the CF, the value is the same because
+                                         * the average, max, min, and last of a list of identical values is
+                                         * the same, namely, the value itself. */
+                                        rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val = pdp_temp[ii];
+                                 }
+                     
+                                 if (rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt
+                                     > rrd.rra_def[i].pdp_cnt*
+                                     rrd.rra_def[i].par[RRA_cdp_xff_val].u_val)
+                                 {
+                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = DNAN;
+                                        /* initialize carry over */
+                                        if (current_cf == CF_AVERAGE) {
+                                                  if (isnan(pdp_temp[ii])) { 
+                                                         rrd.cdp_prep[iii].scratch[CDP_val].u_val = DNAN;
+                                                  } else {
+                                                         rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii] *
+                                                                ((elapsed_pdp_st - start_pdp_offset) % rrd.rra_def[i].pdp_cnt);
+                                                  }
+                                        } else {
+                                               rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                        }
+                                 } else {
+                                        rrd_value_t cum_val, cur_val; 
+                                    switch (current_cf) {
+                                               case CF_AVERAGE:
+                                                 cum_val = IFDNAN(rrd.cdp_prep[iii].scratch[CDP_val].u_val, 0.0);
+                                                 cur_val = IFDNAN(pdp_temp[ii],0.0);
+                          rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val =
+                                              (cum_val + cur_val) /
+                                          (rrd.rra_def[i].pdp_cnt
+                                              -rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt);
+                                                  /* initialize carry over value */
+                                                  if (isnan(pdp_temp[ii])) { 
+                                                         rrd.cdp_prep[iii].scratch[CDP_val].u_val = DNAN;
+                                                  } else {
+                                                         rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii] *
+                                                                ((elapsed_pdp_st - start_pdp_offset) % rrd.rra_def[i].pdp_cnt);
+                                                  }
+                                                  break;
+                                               case CF_MAXIMUM:
+                                                 cum_val = IFDNAN(rrd.cdp_prep[iii].scratch[CDP_val].u_val, -DINF);
+                                                 cur_val = IFDNAN(pdp_temp[ii],-DINF);
+#ifdef DEBUG
+                                                 if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val) &&
+                                                         isnan(pdp_temp[ii])) {
+                                                    fprintf(stderr,
+                                                               "RRA %lu, DS %lu, both CDP_val and pdp_temp are DNAN!",
+                                                               i,ii);
+                                                        exit(-1);
+                                                 }
+#endif
+                                                 if (cur_val > cum_val)
+                                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = cur_val;
+                                                 else
+                                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = cum_val;
+                                                 /* initialize carry over value */
+                                                 rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                                 break;
+                                               case CF_MINIMUM:
+                                                 cum_val = IFDNAN(rrd.cdp_prep[iii].scratch[CDP_val].u_val, DINF);
+                                                 cur_val = IFDNAN(pdp_temp[ii],DINF);
+#ifdef DEBUG
+                                                 if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val) &&
+                                                         isnan(pdp_temp[ii])) {
+                                                    fprintf(stderr,
+                                                               "RRA %lu, DS %lu, both CDP_val and pdp_temp are DNAN!",
+                                                               i,ii);
+                                                        exit(-1);
+                                                 }
+#endif
+                                                 if (cur_val < cum_val)
+                                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = cur_val;
+                                                 else
+                                                        rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = cum_val;
+                                                 /* initialize carry over value */
+                                                 rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                                 break;
+                                               case CF_LAST:
+                                               default:
+                                                  rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = pdp_temp[ii];
+                                                  /* initialize carry over value */
+                                                  rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                               break;
+                                        }
+                                 } /* endif meets xff value requirement for a valid value */
+                                 /* initialize carry over CDP_unkn_pdp_cnt, this must after CDP_primary_val
+                                  * is set because CDP_unkn_pdp_cnt is required to compute that value. */
+                                 if (isnan(pdp_temp[ii]))
+                                        rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt = 
+                                               (elapsed_pdp_st - start_pdp_offset) % rrd.rra_def[i].pdp_cnt;
+                                 else
+                                        rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt = 0;
+               } else  /* rra_step_cnt[i]  == 0 */
+                          {
+#ifdef DEBUG
+                                 if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val)) {
+                                 fprintf(stderr,"schedule CDP_val update, RRA %lu DS %lu, DNAN\n",
+                                        i,ii);
+                                 } else {
+                                 fprintf(stderr,"schedule CDP_val update, RRA %lu DS %lu, %10.2f\n",
+                                        i,ii,rrd.cdp_prep[iii].scratch[CDP_val].u_val);
+                                 }
+#endif
+                                 if (isnan(pdp_temp[ii])) {
+                                rrd.cdp_prep[iii].scratch[CDP_unkn_pdp_cnt].u_cnt += elapsed_pdp_st;
+                                 } else if (isnan(rrd.cdp_prep[iii].scratch[CDP_val].u_val))
+                                 {
+                                        if (current_cf == CF_AVERAGE) {
+                                           rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii] *
+                                                  elapsed_pdp_st;
+                                        } else {
+                                           rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                        }
+#ifdef DEBUG
+                                        fprintf(stderr,"Initialize CDP_val for RRA %lu DS %lu: %10.2f\n",
+                                           i,ii,rrd.cdp_prep[iii].scratch[CDP_val].u_val);
+#endif
+                                 } else {
+                                        switch (current_cf) {
+                                        case CF_AVERAGE:
+                                           rrd.cdp_prep[iii].scratch[CDP_val].u_val += pdp_temp[ii] *
+                                                  elapsed_pdp_st;
+                                               break;
+                                        case CF_MINIMUM:
+                                               if (pdp_temp[ii] < rrd.cdp_prep[iii].scratch[CDP_val].u_val)
+                                                  rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                               break; 
+                                        case CF_MAXIMUM:
+                                               if (pdp_temp[ii] > rrd.cdp_prep[iii].scratch[CDP_val].u_val)
+                                                  rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                               break; 
+                                        case CF_LAST:
+                                        default:
+                                               rrd.cdp_prep[iii].scratch[CDP_val].u_val = pdp_temp[ii];
+                                               break;
+                                        }
+                                 }
+                          }
+                       } else { /* rrd.rra_def[i].pdp_cnt == 1 */
+                          if (elapsed_pdp_st > 2)
+                          {
+                                  switch (current_cf) {
+                                  case CF_AVERAGE:
+                                  default:
+                                 rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val=pdp_temp[ii];
+                                 rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val=pdp_temp[ii];
+                                         break;
+                   case CF_SEASONAL:
+                                  case CF_DEVSEASONAL:
+                                         /* need to update cached seasonal values, so they are consistent
+                                          * with the bulk update */
+                      /* WARNING: code relies on the fact that CDP_hw_last_seasonal and
+                                          * CDP_last_deviation are the same. */
+                      rrd.cdp_prep[iii].scratch[CDP_hw_last_seasonal].u_val =
+                                                last_seasonal_coef[ii];
+                                         rrd.cdp_prep[iii].scratch[CDP_hw_seasonal].u_val =
+                                                seasonal_coef[ii];
+                                         break;
+                   case CF_HWPREDICT:
+                                         /* need to update the null_count and last_null_count.
+                                          * even do this for non-DNAN pdp_temp because the
+                                          * algorithm is not learning from batch updates. */
+                                         rrd.cdp_prep[iii].scratch[CDP_null_count].u_cnt += 
+                                                elapsed_pdp_st;
+                                         rrd.cdp_prep[iii].scratch[CDP_last_null_count].u_cnt += 
+                                                elapsed_pdp_st - 1;
+                                         /* fall through */
+                                  case CF_DEVPREDICT:
+                                 rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = DNAN;
+                                 rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val = DNAN;
+                                         break;
+                   case CF_FAILURES:
+                                         /* do not count missed bulk values as failures */
+                                 rrd.cdp_prep[iii].scratch[CDP_primary_val].u_val = 0;
+                                 rrd.cdp_prep[iii].scratch[CDP_secondary_val].u_val = 0;
+                                         /* need to reset violations buffer.
+                                          * could do this more carefully, but for now, just
+                                          * assume a bulk update wipes away all violations. */
+                      erase_violations(&rrd, iii, i);
+                                         break;
+                                  }
+                          } 
+                       } /* endif rrd.rra_def[i].pdp_cnt == 1 */
+
+                       if (rrd_test_error()) break;
+
+                       } /* endif data sources loop */
+        } /* end RRA Loop */
+
+               /* this loop is only entered if elapsed_pdp_st < 3 */
+               for (j = elapsed_pdp_st, scratch_idx = CDP_primary_val; 
+                        j > 0 && j < 3; j--, scratch_idx = CDP_secondary_val)
+               {
+              for(i = 0, rra_start = rra_begin;
+                  i < rrd.stat_head->rra_cnt;
+              rra_start += rrd.rra_def[i].row_cnt * rrd.stat_head -> ds_cnt * sizeof(rrd_value_t),
+                  i++)
+                  {
+                         if (rrd.rra_def[i].pdp_cnt > 1) continue;
+
+                 current_cf = cf_conv(rrd.rra_def[i].cf_nam);
+                         if (current_cf == CF_SEASONAL || current_cf == CF_DEVSEASONAL)
+                         {
+                        lookup_seasonal(&rrd,i,rra_start,rrd_file,
+                                   elapsed_pdp_st + (scratch_idx == CDP_primary_val ? 1 : 2),
+                               &seasonal_coef);
+                         }
+                         if (rrd_test_error()) break;
+                     /* loop over data soures within each RRA */
+                     for(ii = 0;
+                         ii < rrd.stat_head->ds_cnt;
+                         ii++)
+                         {
+                            update_aberrant_CF(&rrd,pdp_temp[ii],current_cf,
+                                       i*(rrd.stat_head->ds_cnt) + ii,i,ii,
+                                   scratch_idx, seasonal_coef);
+                         }
+           } /* end RRA Loop */
+                  if (rrd_test_error()) break;
+           } /* end elapsed_pdp_st loop */
+
+               if (rrd_test_error()) break;
+
+               /* Ready to write to disk */
+               /* Move sequentially through the file, writing one RRA at a time.
+                * Note this architecture divorces the computation of CDP with
+                * flushing updated RRA entries to disk. */
+           for(i = 0, rra_start = rra_begin;
+               i < rrd.stat_head->rra_cnt;
+           rra_start += rrd.rra_def[i].row_cnt * rrd.stat_head -> ds_cnt * sizeof(rrd_value_t),
+               i++) {
+               /* is there anything to write for this RRA? If not, continue. */
+        if (rra_step_cnt[i] == 0) continue;
+
+               /* write the first row */
+#ifdef DEBUG
+        fprintf(stderr,"  -- RRA Preseek %ld\n",ftell(rrd_file));
+#endif
+           rrd.rra_ptr[i].cur_row++;
+           if (rrd.rra_ptr[i].cur_row >= rrd.rra_def[i].row_cnt)
+                  rrd.rra_ptr[i].cur_row = 0; /* wrap around */
+               /* positition on the first row */
+               rra_pos_tmp = rra_start +
+                  (rrd.stat_head->ds_cnt)*(rrd.rra_ptr[i].cur_row)*sizeof(rrd_value_t);
+               if(rra_pos_tmp != rra_current) {
+                  if(fseek(rrd_file, rra_pos_tmp, SEEK_SET) != 0){
+                     rrd_set_error("seek error in rrd");
+                     break;
+                  }
+                  rra_current = rra_pos_tmp;
+               }
+#ifdef DEBUG
+           fprintf(stderr,"  -- RRA Postseek %ld\n",ftell(rrd_file));
+#endif
+               scratch_idx = CDP_primary_val;
+               write_RRA_row(&rrd, i, &rra_current, scratch_idx, rrd_file);
+               if (rrd_test_error()) break;
+
+               /* write other rows of the bulk update, if any */
+               scratch_idx = CDP_secondary_val;
+               for ( ; rra_step_cnt[i] > 1; 
+                    rra_step_cnt[i]--, rrd.rra_ptr[i].cur_row++)
+               {
+                  if (rrd.rra_ptr[i].cur_row == rrd.rra_def[i].row_cnt)
+                  {
+#ifdef DEBUG
+              fprintf(stderr,"Wraparound for RRA %s, %lu updates left\n",
+                         rrd.rra_def[i].cf_nam, rra_step_cnt[i] - 1);
+#endif
+                         /* wrap */
+                         rrd.rra_ptr[i].cur_row = 0;
+                         /* seek back to beginning of current rra */
+                     if (fseek(rrd_file, rra_start, SEEK_SET) != 0)
+                         {
+                        rrd_set_error("seek error in rrd");
+                        break;
+                         }
+#ifdef DEBUG
+                 fprintf(stderr,"  -- Wraparound Postseek %ld\n",ftell(rrd_file));
+#endif
+                         rra_current = rra_start;
+                  }
+                  write_RRA_row(&rrd, i, &rra_current, scratch_idx, rrd_file);
+               }
+               
+               if (rrd_test_error())
+                 break;
+               } /* RRA LOOP */
+
+           /* break out of the argument parsing loop if error_string is set */
+           if (rrd_test_error()){
+                  free(step_start);
+                  break;
+           } 
+           
+       } /* endif a pdp_st has occurred */ 
+       rrd.live_head->last_up = current_time;
+       free(step_start);
+    } /* function argument loop */
+
+    if (seasonal_coef != NULL) free(seasonal_coef);
+    if (last_seasonal_coef != NULL) free(last_seasonal_coef);
+       if (rra_step_cnt != NULL) free(rra_step_cnt);
+
+    /* if we got here and if there is an error and if the file has not been
+     * written to, then close things up and return. */
+    if (rrd_test_error()) {
+       free(updvals);
+       free(tmpl_idx);
+       rrd_free(&rrd);
+       free(pdp_temp);
+       free(pdp_new);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    /* aargh ... that was tough ... so many loops ... anyway, its done.
+     * we just need to write back the live header portion now*/
+
+    if (fseek(rrd_file, (sizeof(stat_head_t)
+                        + sizeof(ds_def_t)*rrd.stat_head->ds_cnt 
+                        + sizeof(rra_def_t)*rrd.stat_head->rra_cnt),
+             SEEK_SET) != 0) {
+       rrd_set_error("seek rrd for live header writeback");
+       free(updvals);
+       free(tmpl_idx);
+       rrd_free(&rrd);
+       free(pdp_temp);
+       free(pdp_new);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    if(fwrite( rrd.live_head,
+              sizeof(live_head_t), 1, rrd_file) != 1){
+       rrd_set_error("fwrite live_head to rrd");
+       free(updvals);
+       rrd_free(&rrd);
+       free(tmpl_idx);
+       free(pdp_temp);
+       free(pdp_new);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    if(fwrite( rrd.pdp_prep,
+              sizeof(pdp_prep_t),
+              rrd.stat_head->ds_cnt, rrd_file) != rrd.stat_head->ds_cnt){
+       rrd_set_error("ftwrite pdp_prep to rrd");
+       free(updvals);
+       rrd_free(&rrd);
+       free(tmpl_idx);
+       free(pdp_temp);
+       free(pdp_new);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    if(fwrite( rrd.cdp_prep,
+              sizeof(cdp_prep_t),
+              rrd.stat_head->rra_cnt *rrd.stat_head->ds_cnt, rrd_file) 
+       != rrd.stat_head->rra_cnt *rrd.stat_head->ds_cnt){
+
+       rrd_set_error("ftwrite cdp_prep to rrd");
+       free(updvals);
+       free(tmpl_idx);
+       rrd_free(&rrd);
+       free(pdp_temp);
+       free(pdp_new);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    if(fwrite( rrd.rra_ptr,
+              sizeof(rra_ptr_t), 
+              rrd.stat_head->rra_cnt,rrd_file) != rrd.stat_head->rra_cnt){
+       rrd_set_error("fwrite rra_ptr to rrd");
+       free(updvals);
+       free(tmpl_idx);
+       rrd_free(&rrd);
+       free(pdp_temp);
+       free(pdp_new);
+        fclose(rrd_file);
+       return(-1);
+    }
+
+    /* OK now close the files and free the memory */
+    if(fclose(rrd_file) != 0){
+       rrd_set_error("closing rrd");
+       free(updvals);
+       free(tmpl_idx);
+       rrd_free(&rrd);
+       free(pdp_temp);
+       free(pdp_new);
+       return(-1);
+    }
+
+    /* calling the smoothing code here guarantees at most
+        * one smoothing operation per rrd_update call. Unfortunately,
+        * it is possible with bulk updates, or a long-delayed update
+        * for smoothing to occur off-schedule. This really isn't
+        * critical except during the burning cycles. */
+       if (schedule_smooth)
+       {
+#ifndef WIN32
+         rrd_file = fopen(argv[optind],"r+");
+#else
+         rrd_file = fopen(argv[optind],"rb+");
+#endif
+         rra_start = rra_begin;
+         for (i = 0; i < rrd.stat_head -> rra_cnt; ++i)
+         {
+           if (cf_conv(rrd.rra_def[i].cf_nam) == CF_DEVSEASONAL ||
+               cf_conv(rrd.rra_def[i].cf_nam) == CF_SEASONAL)
+           {
+#ifdef DEBUG
+             fprintf(stderr,"Running smoother for rra %ld\n",i);
+#endif
+             apply_smoother(&rrd,i,rra_start,rrd_file);
+             if (rrd_test_error())
+               break;
+           }
+           rra_start += rrd.rra_def[i].row_cnt
+             *rrd.stat_head->ds_cnt*sizeof(rrd_value_t);
+         }
+         fclose(rrd_file);
+       }
+    rrd_free(&rrd);
+    free(updvals);
+    free(tmpl_idx);
+    free(pdp_new);
+    free(pdp_temp);
+    return(0);
+}
+
+/*
+ * get exclusive lock to whole file.
+ * lock gets removed when we close the file
+ *
+ * returns 0 on success
+ */
+int
+LockRRD(FILE *rrdfile)
+{
+    int        rrd_fd;         /* File descriptor for RRD */
+    int                        stat;
+
+    rrd_fd = fileno(rrdfile);
+
+       {
+#ifndef WIN32    
+               struct flock    lock;
+    lock.l_type = F_WRLCK;    /* exclusive write lock */
+    lock.l_len = 0;          /* whole file */
+    lock.l_start = 0;        /* start of file */
+    lock.l_whence = SEEK_SET;   /* end of file */
+
+    stat = fcntl(rrd_fd, F_SETLK, &lock);
+#else
+               struct _stat st;
+
+               if ( _fstat( rrd_fd, &st ) == 0 ) {
+                       stat = _locking ( rrd_fd, _LK_NBLCK, st.st_size );
+               } else {
+                       stat = -1;
+               }
+#endif
+       }
+
+    return(stat);
+}
+
+
+void
+write_RRA_row (rrd_t *rrd, unsigned long rra_idx, unsigned long *rra_current,
+   unsigned short CDP_scratch_idx, FILE *rrd_file)
+{
+   unsigned long ds_idx, cdp_idx;
+
+   for (ds_idx = 0; ds_idx < rrd -> stat_head -> ds_cnt; ds_idx++)
+   {
+      /* compute the cdp index */
+      cdp_idx =rra_idx * (rrd -> stat_head->ds_cnt) + ds_idx;
+#ifdef DEBUG
+         fprintf(stderr,"  -- RRA WRITE VALUE %e, at %ld CF:%s\n",
+            rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val,ftell(rrd_file),
+            rrd -> rra_def[rra_idx].cf_nam);
+#endif
+
+         if(fwrite(&(rrd -> cdp_prep[cdp_idx].scratch[CDP_scratch_idx].u_val),
+                sizeof(rrd_value_t),1,rrd_file) != 1)
+         { 
+            rrd_set_error("writing rrd");
+                return;
+         }
+         *rra_current += sizeof(rrd_value_t);
+       }
+}