Imported upstream version 1.4.8

[pkg-rrdtool.git] / doc / cdeftutorial.1

.\" Automatically generated by Pod::Man 2.25 (Pod::Simple 3.16)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is turned on, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.ie \nF \{\
.    de IX
.    tm Index:\\$1\t\\n%\t"\\$2"
..
.    nr % 0
.    rr F
.\}
.el \{\
.    de IX
..
.\}
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear.  Run.  Save yourself.  No user-serviceable parts.
.    \" fudge factors for nroff and troff
.if n \{\
.    ds #H 0
.    ds #V .8m
.    ds #F .3m
.    ds #[ \f1
.    ds #] \fP
.\}
.if t \{\
.    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
.    ds #V .6m
.    ds #F 0
.    ds #[ \&
.    ds #] \&
.\}
.    \" simple accents for nroff and troff
.if n \{\
.    ds ' \&
.    ds ` \&
.    ds ^ \&
.    ds , \&
.    ds ~ ~
.    ds /
.\}
.if t \{\
.    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
.    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
.    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
.    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
.    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
.    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
.    \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
.    \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
.    \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
.    ds : e
.    ds 8 ss
.    ds o a
.    ds d- d\h'-1'\(ga
.    ds D- D\h'-1'\(hy
.    ds th \o'bp'
.    ds Th \o'LP'
.    ds ae ae
.    ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "CDEFTUTORIAL 1"
.TH CDEFTUTORIAL 1 "2013-05-23" "1.4.8" "rrdtool"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
cdeftutorial \- Alex van den Bogaerdt's CDEF tutorial
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
Intention of this document: to provide some examples of the commonly
used parts of RRDtool's \s-1CDEF\s0 language.
.PP
If you think some important feature is not explained properly, and if
adding it to this document would benefit most users, please do ask me
to add it.  I will then try to provide an answer in the next release
of this tutorial.  No feedback equals no changes! Additions to
this document are also welcome.  \*(-- Alex van den Bogaerdt
<alex@vandenbogaerdt.nl>
.SS "Why this tutorial?"
.IX Subsection "Why this tutorial?"
One of the powerful parts of RRDtool is its ability to do all sorts
of calculations on the data retrieved from its databases. However,
RRDtool's many options and syntax make it difficult for the average
user to understand. The manuals are good at explaining what these
options do; however they do not (and should not) explain in detail
why they are useful. As with my RRDtool tutorial: if you want a
simple document in simple language you should read this tutorial.
If you are happy with the official documentation, you may find this
document too simple or even boring. If you do choose to read this
tutorial, I also expect you to have read and fully understand my
other tutorial.
.SS "More reading"
.IX Subsection "More reading"
If you have difficulties with the way I try to explain it please read
Steve Rader's rpntutorial. It may help you understand how this all works.
.SH "What are CDEFs?"
.IX Header "What are CDEFs?"
When retrieving data from an \s-1RRD\s0, you are using a \*(L"\s-1DEF\s0\*(R" to work with
that data. Think of it as a variable that changes over time (where
time is the x\-axis). The value of this variable is what is found in
the database at that particular time and you can't do any
modifications on it. This is what CDEFs are for: they takes values
from DEFs and perform calculations on them.
.SH "Syntax"
.IX Header "Syntax"
.Vb 2
\&   DEF:var_name_1=some.rrd:ds_name:CF
\&   CDEF:var_name_2=RPN_expression
.Ve
.PP
You first define \*(L"var_name_1\*(R" to be data collected from data source
\&\*(L"ds_name\*(R" found in \s-1RRD\s0 \*(L"some.rrd\*(R" with consolidation function \*(L"\s-1CF\s0\*(R".
.PP
Assume the ifInOctets \s-1SNMP\s0 counter is saved in mrtg.rrd as the \s-1DS\s0 \*(L"in\*(R".
Then the following \s-1DEF\s0 defines a variable for the average of that
data source:
.PP
.Vb 1
\&   DEF:inbytes=mrtg.rrd:in:AVERAGE
.Ve
.PP
Say you want to display bits per second (instead of bytes per second
as stored in the database.)  You have to define a calculation
(hence \*(L"\s-1CDEF\s0\*(R") on variable \*(L"inbytes\*(R" and use that variable (inbits)
instead of the original:
.PP
.Vb 1
\&   CDEF:inbits=inbytes,8,*
.Ve
.PP
This tells RRDtool to multiply inbytes by eight to get inbits. I'll
explain later how this works. In the graphing or printing functions,
you can now use inbits where you would use inbytes otherwise.
.PP
Note that the variable name used in the \s-1CDEF\s0 (inbits) must not be the
same as the variable named in the \s-1DEF\s0 (inbytes)!
.SH "RPN-expressions"
.IX Header "RPN-expressions"
\&\s-1RPN\s0 is short-hand for Reverse Polish Notation. It works as follows.
You put the variables or numbers on a stack. You also put operations
(things-to-do) on the stack and this stack is then processed. The result
will be placed on the stack. At the end, there should be exactly one
number left: the outcome of the series of operations. If there is not
exactly one number left, RRDtool will complain loudly.
.PP
Above multiplication by eight will look like:
.IP "1." 4
Start with an empty stack
.IP "2." 4
Put the content of variable inbytes on the stack
.IP "3." 4
Put the number eight on the stack
.IP "4." 4
Put the operation multiply on the stack
.IP "5." 4
Process the stack
.IP "6." 4
Retrieve the value from the stack and put it in variable inbits
.PP
We will now do an example with real numbers. Suppose the variable
inbytes would have value 10, the stack would be:
.IP "1." 4
||
.IP "2." 4
|10|
.IP "3." 4
|10|8|
.IP "4." 4
|10|8|*|
.IP "5." 4
|80|
.IP "6." 4
||
.PP
Processing the stack (step 5) will retrieve one value from the stack
(from the right at step 4). This is the operation multiply and this
takes two values off the stack as input. The result is put back on the
stack (the value 80 in this case). For multiplication the order doesn't
matter, but for other operations like subtraction and division it does.
Generally speaking you have the following order:
.PP
.Vb 1
\&   y = A \- B  \-\->  y=minus(A,B)  \-\->  CDEF:y=A,B,\-
.Ve
.PP
This is not very intuitive (at least most people don't think so). For
the function f(A,B) you reverse the position of \*(L"f\*(R", but you do not
reverse the order of the variables.
.SH "Converting your wishes to RPN"
.IX Header "Converting your wishes to RPN"
First, get a clear picture of what you want to do. Break down the problem
in smaller portions until they cannot be split anymore. Then it is rather
simple to convert your ideas into \s-1RPN\s0.
.PP
Suppose you have several RRDs and would like to add up some counters in
them. These could be, for instance, the counters for every \s-1WAN\s0 link you
are monitoring.
.PP
You have:
.PP
.Vb 3
\&   router1.rrd with link1in link2in
\&   router2.rrd with link1in link2in
\&   router3.rrd with link1in link2in
.Ve
.PP
Suppose you would like to add up all these counters, except for link2in
inside router2.rrd. You need to do:
.PP
(in this example, \*(L"router1.rrd:link1in\*(R" means the \s-1DS\s0 link1in inside the
\&\s-1RRD\s0 router1.rrd)
.PP
.Vb 7
\&   router1.rrd:link1in
\&   router1.rrd:link2in
\&   router2.rrd:link1in
\&   router3.rrd:link1in
\&   router3.rrd:link2in
\&   \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-   +
\&   (outcome of the sum)
.Ve
.PP
As a mathematical function, this could be written:
.PP
\&\f(CW\*(C`add(router1.rrd:link1in , router1.rrd:link2in , router2.rrd:link1in , router3.rrd:link1in , router3.rrd:link2.in)\*(C'\fR
.PP
With RRDtool and \s-1RPN\s0, first, define the inputs:
.PP
.Vb 5
\&   DEF:a=router1.rrd:link1in:AVERAGE
\&   DEF:b=router1.rrd:link2in:AVERAGE
\&   DEF:c=router2.rrd:link1in:AVERAGE
\&   DEF:d=router3.rrd:link1in:AVERAGE
\&   DEF:e=router3.rrd:link2in:AVERAGE
.Ve
.PP
Now, the mathematical function becomes: \f(CW\*(C`add(a,b,c,d,e)\*(C'\fR
.PP
In \s-1RPN\s0, there's no operator that sums more than two values so you need
to do several additions. You add a and b, add c to the result, add d
to the result and add e to the result.
.PP
.Vb 5
\&   push a:         a     stack contains the value of a
\&   push b and add: b,+   stack contains the result of a+b
\&   push c and add: c,+   stack contains the result of a+b+c
\&   push d and add: d,+   stack contains the result of a+b+c+d
\&   push e and add: e,+   stack contains the result of a+b+c+d+e
.Ve
.PP
What was calculated here would be written down as:
.PP
.Vb 1
\&   ( ( ( (a+b) + c) + d) + e) >
.Ve
.PP
This is in \s-1RPN:\s0  \f(CW\*(C`CDEF:result=a,b,+,c,+,d,+,e,+\*(C'\fR
.PP
This is correct but it can be made more clear to humans. It does
not matter if you add a to b and then add c to the result or first
add b to c and then add a to the result. This makes it possible to
rewrite the \s-1RPN\s0 into \f(CW\*(C`CDEF:result=a,b,c,d,e,+,+,+,+\*(C'\fR which is
evaluated differently:
.PP
.Vb 10
\&   push value of variable a on the stack: a
\&   push value of variable b on the stack: a b
\&   push value of variable c on the stack: a b c
\&   push value of variable d on the stack: a b c d
\&   push value of variable e on the stack: a b c d e
\&   push operator + on the stack:          a b c d e +
\&   and process it:                        a b c P   (where P == d+e)
\&   push operator + on the stack:          a b c P +
\&   and process it:                        a b Q     (where Q == c+P)
\&   push operator + on the stack:          a b Q +
\&   and process it:                        a R       (where R == b+Q)
\&   push operator + on the stack:          a R +
\&   and process it:                        S         (where S == a+R)
.Ve
.PP
As you can see the \s-1RPN\s0 expression \f(CW\*(C`a,b,c,d,e,+,+,+,+,+\*(C'\fR will evaluate in
\&\f(CW\*(C`((((d+e)+c)+b)+a)\*(C'\fR and it has the same outcome as \f(CW\*(C`a,b,+,c,+,d,+,e,+\*(C'\fR.
This is called the commutative law of addition,
but you may forget this right away, as long as you remember what it
means.
.PP
Now look at an expression that contains a multiplication:
.PP
First in normal math: \f(CW\*(C`let result = a+b*c\*(C'\fR. In this case you can't
choose the order yourself, you have to start with the multiplication
and then add a to it. You may alter the position of b and c, you must
not alter the position of a and b.
.PP
You have to take this in consideration when converting this expression
into \s-1RPN\s0. Read it as: \*(L"Add the outcome of b*c to a\*(R" and then it is
easy to write the \s-1RPN\s0 expression: \f(CW\*(C`result=a,b,c,*,+\*(C'\fR
Another expression that would return the same: \f(CW\*(C`result=b,c,*,a,+\*(C'\fR
.PP
In normal math, you may encounter something like \*(L"a*(b+c)\*(R" and this
can also be converted into \s-1RPN\s0. The parenthesis just tell you to first
add b and c, and then multiply a with the result. Again, now it is
easy to write it in \s-1RPN:\s0 \f(CW\*(C`result=a,b,c,+,*\*(C'\fR. Note that this is very
similar to one of the expressions in the previous paragraph, only the
multiplication and the addition changed places.
.PP
When you have problems with \s-1RPN\s0 or when RRDtool is complaining, it's
usually a good thing to write down the stack on a piece of paper
and see what happens. Have the manual ready and pretend to be RRDtool.
Just do all the math by hand to see what happens, I'm sure this will
solve most, if not all, problems you encounter.
.SH "Some special numbers"
.IX Header "Some special numbers"
.SS "The unknown value"
.IX Subsection "The unknown value"
Sometimes collecting your data will fail. This can be very common,
especially when querying over busy links. RRDtool can be configured
to allow for one (or even more) unknown value(s) and calculate the missing
update. You can, for instance, query your device every minute. This is
creating one so called \s-1PDP\s0 or primary data point per minute. If you
defined your \s-1RRD\s0 to contain an \s-1RRA\s0 that stores 5\-minute values, you need
five of those PDPs to create one \s-1CDP\s0 (consolidated data point).
These PDPs can become unknown in two cases:
.IP "1." 4
The updates are too far apart. This is tuned using the \*(L"heartbeat\*(R" setting.
.IP "2." 4
The update was set to unknown on purpose by inserting no value (using the
template option) or by using \*(L"U\*(R" as the value to insert.
.PP
When a \s-1CDP\s0 is calculated, another mechanism determines if this \s-1CDP\s0 is valid
or not. If there are too many PDPs unknown, the \s-1CDP\s0 is unknown as well.
This is determined by the xff factor. Please note that one unknown counter
update can result in two unknown PDPs! If you only allow for one unknown
\&\s-1PDP\s0 per \s-1CDP\s0, this makes the \s-1CDP\s0 go unknown!
.PP
Suppose the counter increments with one per second and you retrieve it
every minute:
.PP
.Vb 7
\&   counter value    resulting rate
\&   10\*(Aq000
\&   10\*(Aq060            1; (10\*(Aq060\-10\*(Aq000)/60 == 1
\&   10\*(Aq120            1; (10\*(Aq120\-10\*(Aq060)/60 == 1
\&   unknown           unknown; you don\*(Aqt know the last value
\&   10\*(Aq240            unknown; you don\*(Aqt know the previous value
\&   10\*(Aq300            1; (10\*(Aq300\-10\*(Aq240)/60 == 1
.Ve
.PP
If the \s-1CDP\s0 was to be calculated from the last five updates, it would get
two unknown PDPs and three known PDPs. If xff would have been set to 0.5
which by the way is a commonly used factor, the \s-1CDP\s0 would have a known
value of 1. If xff would have been set to 0.2 then the resulting \s-1CDP\s0
would be unknown.
.PP
You have to decide the proper values for heartbeat, number of PDPs per
\&\s-1CDP\s0 and the xff factor. As you can see from the previous text they define
the behavior of your \s-1RRA\s0.
.SS "Working with unknown data in your database"
.IX Subsection "Working with unknown data in your database"
As you have read in the previous chapter, entries in an \s-1RRA\s0 can be
set to the unknown value. If you do calculations with this type of
value, the result has to be unknown too. This means that an expression
such as \f(CW\*(C`result=a,b,+\*(C'\fR will be unknown if either a or b is unknown.
It would be wrong to just ignore the unknown value and return the value
of the other parameter. By doing so, you would assume \*(L"unknown\*(R" means \*(L"zero\*(R"
and this is not true.
.PP
There has been a case where somebody was collecting data for over a year.
A new piece of equipment was installed, a new \s-1RRD\s0 was created and the
scripts were changed to add a counter from the old database and a counter
from the new database. The result was disappointing, a large part of
the statistics seemed to have vanished mysteriously ...
They of course didn't, values from the old database (known values) were
added to values from the new database (unknown values) and the result was
unknown.
.PP
In this case, it is fairly reasonable to use a \s-1CDEF\s0 that alters unknown
data into zero. The counters of the device were unknown (after all, it
wasn't installed yet!) but you know that the data rate through the device
had to be zero (because of the same reason: it was not installed).
.PP
There are some examples below that make this change.
.SS "Infinity"
.IX Subsection "Infinity"
Infinite data is another form of a special number. It cannot be
graphed because by definition you would never reach the infinite
value. You can think of positive and negative infinity depending on
the position relative to zero.
.PP
RRDtool is capable of representing (\-not\- graphing!) infinity by stopping
at its current maximum (for positive infinity) or minimum (for negative
infinity) without knowing this maximum (minimum).
.PP
Infinity in RRDtool is mostly used to draw an \s-1AREA\s0 without knowing its
vertical dimensions. You can think of it as drawing an \s-1AREA\s0 with an
infinite height and displaying only the part that is visible in the
current graph. This is probably a good way to approximate infinity
and it sure allows for some neat tricks. See below for examples.
.SS "Working with unknown data and infinity"
.IX Subsection "Working with unknown data and infinity"
Sometimes you would like to discard unknown data and pretend it is zero
(or any other value for that matter) and sometimes you would like to
pretend that known data is unknown (to discard known-to-be-wrong data).
This is why CDEFs have support for unknown data. There are also examples
available that show unknown data by using infinity.
.SH "Some examples"
.IX Header "Some examples"
.SS "Example: using a recently created \s-1RRD\s0"
.IX Subsection "Example: using a recently created RRD"
You are keeping statistics on your router for over a year now. Recently
you installed an extra router and you would like to show the combined
throughput for these two devices.
.PP
If you just add up the counters from router.rrd and router2.rrd, you
will add known data (from router.rrd) to unknown data (from router2.rrd) for
the bigger part of your stats. You could solve this in a few ways:
.IP "\(bu" 4
While creating the new database, fill it with zeros from the start to now.
You have to make the database start at or before the least recent time in
the other database.
.IP "\(bu" 4
Alternatively, you could use \s-1CDEF\s0 and alter unknown data to zero.
.PP
Both methods have their pros and cons. The first method is troublesome and
if you want to do that you have to figure it out yourself. It is not
possible to create a database filled with zeros, you have to put them in
manually. Implementing the second method is described next:
.PP
What we want is: \*(L"if the value is unknown, replace it with zero\*(R". This
could be written in pseudo-code as:  if (value is unknown) then (zero)
else (value). When reading the rrdgraph manual you notice the \*(L"\s-1UN\s0\*(R"
function that returns zero or one. You also notice the \*(L"\s-1IF\s0\*(R" function
that takes zero or one as input.
.PP
First look at the \*(L"\s-1IF\s0\*(R" function. It takes three values from the stack,
the first value is the decision point, the second value is returned to
the stack if the evaluation is \*(L"true\*(R" and if not, the third value is
returned to the stack. We want the \*(L"\s-1UN\s0\*(R" function to decide what happens
so we combine those two functions in one \s-1CDEF\s0.
.PP
Lets write down the two possible paths for the \*(L"\s-1IF\s0\*(R" function:
.PP
.Vb 2
\&   if true  return a
\&   if false return b
.Ve
.PP
In \s-1RPN:\s0  \f(CW\*(C`result=x,a,b,IF\*(C'\fR where \*(L"x\*(R" is either true or false.
.PP
Now we have to fill in \*(L"x\*(R", this should be the \*(L"(value is unknown)\*(R" part
and this is in \s-1RPN:\s0  \f(CW\*(C`result=value,UN\*(C'\fR
.PP
We now combine them: \f(CW\*(C`result=value,UN,a,b,IF\*(C'\fR and when we fill in the
appropriate things for \*(L"a\*(R" and \*(L"b\*(R" we're finished:
.PP
\&\f(CW\*(C`CDEF:result=value,UN,0,value,IF\*(C'\fR
.PP
You may want to read Steve Rader's \s-1RPN\s0 guide if you have difficulties
with the way I explained this last example.
.PP
If you want to check this \s-1RPN\s0 expression, just mimic RRDtool behavior:
.PP
.Vb 4
\&   For any known value, the expression evaluates as follows:
\&   CDEF:result=value,UN,0,value,IF  (value,UN) is not true so it becomes 0
\&   CDEF:result=0,0,value,IF         "IF" will return the 3rd value
\&   CDEF:result=value                The known value is returned
\&
\&   For the unknown value, this happens:
\&   CDEF:result=value,UN,0,value,IF  (value,UN) is true so it becomes 1
\&   CDEF:result=1,0,value,IF         "IF" sees 1 and returns the 2nd value
\&   CDEF:result=0                    Zero is returned
.Ve
.PP
Of course, if you would like to see another value instead of zero, you
can use that other value.
.PP
Eventually, when all unknown data is removed from the \s-1RRD\s0, you may want
to remove this rule so that unknown data is properly displayed.
.SS "Example: better handling of unknown data, by using time"
.IX Subsection "Example: better handling of unknown data, by using time"
The above example has one drawback. If you do log unknown data in
your database after installing your new equipment, it will also be
translated into zero and therefore you won't see that there was a
problem. This is not good and what you really want to do is:
.IP "\(bu" 4
If there is unknown data, look at the time that this sample was taken.
.IP "\(bu" 4
If the unknown value is before time xxx, make it zero.
.IP "\(bu" 4
If it is after time xxx, leave it as unknown data.
.PP
This is doable: you can compare the time that the sample was taken
to some known time. Assuming you started to monitor your device on
Friday September 17, 1999, 00:35:57 \s-1MET\s0 \s-1DST\s0. Translate this time in seconds
since 1970\-01\-01 and it becomes 937'521'357. If you process unknown values
that were received after this time, you want to leave them unknown and
if they were \*(L"received\*(R" before this time, you want to translate them
into zero (so you can effectively ignore them while adding them to your
other routers counters).
.PP
Translating Friday September 17, 1999, 00:35:57 \s-1MET\s0 \s-1DST\s0 into 937'521'357 can
be done by, for instance, using gnu date:
.PP
.Vb 1
\&   date \-d "19990917 00:35:57" +%s
.Ve
.PP
You could also dump the database and see where the data starts to be
known. There are several other ways of doing this, just pick one.
.PP
Now we have to create the magic that allows us to process unknown
values different depending on the time that the sample was taken.
This is a three step process:
.IP "1." 4
If the timestamp of the value is after 937'521'357, leave it as is.
.IP "2." 4
If the value is a known value, leave it as is.
.IP "3." 4
Change the unknown value into zero.
.PP
Lets look at part one:
.PP
.Vb 1
\&    if (true) return the original value
.Ve
.PP
We rewrite this:
.PP
.Vb 2
\&    if (true) return "a"
\&    if (false) return "b"
.Ve
.PP
We need to calculate true or false from step 1. There is a function
available that returns the timestamp for the current sample. It is
called, how surprisingly, \*(L"\s-1TIME\s0\*(R". This time has to be compared to
a constant number, we need \*(L"\s-1GT\s0\*(R". The output of \*(L"\s-1GT\s0\*(R" is true or false
and this is good input to \*(L"\s-1IF\s0\*(R". We want \*(L"if (time > 937521357) then
(return a) else (return b)\*(R".
.PP
This process was already described thoroughly in the previous chapter
so lets do it quick:
.PP
.Vb 4
\&   if (x) then a else b
\&      where x represents "time>937521357"
\&      where a represents the original value
\&      where b represents the outcome of the previous example
\&
\&   time>937521357       \-\-> TIME,937521357,GT
\&
\&   if (x) then a else b \-\-> x,a,b,IF
\&   substitute x         \-\-> TIME,937521357,GT,a,b,IF
\&   substitute a         \-\-> TIME,937521357,GT,value,b,IF
\&   substitute b         \-\-> TIME,937521357,GT,value,value,UN,0,value,IF,IF
.Ve
.PP
We end up with:
\&\f(CW\*(C`CDEF:result=TIME,937521357,GT,value,value,UN,0,value,IF,IF\*(C'\fR
.PP
This looks very complex, however, as you can see, it was not too hard to
come up with.
.SS "Example: Pretending weird data isn't there"
.IX Subsection "Example: Pretending weird data isn't there"
Suppose you have a problem that shows up as huge spikes in your graph.
You know this happens and why, so you decide to work around the problem.
Perhaps you're using your network to do a backup at night and by doing
so you get almost 10mb/s while the rest of your network activity does
not produce numbers higher than 100kb/s.
.PP
There are two options:
.IP "1." 4
If the number exceeds 100kb/s it is wrong and you want it masked out
by changing it into unknown.
.IP "2." 4
You don't want the graph to show more than 100kb/s.
.PP
Pseudo code: if (number > 100) then unknown else number
or
Pseudo code: if (number > 100) then 100 else number.
.PP
The second \*(L"problem\*(R" may also be solved by using the rigid option of
RRDtool graph, however this has not the same result. In this example
you can end up with a graph that does autoscaling. Also, if you use
the numbers to display maxima they will be set to 100kb/s.
.PP
We use \*(L"\s-1IF\s0\*(R" and \*(L"\s-1GT\s0\*(R" again. \*(L"if (x) then (y) else (z)\*(R" is written
down as \*(L"CDEF:result=x,y,z,IF\*(R"; now fill in x, y and z.
For x you fill in \*(L"number greater than 100kb/s\*(R" becoming
\&\*(L"number,100000,GT\*(R" (kilo is 1'000 and b/s is what we measure!).
The \*(L"z\*(R" part is \*(L"number\*(R" in both cases and the \*(L"y\*(R" part is either
\&\*(L"\s-1UNKN\s0\*(R" for unknown or \*(L"100000\*(R" for 100kb/s.
.PP
The two \s-1CDEF\s0 expressions would be:
.PP
.Vb 2
\&    CDEF:result=number,100000,GT,UNKN,number,IF
\&    CDEF:result=number,100000,GT,100000,number,IF
.Ve
.SS "Example: working on a certain time span"
.IX Subsection "Example: working on a certain time span"
If you want a graph that spans a few weeks, but would only want to
see some routers' data for one week, you need to \*(L"hide\*(R" the rest of
the time frame. Don't ask me when this would be useful, it's just
here for the example :)
.PP
We need to compare the time stamp to a begin date and an end date.
Comparing isn't difficult:
.PP
.Vb 2
\&        TIME,begintime,GE
\&        TIME,endtime,LE
.Ve
.PP
These two parts of the \s-1CDEF\s0 produce either 0 for false or 1 for true.
We can now check if they are both 0 (or 1) using a few \s-1IF\s0 statements
but, as Wataru Satoh pointed out, we can use the \*(L"*\*(R" or \*(L"+\*(R" functions
as logical \s-1AND\s0 and logical \s-1OR\s0.
.PP
For \*(L"*\*(R", the result will be zero (false) if either one of the two
operators is zero.  For \*(L"+\*(R", the result will only be false (0) when
two false (0) operators will be added.  Warning: *any* number not
equal to 0 will be considered \*(L"true\*(R". This means that, for instance,
\&\*(L"\-1,1,+\*(R" (which should be \*(L"true or true\*(R") will become \s-1FALSE\s0 ...
In other words, use \*(L"+\*(R" only if you know for sure that you have positive
numbers (or zero) only.
.PP
Let's compile the complete \s-1CDEF:\s0
.PP
.Vb 2
\&        DEF:ds0=router1.rrd:AVERAGE
\&        CDEF:ds0modified=TIME,begintime,GT,TIME,endtime,LE,*,ds0,UNKN,IF
.Ve
.PP
This will return the value of ds0 if both comparisons return true. You
could also do it the other way around:
.PP
.Vb 2
\&        DEF:ds0=router1.rrd:AVERAGE
\&        CDEF:ds0modified=TIME,begintime,LT,TIME,endtime,GT,+,UNKN,ds0,IF
.Ve
.PP
This will return an \s-1UNKNOWN\s0 if either comparison returns true.
.SS "Example: You suspect to have problems and want to see unknown data."
.IX Subsection "Example: You suspect to have problems and want to see unknown data."
Suppose you add up the number of active users on several terminal servers.
If one of them doesn't give an answer (or an incorrect one) you get \*(L"NaN\*(R"
in the database (\*(L"Not a Number\*(R") and NaN is evaluated as Unknown.
.PP
In this case, you would like to be alerted to it and the sum of the
remaining values is of no value to you.
.PP
It would be something like:
.PP
.Vb 5
\&    DEF:users1=location1.rrd:onlineTS1:LAST
\&    DEF:users2=location1.rrd:onlineTS2:LAST
\&    DEF:users3=location2.rrd:onlineTS1:LAST
\&    DEF:users4=location2.rrd:onlineTS2:LAST
\&    CDEF:allusers=users1,users2,users3,users4,+,+,+
.Ve
.PP
If you now plot allusers, unknown data in one of users1..users4 will
show up as a gap in your graph. You want to modify this to show a
bright red line, not a gap.
.PP
Define an extra \s-1CDEF\s0 that is unknown if all is okay and is infinite if
there is an unknown value:
.PP
.Vb 1
\&    CDEF:wrongdata=allusers,UN,INF,UNKN,IF
.Ve
.PP
\&\*(L"allusers,UN\*(R" will evaluate to either true or false, it is the (x) part
of the \*(L"\s-1IF\s0\*(R" function and it checks if allusers is unknown.
The (y) part of the \*(L"\s-1IF\s0\*(R" function is set to \*(L"\s-1INF\s0\*(R" (which means infinity)
and the (z) part of the function returns \*(L"\s-1UNKN\s0\*(R".
.PP
The logic is: if (allusers == unknown) then return \s-1INF\s0 else return \s-1UNKN\s0.
.PP
You can now use \s-1AREA\s0 to display this \*(L"wrongdata\*(R" in bright red. If it
is unknown (because allusers is known) then the red \s-1AREA\s0 won't show up.
If the value is \s-1INF\s0 (because allusers is unknown) then the red \s-1AREA\s0 will
be filled in on the graph at that particular time.
.PP
.Vb 2
\&   AREA:allusers#0000FF:combined user count
\&   AREA:wrongdata#FF0000:unknown data
.Ve
.SS "Same example useful with STACKed data:"
.IX Subsection "Same example useful with STACKed data:"
If you use stack in the previous example (as I would do) then you don't
add up the values. Therefore, there is no relationship between the
four values and you don't get a single value to test.
Suppose users3 would be unknown at one point in time: users1 is plotted,
users2 is stacked on top of users1, users3 is unknown and therefore
nothing happens, users4 is stacked on top of users2.
Add the extra CDEFs anyway and use them to overlay the \*(L"normal\*(R" graph:
.PP
.Vb 11
\&   DEF:users1=location1.rrd:onlineTS1:LAST
\&   DEF:users2=location1.rrd:onlineTS2:LAST
\&   DEF:users3=location2.rrd:onlineTS1:LAST
\&   DEF:users4=location2.rrd:onlineTS2:LAST
\&   CDEF:allusers=users1,users2,users3,users4,+,+,+
\&   CDEF:wrongdata=allusers,UN,INF,UNKN,IF
\&   AREA:users1#0000FF:users at ts1
\&   STACK:users2#00FF00:users at ts2
\&   STACK:users3#00FFFF:users at ts3
\&   STACK:users4#FFFF00:users at ts4
\&   AREA:wrongdata#FF0000:unknown data
.Ve
.PP
If there is unknown data in one of users1..users4, the \*(L"wrongdata\*(R" \s-1AREA\s0
will be drawn and because it starts at the X\-axis and has infinite height
it will effectively overwrite the STACKed parts.
.PP
You could combine the two \s-1CDEF\s0 lines into one (we don't use \*(L"allusers\*(R")
if you like.  But there are good reasons for writing two \s-1CDEFS:\s0
.IP "\(bu" 4
It improves the readability of the script.
.IP "\(bu" 4
It can be used inside \s-1GPRINT\s0 to display the total number of users.
.PP
If you choose to combine them, you can substitute the \*(L"allusers\*(R" in the
second \s-1CDEF\s0 with the part after the equal sign from the first line:
.PP
.Vb 1
\&   CDEF:wrongdata=users1,users2,users3,users4,+,+,+,UN,INF,UNKN,IF
.Ve
.PP
If you do so, you won't be able to use these next GPRINTs:
.PP
.Vb 5
\&   COMMENT:"Total number of users seen"
\&   GPRINT:allusers:MAX:"Maximum: %6.0lf"
\&   GPRINT:allusers:MIN:"Minimum: %6.0lf"
\&   GPRINT:allusers:AVERAGE:"Average: %6.0lf"
\&   GPRINT:allusers:LAST:"Current: %6.0lf\en"
.Ve
.SH "The examples from the RRD graph manual page"
.IX Header "The examples from the RRD graph manual page"
.SS "Degrees Celsius vs. Degrees Fahrenheit"
.IX Subsection "Degrees Celsius vs. Degrees Fahrenheit"
To convert Celsius into Fahrenheit use the formula
F=9/5*C+32
.PP
.Vb 5
\&   rrdtool graph demo.png \-\-title="Demo Graph" \e
\&      DEF:cel=demo.rrd:exhaust:AVERAGE \e
\&      CDEF:far=9,5,/,cel,*,32,+ \e
\&      LINE2:cel#00a000:"D. Celsius" \e
\&      LINE2:far#ff0000:"D. Fahrenheit\ec"
.Ve
.PP
This example gets the \s-1DS\s0 called \*(L"exhaust\*(R" from database \*(L"demo.rrd\*(R"
and puts the values in variable \*(L"cel\*(R". The \s-1CDEF\s0 used is evaluated
as follows:
.PP
.Vb 10
\&   CDEF:far=9,5,/,cel,*,32,+
\&   1. push 9, push 5
\&   2. push function "divide" and process it
\&      the stack now contains 9/5
\&   3. push variable "cel"
\&   4. push function "multiply" and process it
\&      the stack now contains 9/5*cel
\&   5. push 32
\&   6. push function "plus" and process it
\&      the stack contains now the temperature in Fahrenheit
.Ve
.SS "Changing unknown into zero"
.IX Subsection "Changing unknown into zero"
.Vb 9
\&   rrdtool graph demo.png \-\-title="Demo Graph" \e
\&      DEF:idat1=interface1.rrd:ds0:AVERAGE \e
\&      DEF:idat2=interface2.rrd:ds0:AVERAGE \e
\&      DEF:odat1=interface1.rrd:ds1:AVERAGE \e
\&      DEF:odat2=interface2.rrd:ds1:AVERAGE \e
\&      CDEF:agginput=idat1,UN,0,idat1,IF,idat2,UN,0,idat2,IF,+,8,* \e
\&      CDEF:aggoutput=odat1,UN,0,odat1,IF,odat2,UN,0,odat2,IF,+,8,* \e
\&      AREA:agginput#00cc00:Input Aggregate \e
\&      LINE1:aggoutput#0000FF:Output Aggregate
.Ve
.PP
These two CDEFs are built from several functions. It helps to split
them when viewing what they do. Starting with the first \s-1CDEF\s0 we would
get:
.PP
.Vb 4
\& idat1,UN \-\-> a
\& 0        \-\-> b
\& idat1    \-\-> c
\& if (a) then (b) else (c)
.Ve
.PP
The result is therefore \*(L"0\*(R" if it is true that \*(L"idat1\*(R" equals \*(L"\s-1UN\s0\*(R".
If not, the original value of \*(L"idat1\*(R" is put back on the stack.
Lets call this answer \*(L"d\*(R". The process is repeated for the next
five items on the stack, it is done the same and will return answer
\&\*(L"h\*(R". The resulting stack is therefore \*(L"d,h\*(R".
The expression has been simplified to \*(L"d,h,+,8,*\*(R" and it will now be
easy to see that we add \*(L"d\*(R" and \*(L"h\*(R", and multiply the result with eight.
.PP
The end result is that we have added \*(L"idat1\*(R" and \*(L"idat2\*(R" and in the
process we effectively ignored unknown values. The result is multiplied
by eight, most likely to convert bytes/s to bits/s.
.SS "Infinity demo"
.IX Subsection "Infinity demo"
.Vb 10
\&   rrdtool graph example.png \-\-title="INF demo" \e
\&      DEF:val1=some.rrd:ds0:AVERAGE \e
\&      DEF:val2=some.rrd:ds1:AVERAGE \e
\&      DEF:val3=some.rrd:ds2:AVERAGE \e
\&      DEF:val4=other.rrd:ds0:AVERAGE \e
\&      CDEF:background=val4,POP,TIME,7200,%,3600,LE,INF,UNKN,IF \e
\&      CDEF:wipeout=val1,val2,val3,val4,+,+,+,UN,INF,UNKN,IF \e
\&      AREA:background#F0F0F0 \e
\&      AREA:val1#0000FF:Value1 \e
\&      STACK:val2#00C000:Value2 \e
\&      STACK:val3#FFFF00:Value3 \e
\&      STACK:val4#FFC000:Value4 \e
\&      AREA:whipeout#FF0000:Unknown
.Ve
.PP
This demo demonstrates two ways to use infinity. It is a bit tricky
to see what happens in the \*(L"background\*(R" \s-1CDEF\s0.
.PP
.Vb 1
\&   "val4,POP,TIME,7200,%,3600,LE,INF,UNKN,IF"
.Ve
.PP
This \s-1RPN\s0 takes the value of \*(L"val4\*(R" as input and then immediately
removes it from the stack using \*(L"\s-1POP\s0\*(R". The stack is now empty but
as a side effect we now know the time that this sample was taken.
This time is put on the stack by the \*(L"\s-1TIME\s0\*(R" function.
.PP
\&\*(L"\s-1TIME\s0,7200,%\*(R" takes the modulo of time and 7'200 (which is two hours).
The resulting value on the stack will be a number in the range from
0 to 7199.
.PP
For people who don't know the modulo function: it is the remainder
after an integer division. If you divide 16 by 3, the answer would
be 5 and the remainder would be 1. So, \*(L"16,3,%\*(R" returns 1.
.PP
We have the result of \*(L"\s-1TIME\s0,7200,%\*(R" on the stack, lets call this
\&\*(L"a\*(R". The start of the \s-1RPN\s0 has become \*(L"a,3600,LE\*(R" and this checks
if \*(L"a\*(R" is less or equal than \*(L"3600\*(R". It is true half of the time.
We now have to process the rest of the \s-1RPN\s0 and this is only a simple
\&\*(L"\s-1IF\s0\*(R" function that returns either \*(L"\s-1INF\s0\*(R" or \*(L"\s-1UNKN\s0\*(R" depending on the
time. This is returned to variable \*(L"background\*(R".
.PP
The second \s-1CDEF\s0 has been discussed earlier in this document so we
won't do that here.
.PP
Now you can draw the different layers. Start with the background
that is either unknown (nothing to see) or infinite (the whole
positive part of the graph gets filled).
.PP
Next you draw the data on top of this background, it will overlay
the background. Suppose one of val1..val4 would be unknown, in that
case you end up with only three bars stacked on top of each other.
You don't want to see this because the data is only valid when all
four variables are valid. This is why you use the second \s-1CDEF\s0, it
will overlay the data with an \s-1AREA\s0 so the data cannot be seen anymore.
.PP
If your data can also have negative values you also need to overwrite
the other half of your graph. This can be done in a relatively simple
way: what you need is the \*(L"wipeout\*(R" variable and place a negative
sign before it:  \*(L"CDEF:wipeout2=wipeout,\-1,*\*(R"
.SS "Filtering data"
.IX Subsection "Filtering data"
You may do some complex data filtering:
.PP
.Vb 1
\&  MEDIAN FILTER: filters shot noise
\&
\&    DEF:var=database.rrd:traffic:AVERAGE
\&    CDEF:prev1=PREV(var)
\&    CDEF:prev2=PREV(prev1)
\&    CDEF:prev3=PREV(prev2)
\&    CDEF:median=prev1,prev2,prev3,+,+,3,/
\&    LINE3:median#000077:filtered
\&    LINE1:prev2#007700:\*(Aqraw data\*(Aq
\&
\&
\&  DERIVATE:
\&
\&    DEF:var=database.rrd:traffic:AVERAGE
\&    CDEF:prev1=PREV(var)
\&    CDEF:time=var,POP,TIME
\&    CDEF:prevtime=PREV(time)
\&    CDEF:derivate=var,prev1,\-,time,prevtime,\-,/
\&    LINE3:derivate#000077:derivate
\&    LINE1:var#007700:\*(Aqraw data\*(Aq
.Ve
.SH "Out of ideas for now"
.IX Header "Out of ideas for now"
This document was created from questions asked by either myself or by
other people on the RRDtool mailing list. Please let me know if you
find errors in it or if you have trouble understanding it. If you
think there should be an addition, mail me:
<alex@vandenbogaerdt.nl>
.PP
Remember: \fBNo feedback equals no changes!\fR
.SH "SEE ALSO"
.IX Header "SEE ALSO"
The RRDtool manpages
.SH "AUTHOR"
.IX Header "AUTHOR"
Alex van den Bogaerdt
<alex@vandenbogaerdt.nl>