Short User's guide for Jugi's Traffic Generator (JTG) The JTG generator is a simple, yet powerful and, if required, accurate traffic generator. jtg differs from other traffic generators in that one instance of a jtg process only sends one stream (e.g. MGEN can send tens of streams) and the characteristics of the stream are given only with command line arguments. On the other hand, a receiving jtg process can be set to loop and receive several streams but, again, only one at a time. The received packets can be logged and the log files can be analyzed with jtg_calc (see later in this file for instructions). jtg6 is the binary for IPv6 networks. One of the main reasons for writing this piece of code has been that it was hard to find a traffic generator that did all the right things, and in a proper way. What is needed is a simple piece of software that would accurately, and without surprises, send a single stream to a receiver. Versions of jtg have already been used by several MSc and PhD students in their experiments at the Department of Computer Science in University of Helsinki. Installation is pretty straight forward, just do "make" for compiling jtg and jtg_calc, and "make jtg6" to compile the IPv6-version of jtg. There are no configuration files so you can copy the binaries on the directory of your choice. The only requirement is that if you intend to use jtg with Gnuplot, be aware that jtg writes a special file by default into its working directory, so you need to have write access into that particular directory. If you want, you can change this filename with the "-G" parameter. 1. OPERATING A SENDER One sender process will send one data stream and then exit. In the most basic set up, the sender only needs to know the target host where to send the stream. With default values, the sender will send a TCP stream with 100 write()-calls (note that a single parameter can not define exact packet size to be used with TCP as it provides a byte stream service) of 1000 byte each to the port 2000 on the target host. The sender signals the end of the transmission by closing the connection (note that in typical case the transmission "over the wire" continues long beyond that). In case of a UDP stream, the sender will send after the primary data three (for reliability) small packets that tell the receiver that the stream has ended. It is also possible to define a TCP port where the sender can inform that a UDP transfer has ended - see the "alternative" method below. In all UDP transfers, the packet size must be greater than 12, because each packet holds three 4-byte unsigned long integers, that give the packet sequence number and the time the packet was sent as seconds and micro seconds as given by gettimeofday(). The following parameters can be used on the sender side: -u use UDP instead of TCP -sxx set socket or TCP option=value pairs -Sxx set socket or TCP option=value pairs (restore old value on exit) -l## length of packets written to network in bytes (def: 1000) -n## number of bufs written to network (def: 100) -p## port number to send to (def: 2000) -x## port number in tx (def: 0=free choice) -I## Bind receiver to specified IP address -d## transmission time in seconds. This will overwrite the -n parameter if both were mistakenly given. -T## length of "tick" in usec (default 100000=100ms). The tick is the delay between the transmission of two UDP packets. -B## number of packets per each tick (def: 1) [CBR] -b## User data bandwidth for CBR UDP in bits/s -L enable identical bursts -N## number of identical bursts -D## delay between identical bursts (default 100000=100ms) -o## DSCP for flow (decimal). Possible values: EF: 46, AF1: 10/12/14, AF2: 18/20/22, AF3: 26/28/30, AF4: 34/36/38 -w force the sender to use busy waiting -W force the sender to use select (-w -W: use both based on HZ) -P take into account processing latencies (send packets more accurately) -a use alternative host IP/name to inform that the test has ended, default is receiver's addr and port (using TCP) -Axx define another [addr:]port for communicating with the receiver (using TCP) -fxx take packet size info from file "xx" -Fxx packet delay interval info from file "xx" -k kill the receiver AFTER this transfer (used by only the sender) -e## delay in seconds before ending test (sender & receiver, def: 0) -y give the process super priority "-20" on Linux (for root) -i normalize the timestamps -v verbose mode, tells a lot more about what is happening (UDP only) -q "be quiet!", thus, no output -h help To send a UDP stream, it is possible to give a) a delay between packets and a packet size b) a target bandwidth and packet size: jtg calculates the delay between packets c) a target bandwidth and delay between packets: jtg calculates the proper packet size d) a delay between packets and a file to read packet sizes from e) the packet size and a file to read delay values from f) packet sizes and delay values, both, are read from files Certain parameter combinations are not possible, for example, giving a file where to read delay values or packet size information and simultaneously giving a target bandwidth with "-b". Similarly giving a packet size and packet file, or giving a "tick" and a delay file are not possible, for obvious reasons. To end a transfer, a TCP sender closes the connection, and a UDP sender sends three small packets whose payload tells the receiver that the transfer has ended. With the "alternative" method, a TCP port is used, where the sender first makes a connection, and, after the transfer, informs the receiver that all data has been sent. With "-a" the TCP port is the same as the UDP port used to receive data, with "-A" you can give another port and even another IP address. This is useful, for example, when you have a test lab where the primary data goes through a series of hosts and/or simulators, but the control information should be send through another route. With the "-k" parameter, the sender can force the receiver to exit after the current transfer. It is possible to kill the receiver without sending any actual data packet. Just set the number of packets to zero and set the "-k" parameter. The typical parameters specify a single burst. It is possible to send many identical bursts in the stream with a configurable delay between the bursts. By default, the stream includes just one burst. Thus, you can, e.g., send n packets with a fixed interval, wait a certain time, and then send the same set of packets. See in Section 1.3 for an example. 1.1 READING VALUES FROM FILES jtg can also read the information about packet transmission intervals and packet sizes from files. Thus, it is possible to create an exact duplicate of a trace from some stream, or to create random values outside of jtg and use those values to define the pattern of transmitted UDP packets. The values are used in sequence and the reading will loop back to the start when all numbers have been used. To limit the transmission to the number of values in the file, use the "-n xxx" to give the number of packets to send (which should be equal to the number of values in the file, right?). Two parameters are available: -f filename: gives the filename that includes positive integer values to be used for the transmitted packet size. The value defines bytes. -F filename: gives the filename that includes positive integer values to be used for the delay between sending a packet. The value defines micro seconds (us), thus, "12000" means 12 milli seconds (ms). Note that using -f or -F forces also forces a UDP stream. 1.2 ACCURACY OF TRANSMISSIONS Since a Linux kernel is not a real-time kernel and there are other processes running, too, the accuracy of the transmission can suffer. The most common cause is the select()-call which in many Linux kernels has an accuracy of around 10 ms. This means, that, for example, if one wants to send packets every 12 ms, the actual transmission interval is closer to 20 ms. This accuracy can be controlled with the kernel compilation-time HZ-definition (in header file linux/include/asm-YOUR_ARCH/param.h). Still, even with a proper kernel, the accuracy can suffer. jtg has several parameters that can be used to control the accuracy. -w can be used to force the sender to do busy-waiting when "sleeping" between packet transmissions. However, this puts the process at 100% load. -W forces the sender to use the select()-call and the accuracy is as good as the kernel HZ-value allows. Using -w and -W together results in a hybrid mode, which means that the delay()-function in the code will first try to sleep with the select()-call an amount less than the requested, and then use busy-waiting to sleep the remaining time, if any. The "-P" parameter can be used to take into account the processing latency and the error in the select()-call. The result is that packets will not be sent exactly as instructed by the delay ("tick") value but the transmission time will be calculated relative to the start of the test. This results in more jitter between transmitted packets, but the overall time to send the packets (and thus actual bandwidth on the link) will be more accurate. As an example, consider the case where you would send 11 packets of 1000 bytes each with an interval of 25 ms. With a select()-call accuracy of 10 ms (and, thus, an over-sleep of 5 ms), the packet would be sent roughly according to the left column and with the -P parameter according to the right column Pkt # w/o -P with -P 0 0:00.000 0:00.00 1 0:00.030 0:00.30 (tried for 25ms, over-slept 5 ms) 2 0:00.060 0:00.50 (tried for 5ms less than before) 3 0:00.090 0:00.80 (tried for 25ms, got 30ms) 4 0:00.120 0:00.100 (tried for 5ms less than before) 5 0:00.150 0:00.130 (tried for 25ms, got 30ms) 6 0:00.180 0:00.150 (tried for 5ms less than before) 7 0:00.210 0:00.180 (tried for 25ms, got 30ms) 8 0:00.240 0:00.200 (tried for 5ms less than before) 9 0:00.270 0:00.230 (tried for 25ms, got 30ms) 10 0:00.300 0:00.250 (tried for 5ms less than before) 1.3 SOME EXAMPLES jtg -t -u -n50 rx.cs.helsinki.fi Sends 50 UDP packets every 100 ms (default) to rx.cs.helsinki.fi jtg -t -u -b64000 -d5 -l160 -o 46 rx.cs.helsinki.fi Sends a UDP stream of 64 Kbps user data bandwidth and packet size 160 bytes. The transmission will last 5 seconds. The sender also marks packets with the DiffServ Expedited Forwarding Code Point (decimal number 46). jtg -t -u -w -f packet_sizes.log -T25000 rx.cs.helsinki.fi Sends a UDP stream, one packet every 25 ms and the packet sizes are read from the file packet_sizes.log. The delay between packet consecutive packets is measured using a busy loop (-w). jtg -t -p3456 -n200 -l1000 rx.cs.helsinki.fi Sends a 200 Kbyte TCP stream to receiver's port 3456. jtg -t -p3456 -n200 -l1000 -I 10.10.1.11 rx.cs.helsinki.fi Sends a 200 Kbyte TCP stream to receiver's port 3456, and binds the sender to the interface whose address is 10.10.1.11. jtg -t -u -n50 -L -N3 -D2000000 -w -v rx.cs.helsinki.fi Sends 3 identical bursts (-L -N3) of 50 UDP packets every 100 ms (default). The bursts have 2 seconds space between each (-D). The packets have consecutive sequence numbers and do not require the receiver to handle multiple streams with the receiver -L option. 2. OPERATING A RECEIVER Operating a receiver is a lot simpler than setting the sender. The parameters for the receiver are: -u receive UDP, without this the default is to receive TCP streams -sxx set socket or TCP option=value pairs -Sxx set socket or TCP option=value pairs (restore old value on exit) -p# port number at which we are listening for streams (def is 2000) -A# use a specific TCP port where the sender will inform that all packets have been sent, only meaningful for UDP transfers (def. is the same port number as in the UDP stream) -e# wait a few seconds before closing the UDP socket (may help in avoiding ICMP error messages if the receiving socket is closed too quickly) -l# set the buffer for the read-call -L loop and receive more than one stream -N# exit after having received this many streams -i normalize the timestamps written into the logfile -g output a log compatible with Gnuplot -G specify the filename for Gnuplot (def. jtg.dat) -y give the process super priority "-20" on Linux (for root) -v verbose mode, prints more info in the terminal -V extra information into log files (UDP only, presents some diffs of sender and receiver jitter etc.) -q quiet mode, no printing of anything on stdout -Q quiet, but prints to stdout the log of received packets (useful, if jtg is integrated with other software) 2.1 EXAMPLES FOR SETTING DIFFERENT RECEIVERS jtg -r Sets up a very basic receiver that waits for one TCP stream at port 2000. jtg -r -u -p31000 Sets up a very basic receiver that waits for one UDP stream at port 31000. jtg -r -u -L Sets up a receiver that listens for an unspecified number of UDP streams. This receiver can be stopped with CTRL-C or with a message sent by the sender (-k) that closes the receiver. jtg -r -u -L -a Sets up a receiver that listens for an unspecified number of UDP streams and waits for control messages at TCP port 2000. jtg -r -u -L -A2222 Sets up a receiver that listens for an unspecified number of UDP streams and waits for control messages at TCP port 2222. jtg -r -u -L -N3 -a Sets up a receiver that listens for three UDP streams and then exits. The receiver may also exit if it is ordered to by the sender with a kill message at TCP port 2000. 2.2 THE PACKET LOG The receiving jtg can write a log of the received data. Both UDP and TCP are supported, but the logs differ. The packet log for UDP looks like this: # Test started Wed May 14 11:01:17 200, sender 127.0.0.1:27777 Seq>0 TxTime>11:01:17.118654 RxTime>11:01:17.118683 Size>1000 Seq>1 TxTime>11:01:17.221263 RxTime>11:01:17.221296 Size>1000 Seq>2 TxTime>11:01:17.322816 RxTime>11:01:17.322847 Size>1000 Seq>3 TxTime>11:01:17.424367 RxTime>11:01:17.424385 Size>1000 Seq>4 TxTime>11:01:17.525924 RxTime>11:01:17.525957 Size>1000 where Seq is the sequence number of the packet and TxTime is the time the packet was sent from the process, set into the packet by the sender. RxTime is the time stamp when the packet was read by the receiver. This is taken with the ioctl() command from the kernel, or, if the user does not have privileges, it is taken with the gettimeofday() call. Size is the number of bytes returned by the read()-call. The log of a TCP transfer only gives the basic statistics, since there is no notion of packets at the application layer in a TCP byte stream. The log can look, for example, like this: # Test started Thu May 21 13:30:19 2003, sender 128.214.11.174:32807 transmission time 8.515695 s. received bytes 97345000, read()-calls 67479 average user data bandwidth 91450.0 Kbit/s You can also use "-R" in the receiver to log timestamps are given by gettimeofday. These are not as easily readable, but will allow you to more safely run experiments, where the days change as the experiments run. This log would look as follows: # Test started Tue Nov 15 16:09:16 2005, sender 127.0.0.1:32827 Seq>0 TxTime>1132063756.273401 RxTime>1132063756.273466 Size>1000 Seq>1 TxTime>1132063756.375820 RxTime>1132063756.375844 Size>1000 Seq>2 TxTime>1132063756.475829 RxTime>1132063756.475853 Size>1000 Seq>3 TxTime>1132063756.575841 RxTime>1132063756.575867 Size>1000 Seq>4 TxTime>1132063756.675855 RxTime>1132063756.675885 Size>1000 Seq>5 TxTime>1132063756.775879 RxTime>1132063756.775920 Size>1000 Seq>6 TxTime>1132063756.875871 RxTime>1132063756.875895 Size>1000 Seq>7 TxTime>1132063756.975884 RxTime>1132063756.975922 Size>1000 Seq>8 TxTime>1132063757.075889 RxTime>1132063757.075914 Size>1000 Seq>9 TxTime>1132063757.175933 RxTime>1132063757.175956 Size>1000 2.2.1 SOME EXAMPLES OF LOGGING jtg -r -u test.log Sets up a receiver that listens for one UDP stream and writes a log of the received packets. jtg -r -L test.log Sets up a receiver that listens for an unlimited number of TCP streams and writes a log of the received packets of each stream into separate log files. The files are numbered "test.log.1" for the first flow, "test.log.2" for the second flow etc. jtg -r -u -Q Sets up a UDP receiver which outputs the log to stdout. Only the log is written out, no extra information usually output by jtg. 3. SETTING TCP PARAMETERS jtg allows setting generic- and TCP-related socket options and system variables found under /proc. More specifically, jtg allows setting o 8 "standard" socket options mentioned in Steven's book, or o on Linux 14 setsockopt()-parameters and any parameter found under /proc Note that setting /proc parameters requires root privileges. The command-line format for setting a parameter is "-s option=value", for example, "-s timestamps=1". The option names are shorter versions of the socket options or of the entries under /proc - take a look at the header jtg.h. Numerous options can be set by using individual "-s" parameters, that is, one for each option, for example "jtg -s MAXSEG=200 -s timestamps=0 -s ecn=1". Note that the /proc entries are system wide, thus, if you set an entry, it will be used for all connections from the host until it is touched again. jtg allows you to restore the option on exit, that is, you can use the /proc entry only for your test and it will be restored to the old setting on exit. This is achieved by using "-S" instead of "-s". You can add your favorite parameters quite easily by adding a new struct entry into the right tcpopts[] table. See the jtg.h header file and the struct tcpopts_struct for the entries: set the parameter string representation, the length of the string, the (default) value and the old value do not need to be set (in other entries these are set to zero). Set the old value, the kernel define for the parameter, or the path under /proc, either "USE_SETSOCKOPT" or "USE_PROC", depending on which interface the parameter applies to, and set the parameter to "NOT_USED" initially. To be exact, you should add the entry before the last null entry. 4. NORMALIZING THE TIMESTAMPS The jtg sender and receiver can also be made to normalize the timestamps written in log files. This means that the first packet sent has timestamp 0:0:0.000000 and all subsequent values and stored relative to the first value. The parameter in both the sender and receiver is "-i". Normalizing requires the sender to send two timestamps, the wall clock of the current packet and the wall clock when the transmission started. Thus, the packet size must be at least 20 bytes. The receiver then does all calculations to normalize the values when the information is logged. The parameter must be given in the sender and receiver. If only the receiver did the normalizing and the first packet of the stream was lost, it could not tell what is the baseline from where to start the normalizing. The login thus works in the following way: - The sender sends two timestamps in each packet, the wall clock of the current packet, and the time the transmision started. - The receiver calculates its own TIME_BASE relative to the first packet it receives. Thus, TIME_BASE = (wall_clock - transmission_start_time). Transmision time is calculated from the wall clocks of the sender and receiver. All timestamps written into the log files are normalized relative to the start time at the sender. 5. RUNNING JTG WITH GNUPLOT jtg can be set to write a ready-to-plot data file of a running test. It is also possible to draw a real-time plot of the stream. To draw a real-time plot, start the sender as you normally would. At the receiver, you need to give the parameter "-g". Note, that in this case, no other logs will be created. The Gnuplot-feature uses one special log file and three ready-made Gnuplot files to drawing the real-time graph. For a UDP transfer, the files are: "jtg.dat": has four columns, where each row is one packet. First column gives the time (relative to a start time of zero) in seconds when the packet arrived, second column is the packet sequence number, the third column gives the transmission delay, and fourth column is the jitter. If no value is available, a "-" is output, for example: 0.0 0 0.032 - 0.103 1 0.034 102.885 0.214 2 0.035 111.307 0.335 3 0.039 121.083 0.468 4 0.034 132.634 0.606 5 0.038 138.828 0.757 6 0.037 150.377 0.917 7 0.034 160.154 1.089 8 0.035 171.857 In this example, we can not calculate a jitter for the first packet. Thus, the first line only has the arrival (start) time of the transfer, the packet number (zero is the first number) and the transmission delay in milli seconds. "jtg_jitter.gpl": used to draw the real-time jitter graph "jtg_delay.gpl": used to draw the real-time delay graph "jtg_pckts.gpl": used to draw the graph of arrived (and lost) packets For a TCP transfer, the file jtg.dat will include the application's view of when data was received. The series shows the arrival of data (cumulative) as a function of time. The file jtg.dat only includes two columns, the time and the cumulative amount of data. The file "jtg_tcp.gpl" can be used to draw the graph, either real-time or after the test run. Note that this information is very different from what you would get with tcpdump, for example. To show the graphs, first start the receiver, then the sender, and only then you can give "gnuplot jtg_xxx.gpl" for drawing graphs. One or more graphs of a running UDP stream can be drawn at the same time. The drawing takes into account lost packets, and leaves those pieces out. The packet sequence number graph shows lost packets as value zero (0). You can change the default filename "jtg.dat" with the "-G" parameter. Note that the gnuplot scripts expect to find a file named "jtg.dat" in the working directory, so edit the scripts as required. 6. HOW TO ANALYZE THE LOGS? You can use your own software to analyze the logs, or use the jtg_calc tool. The tool reads the file from stdin and saves a lot of information to several files, including various graphs. The tool expects the log line structure of UDP logs written by jtg, for example: Seq>3 TxTime>11:01:17.424367 RxTime>11:01:17.424385 Size>1000 Lines starting with '#' are considered comments and will be skipped. Basic usage is: Usage: jtg_calc < logfile Parameters: -s make a sanity check of the values (def. no sanity check) -m# minimum delay considered sane in us (def: 1000 us = 1 ms) -M# maximum delay considered sane in us (def: 1000000 us = 1 s.) -d# add a delay correction of the clocks (us) -f# add a drift correction of the clocks (+/- us in 1 sec) -i# add an ID (string) to identify these files -a# set the moving average value of pkts for graph (def. 200) -l no files, just output performance numbers to screen -n if duplicate packets encountered, use new value (def. use first) -N just normalize the timestamps, no analyze -pmin,max only consider a subsection of packet numbers [min,max] -R treat all values as given by gettimeofday(), same as in jtg. -h this help. -v verbose eg. jtg_calc -i test1 < test.log jtg_calc then creates several files stats_test1.txt overall statistics of the stream delay_test1.dat graph of the delay as a function of time jitter_test1.dat graph of the jitter as a function of time ipdv_test1.dat graph of the IPDV as a function of time delay_test1_pdf.dat PDF of the delay jitter_test1_pdf.dat PDF of the jitter ipdv_test1_pdf.dat PDF of the IPDV delay_test1_df.dat DF of the delay jitter_test1_df.dat DF of the jitter ipdv_test1_df.dat DF of the IPDV delay_test1_avg.dat moving average of the delay lost_test1.dat lost packet as a function of time The correction values can be used to play with the time stamp values, if the clocks of the sender and receiver are not in synch and move at a different pace: "delay correction" is a static correction of the two clocks, the difference of the clocks at the beginning of the test. "drift correction" is the drift of the clocks during one second, that is, how much do the clocks advance differently. The problem with the clocks drifting becomes quite important when the transmission times are in the range of a few milliseconds. You can see the phenomenon when you send a stream in an empty network and a graph of the test shows that the transmission times of packets either grow or packets somehow start to arrive constantly earlier, which eventually may result in packets arriving from the future (at least from the receiver's point of view :). You can also get the statistics of a certain part of the stream by using the "-p" parameter. The limits are the packet numbers, including the given numbers, taken into account in the statistics. Yet, for example, to get info from the last 100 packets, you could also just select the lines from the log file you want analyzed and pipe them to jtg_calc. To get the last 100 packets, you can use "tail -100 test.log | jtg_cal -l" If you used -R in the receiver, and stored timestamps as given by gettimeofday, you should use the same "-R" in jtg_calc. Yet, jtg_calc tries to figure this out by itself, too. jtg_calc can also be used to normalize the timeout values logged by the jtg-receiver. For example, the following log lines # Test started Fri Oct 10 14:55:11 2003, sender 127.0.0.1:33450 Seq>0 TxTime>14:55:11.006751 RxTime>14:55:11.006792 Size>1000 Seq>1 TxTime>14:55:11.027683 RxTime>14:55:11.027729 Size>1000 would be turned into # Test started Fri Oct 10 14:55:11 2003, sender 127.0.0.1:33450 Seq>0 TxTime>0:00:00.000000 RxTime>0:00:00.000041 Size>1000 Seq>1 TxTime>0:00:00.020932 RxTime>0:00:00.020978 Size>1000 The usage is: jtg_calc -N < test.log > test.log.normalized NOTE: due to a lazy implementation, jtg_calc only calculates statistics from the first 1'000'000 packets ("Should be enough for anyone"). If you send more than that, change the "#define MAX 1000000" in the header jtg_calc.h to something more appropriate and re-compile. This limitation is due to the code that stores values into a table. You could change this to some nice linked list structure that grows in size as needed. However, this adds complexity, processing, and we can argue whether it is worth the effort. Just add more table entries with the "#define MAX" if you need. 7. CREATING RANDOM VALUES The tool "jrg" (Jugi's Random (value) Generator) can be used to create series of random values. There values can be used to create variable bit-rate streams using variable packet sizes and/or intervals between packets. The tool recognizes several distributions: triangular, uniform, normal, exponential, hyper-exponential, Cauchy, log-normal, gamma, beta, static, Poisson, and a two-state Markov distribution. The tool writes the values to stdout, which you can pipe to any file you desire. Usage example, creating 1200 values from a uniform distribution between 10 and 100: jrg uniform 10,100 1200 > values.txt The tool "jrg_test" can be used to output a distribution of values to see whether a set of values conforms to a required distribution. Currently, jrg_test only takes one parameter, the number of intervals to divide the values into. Usage example: jrg_test 100 < values.txt or you could even make a lengthy pipeline, like: jrg uniform 10,100 1200 | jrg_test 100 | xgraph AUTHOR Jukka Manner University of Helsinki Department of Computer Science http://www.cs.helsinki.fi/u/jmanner/ STD DISCLAIMER This software is provided "AS IS". The author may be able to provide assistance, but, then again, may not. The author will gladly receive comments, suggestions and ideas for enhancements. The author may not be held responsible for any bugs the software has, nor may he be held responsible for any damage caused by this software, including (but not limited to) downloading the software, compiling it, starting it, using it, relying on it. All I can say is that I trust my work, for what it is worth...