howtos:toolcoding:performance_tuning
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
howtos:toolcoding:performance_tuning [2011/06/22 18:31] chris.strashok |
howtos:toolcoding:performance_tuning [2011/06/22 21:01] (current) chris.strashok |
||
|---|---|---|---|
| Line 13: | Line 13: | ||
| Facts for a 32bit machine: | Facts for a 32bit machine: | ||
| * The actual size of a TOOL object is usually 8 * number of elements since 8 bytes are required for each double. | * The actual size of a TOOL object is usually 8 * number of elements since 8 bytes are required for each double. | ||
| - | * To bring a whole TOOL object into memory to cover that size (we don't always bring the full thing into memory though!) | + | * To bring a whole TOOL object into memory you need to ensure that you have enough memory available on your machine to handle the size of your biggest TOOL object (although TOOL usually does not always bring the full object into memory as seen in the examples below) |
| * A TOOL object always refers to a file which is the compressed size of the TOOL object (compression depends on the #0's in the object) | * A TOOL object always refers to a file which is the compressed size of the TOOL object (compression depends on the #0's in the object) | ||
| * The maximum signed integer is 2^31 = 2147483648 | * The maximum signed integer is 2^31 = 2147483648 | ||
| Line 20: | Line 20: | ||
| These are the limitations we have until we move to 64bit machines and optimize the TOOL language to use it. | These are the limitations we have until we move to 64bit machines and optimize the TOOL language to use it. | ||
| - | Any tool will need memory for all it's inputs AND outputs but not always enough for the full objects to be in memory. How much of the objects are pulled into memory depends on how the objects are ordered. Read on! | + | Any tool will need memory for all it's inputs AND outputs but not always enough for the full objects to be in memory. How much of each object that is pulled into memory depends on how the objects are ordered. Read on! |
| ===== Specificl TOOL Performance Tuning ===== | ===== Specificl TOOL Performance Tuning ===== | ||
| Line 31: | Line 31: | ||
| The input object is A[a,b,c,d]: | The input object is A[a,b,c,d]: | ||
| - | If the key dimension is d the tool will bring in all of the elements contained in the object d into memory, which can be shown as y = extent(d). It will then process that section and then move on to the next section also size extent(d). This will be repeated x times where x = extent(a) * extent(b) * extent(c). | + | If the key dimension is d, the TOOL will bring in all of the elements contained in the dimension d into memory. The memory requirement can be written as memory = extent[d]. It will then process that section and then move on to the next section also size extent[d]. This will be repeated x times where x = extent[a] * extent[b] * extent[c]. |
| - | If the key dimension is b the tool will bring in all of the elements contained in the objects b, c and d into memory, which can be shown as y = extent(b) * extent(c) * extent(d). It will then process that section and then move on to the next section. This will be repeated x times where x = extent(a) | + | If the key dimension is b, the TOOL will bring in all of the elements contained in the dimensions b, c and d into memory. The memory requirement can be written as memory = extent[b] * extent[c] * extent[d]. It will then process that section and then move on to the next section. This will be repeated x times where x = extent[a] |
| === Case Study: dimension mapping of 116M elem object === | === Case Study: dimension mapping of 116M elem object === | ||
| Line 96: | Line 96: | ||
| ==== multiply, divide, add, subtract, (2 input tools) ==== | ==== multiply, divide, add, subtract, (2 input tools) ==== | ||
| - | When looking at the inputs for the multiply, divide, add subtract tools, tool examines the dimensions from left to right. If the dimensions are matching (i.e. a = a) tool will loop over these dimensions until tool finds a non-matching dimension. Once a non-matching dimension is found all of the dimensions including and after this dimension need to be taken into memory. | + | When looking at the inputs for the multiply, divide, add subtract tools, TOOL examines the dimensions from left to right. If the dimensions are matching (i.e. a = a) TOOL will loop over these dimensions until tool finds a non-matching dimension. Once a non-matching dimension is found all of the dimensions including and after this dimension need to be taken into memory. |
| === Example === | === Example === | ||
| Line 102: | Line 102: | ||
| The input objects are A[a,b,c,d] and B[e,f,a,b]: | The input objects are A[a,b,c,d] and B[e,f,a,b]: | ||
| - | If the input objects are used as they are shown above, tool will bring in all of the elements contained in the objects a, b, c, d, e and f into memory, which can be shown as y = extent(a) * extent(b) * extent(c) * extent(d) * extent(e) * extent(f) and then perform the requested operation. | + | If the input objects are used as they are shown above, TOOL will bring in all of the elements contained in the dimensions a, b, c, d, e and f into memory. The memory requirement can be written as memory = extent[a] * extent[b] * extent[c] * extent[d] * extent[e] * extent[f]. |
| - | However, if we reorder the dimensions of variable B to look like B[a,b,e,f] then tool will bring in the all of the elements contained in the object c, d, e and f into memory, which can be shown as y = extent(c) * extent(d) * extent(e) * extent(f). This will be repeated x times where x = extent(a) * extent(b). | + | However, if we reorder the dimensions of object B to look like B[a,b,e,f] then TOOL will bring in the all of the elements contained in the dimensions c, d, e and f into memory. The memory requirement can be written as memory = extent[c] * extent[d] * extent[e] * extent[f]. This will be repeated x times where x = extent[a] * extent[b]. |
| ==== insert ==== | ==== insert ==== | ||
| - | FIXME | ||
| - | -- These tools require bringing the WHOLE input and output object into memory | + | Insert is a unique tool because the shape of the input object and output object need to be the same to ensure that you are inserting apples into apples. Because of this TOOL reads the dimensions of both the input and output into memory. |
| + | |||
| + | === Example === | ||
| + | |||
| + | The input object is A[a,b] and the output object is B[c,d] | ||
| + | |||
| + | TOOL will bring in all of the elements contained in the dimensions a, b, c, and d into memory. The memory requirement can be written as memory = extent[a] * extent[b] * extent[c] * extent[d] | ||
howtos/toolcoding/performance_tuning.1308767511.txt.gz · Last modified: 2011/06/22 18:31 by chris.strashok
Page Tools
