Differences

This shows you the differences between two versions of the page.

--- howtos:toolcoding:performance_tuning [2010/04/01 15:15]
shona.weldon
+++ howtos:toolcoding:performance_tuning [2011/06/22 21:01] (current)
chris.strashok
@@ Line 13: / Line 13: @@
 Facts for a 32bit machine:
   * The actual size of a TOOL object is usually 8 * number of elements since 8 bytes are required for each double.
-    * To bring a whole TOOL object into memory to cover that size (we don't always bring the full thing into memory though!)
+    * To bring a whole TOOL object into memory you need to ensure that you have enough memory available on your machine to handle the size of your biggest TOOL object (although TOOL usually does not always bring the full object into memory as seen in the examples below)
     * A TOOL object always refers to a file which is the compressed size of the TOOL object (compression depends on the #0's in the object)
   * The maximum signed integer is 2^31 = 2147483648
@@ Line 20: / Line 20: @@
 These are the limitations we have until we move to 64bit machines and optimize the TOOL language to use it.
-Any tool will need memory for all it's inputs AND outputs but not always enough for the full objects to be in memory.  How much of the objects are pulled into memory depends on how the objects are ordered.  Read on!
+Any tool will need memory for all it's inputs AND outputs but not always enough for the full objects to be in memory.  How much of each object that is pulled into memory depends on how the objects are ordered.  Read on!
 ===== Specificl TOOL Performance Tuning =====
 ==== map, mapcat, sum, (single input tools with key dimension) ====
-These tools can have the "key" dimension usually specified with //dim=// anywhere in the object but the further to the right it is the less of the input objects will have to be in memory at a time.
+These tools can have the "key" dimension usually specified with //dim=// anywhere in the object, however the further to the right this dimension is the fewer of the input's objects that will have to be read into memory.
-**For example:** if you have an object A[a,b,c,d]
+=== Example ===
-If key dimension is d the tool will bring in y elements of the object into memory at a time where y = extent(d). It will then process that section and then move on to the next section also size extent(d). This will be repeated x times where x = extent(a) * extent(b) * extent(c).
+The input object is A[a,b,c,d]:
+If the key dimension is d, the TOOL will bring in all of the elements contained in the dimension d into memory. The memory requirement can be written as memory = extent[d]. It will then process that section and then move on to the next section also size extent[d]. This will be repeated x times where x = extent[a] * extent[b] * extent[c].
+If the key dimension is b, the TOOL will bring in all of the elements contained in the dimensions b, c and d into memory.  The memory requirement can be written as memory = extent[b] * extent[c] * extent[d]. It will then process that section and then move on to the next section. This will be repeated x times where x = extent[a]
+=== Case Study: dimension mapping of 116M elem object ===
+Heres a real world example and a test that was done.
+Input object:
+<code>
+tmp1[]: array of real numbers, single precision
+ version: 3
+ desc:    local tmp1[] =
+ dim1:    SET; stateLOT; 8
+ dim2:    SET; SD96; 58
+ dim3:    SET; lndCByIrr; 2
+ dim4:    CAT; agrCACat: agrRegType.crop; 48
+ dim5:    SEQ; yearbuilt: 1856:2001:5; year
+ dim6:    SET; agrCondType; 3
+ dim7:    SEQ; time: 1861:2001:5; year
+ units:   tonne / hectare
+   scientific measure: tonne / hectare
+   SI signature:       m^-2 kg
+ data:    116259840 elements, empty
+</code>
+Final output object:
+<code>
+tmp2[]: array of real numbers, single precision
+ version: 3
+ desc:    local tmp2[] = map (tmp1[];stateLOT->state)
+ dim1:    SET; state; 9
+ dim2:    SET; SD96; 58
+ dim3:    SET; lndCByIrr; 2
+ dim4:    CAT; agrCACat: agrRegType.crop; 48
+ dim5:    SEQ; yearbuilt: 1856:2001:5; year
+ dim6:    SET; agrCondType; 3
+ dim7:    SEQ; time: 1861:2001:5; year
+ units:   tonne / hectare
+   scientific measure: tonne / hectare
+   SI signature:       m^-2 kg
+ data:    130792320 elements, empty
+</code>
+**Test1** - map the first dimension (stateLOT) using a dimension mapping:
+<code>
+local tmp2[] = map (tmp1[]; stateLOT->state)
+</code>
+**Test2** - reorder the stateLOT dimension to the far right, do the dimension mapping, and reorder back:
+<code>
+local tmp1Reord[] = reorder (tmp1[]; 2,3,4,5,6,7,1)
+local tmp2Reord[] = map (tmp1Reord[]; stateLOT->state)
+local tmp3Reord[] = reorder (tmp2Reord[]; 7,1,2,3,4,5,6)
+</code>
+**Runtime comparisons**:
+  * Test1 - completed in 11 min 23 sec
+  * Test2 - completed in 2 min 40 sec
+    * first reorder - 23 sec
+    * map call - 25 sec
+    * second reorder - 1 min 52 sec
-If key dimension is b the tool will bring in y elements of the object into memory at a time where y = extent(b) * extent(c) * extent(d). It will then process that section and then move on to the next section. This will be repeated x times where x = extent(a)
 ==== multiply, divide, add, subtract, (2 input tools) ====
-FIXME
--- Again order matters
+When looking at the inputs for the multiply, divide, add subtract tools, TOOL examines the dimensions from left to right. If the dimensions are matching (i.e. a = a) TOOL will loop over these dimensions until tool finds a non-matching dimension. Once a non-matching dimension is found all of the dimensions including and after this dimension need to be taken into memory.
+=== Example ===
+The input objects are A[a,b,c,d] and B[e,f,a,b]:
+If the input objects are used as they are shown above, TOOL will bring in all of the elements contained in the dimensions a, b, c, d, e and f into memory. The memory requirement can be written as memory = extent[a] * extent[b] * extent[c] * extent[d] * extent[e] * extent[f].
+However, if we reorder the dimensions of object B to look like B[a,b,e,f] then TOOL will bring in the all of the elements contained in the dimensions c, d, e and f into memory. The memory requirement can be written as memory = extent[c] * extent[d] * extent[e] * extent[f]. This will be repeated x times where x = extent[a] * extent[b].
+==== insert ====
+Insert is a unique tool because the shape of the input object and output object need to be the same to ensure that you are inserting apples into apples. Because of this TOOL reads the dimensions of both the input and output into memory.
--- Basically if there are matching dimensions keep them together in the LEFT most postions
+=== Example ===
-==== insert, reorder ====
+The input object is A[a,b] and the output object is B[c,d]
-FIXME
--- These tools require bringing the WHOLE input and output object into memory
+TOOL will bring in all of the elements contained in the dimensions a, b, c, and d into memory. The memory requirement can be written as  memory = extent[a] * extent[b] * extent[c] * extent[d]

whatIf? Wiki

User Tools

Site Tools

Differences

Page Tools