26

Feb

2015

How to retrieve output keywords from Call_Procedure

Author: Brian Griglak

When working in IDL, often it will seem like doing something is just not possible. That is, until you talk to the right person who knows language tricks you never would have dreamed of. In my case that happened while I was working on our new ENVITask subclass for wrapping any IDL procedure. I wanted to use Call_Procedure to invoke it, but could not figure out how to get the values of output keywords from the procedure, so I had to resort to using the much more flexible but slower Execute() approach. Then I had a conversation with Adam Lefkoff, creator of the original version of ENVI and all around IDL guru, and he showed me a trick to take advantage of that works with Call_Procedure.

Let me step back a bit and describe the problem better before I demonstrate the solution. Let's say there is an IDL procedure that you want to call, but you have the names of the keywords as string literals instead of having a priori knowledge of them at compile time. Perhaps they come from a database or external source. You have two options for invoking this procedure: Execute and Call_Procedure:

pro testProcedure, FOO=foo, BAR=bar, _REF_EXTRA=refExtra

  compile_opt idl2

 

  help, foo, OUTPUT=fooOut

  help, bar, OUTPUT=barOut

  print, fooOut, barOut

 

  if (ISA(refExtra)) then begin

    foreach keyword, refExtra do begin

      value = Scope_VarFetch(keyword, /REF_EXTRA)

      help, value, OUTPUT=valueOut

      print, keyword, valueOut

    endforeach

  endif

end

 

pro Execute_Wrapper, procName, _REF_EXTRA=refExtra

  compile_opt idl2

 

  params = Hash()

 

  foreach keyword, refExtra do begin

    params[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

  endforeach

 

  print, 'in Execute_Wrapper'

  command = procName + ', _EXTRA=params.ToStruct()'

  ret = Execute(command)

end

 

pro Call_Procedure_Wrapper, procName, _REF_EXTRA=refExtra

  compile_opt idl2

 

  params = Hash()

 

  foreach keyword, refExtra do begin

    params[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

  endforeach

 

  print, 'in Call_Procedure_Wrapper'

  Call_Procedure, procName, _EXTRA=params.ToStruct()

end

 

pro test_procedure_wrapping

  compile_opt idl2

 

  Execute_Wrapper, 'testProcedure', FOO='foo', BAR=!pi, BAZ=List(1,2), QUX=Hash(1,2)

  Call_Procedure_Wrapper, 'testProcedure', FOO='foo', BAR=!pi, BAZ=List(1,2), QUX=Hash(1,2)

end

The example might be a bit contrived, but it demonstrates how you Scope_VarFetch() each of the values in _REF_EXTRA and put them into a Hash object. The Hash is then converted to a struct when invoking the procedure, and by assigning that struct to the _EXTRA keyword it will connect all the keywords appropriately. We can see from the output that both Execute and Call_Procedure work:

in Execute_Wrapper

FOO             STRING    = 'foo'

BAR             FLOAT     =       3.14159

BAZ VALUE           LIST  <ID=1  NELEMENTS=2>

QUX VALUE           HASH  <ID=6  NELEMENTS=1>

in Call_Procedure_Wrapper

FOO             STRING    = 'foo'

BAR             FLOAT     =       3.14159

BAZ VALUE           LIST  <ID=31  NELEMENTS=2>

QUX VALUE           HASH  <ID=36  NELEMENTS=1>

The problem is that the procedure only has input keywords, no output keywords. If we want to be able to dynamically invoke a procedure that has output keywords, a little more effort is required. Here is the updated version of the Execute wrapper in action:

 

pro testProcedure, FOO=foo, BAR=bar, OUT_HASH=outHash, _REF_EXTRA=refExtra

  compile_opt idl2

 

  outHash = Hash('FOO', foo, 'BAR', bar)

 

  if (ISA(refExtra)) then begin

    foreach keyword, refExtra do begin

      outHash[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

    endforeach

  endif

end

 

pro Execute_Wrapper, procName, OUT_NAMES=outNames, _REF_EXTRA=refExtra

  compile_opt idl2

 

  params = Hash()

 

  foreach keyword, refExtra do begin

    if (outNames.HasValue(keyword)) then continue

    params[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

  endforeach

 

  print, 'in Execute_Wrapper'

  command = procName + ', _EXTRA=params.ToStruct()'

  foreach name, outNames do begin

    command += ', ' + name.ToUpper() + '=' + name.ToLower()

  endforeach

  ret = Execute(command)

 

  foreach name, outNames do begin

    (Scope_VarFetch(name, /REF_EXTRA)) = Scope_VarFetch(name)

  endforeach

end

 

pro test_procedure_wrapping

  compile_opt idl2

 

  execOutHash = 0

  Execute_Wrapper, 'testProcedure', FOO='foo', BAR=!pi, BAZ=List(1,2), QUX=Hash(1,2), OUT_NAMES='OUT_HASH', OUT_HASH=execOutHash

  print, execOutHash, /IMPLIED

end

The Execute_Wrapper routine now requires knowledge of which keywords in _REF_EXTRA are input and which are output. So when we construct the Hash we omit the output keywords, but then append them to the command string  as new variables in the local scope. After invoking the procedure, these local variables will have the correct output values, but we need the very odd looking line with two calls to Scope_VarFetch() to copy them into the _REF_EXTRA bag to get them back out to the routine that called Execute_Wrapper. The Scope_VarFetch() call on the left uses the /REF_EXTRA keyword to connect with the callstack, but we need to encapsulate it in parentheses to make it an assignment operation. The Scope_VarFetch() call on the right is used to get the value of the named local variable for this assignment. We also need to make sure that the OUT_HASH keyword is present in the call to Execute_Wrapper so that this _REF_EXTRA trick will work.

While this is functional, it looks a little complicated and redundant with the OUT_HASH keyword repeated as a string literal. The other problem is that it is still using Execute(), which isn't as efficient as Call_Procedure. But we can't modify the Call_Procedure_Wrapper routine in the same manner and have success. The reason for this is that when the struct is built to pass into Call_Procedure it contains copies of the values of each keywords, not references to the variables used to build it. So we need to introduce a new function to get the output keyword values:

pro testProcedure, FOO=foo, BAR=bar, OUT_HASH=outHash, _REF_EXTRA=refExtra

  compile_opt idl2

 

  outHash = Hash('FOO', foo, 'BAR', bar)

 

  if (ISA(refExtra)) then begin

    foreach keyword, refExtra do begin

      outHash[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

    endforeach

  endif

end

 

function Procedure_Wrapper, procName, _REF_EXTRA=refExtra

  compile_opt idl2

 

  Call_Procedure, procName, _EXTRA=refExtra

 

  results = Hash()

  foreach keyword, refExtra do begin

    results[keyword] = Scope_Varfetch(keyword, /REF_EXTRA)

  endforeach

 

  return, results

end

 

pro Call_Procedure_Wrapper, procName, OUT_NAMES=outNames, _REF_EXTRA=refExtra

  compile_opt idl2

 

  params = Hash()

 

  foreach keyword, refExtra do begin

    params[keyword] = Scope_VarFetch(keyword, /REF_EXTRA)

  endforeach

 

  print, 'in Call_Procedure_Wrapper'

  results = Procedure_Wrapper(procName, _EXTRA=params.ToStruct())

 

  foreach name, outNames do begin

    (Scope_VarFetch(name, /REF_EXTRA)) = results[name]

  endforeach

end

 

pro test_procedure_wrapping

  compile_opt idl2

 

  callOutHash = 0

  Call_Procedure_Wrapper, 'testProcedure', FOO='foo', BAR=!pi, BAZ=List(1,2), QUX=Hash(1,2), OUT_NAMES='OUT_HASH', OUT_HASH=callOutHash

  print, callOutHash, /IMPLIED

end

This new function behaves similarly to the modified Execute_Wrapper routine, but since it passes all of its keywords through to Call_Procedure it uses references not values. It can then use Scope_VarFetch(/REF_EXTRA) to copy the values from those keywords into a Hash that it returns to its caller. In there we again use Scope_VarFetch(/REF_EXTRA) to copy the output values from that Hash into the variables from its calling routine. The use of the OUT_NAMES keyword is not completely necessary, but it avoids copying input variables back to the caller, which may or may not be expensive.

In the ENVITask context, we know which keywords are inputs and which are outputs, so instead of passing in the OUT_NAMES keyword and returning a Hash of value we can make Procedure_Wrapper a member function and set the output parameter values directly. But as you'll see in ENVI 5.2 SP1 we can wrap any IDL procedure that uses keywords in an ENVITaskFromProcedure object that knows how to map input and output keywords to ENVITaskParameters for you automatically.

Comments (0) Number of views (138) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

19

Feb

2015

String processing performance in IDL

Author: Atle Borsholm

IDL performs array based operations very efficiently, but most processing tasks do require some amount of string parsing and manipulation. I have selected 3 common string processing tasks to analyze in more depth in order to find the best string processing strategies in each of these cases. The first example is to find all the strings that start with a given substring. IDL 8.4 has many new intrinsic methods for string type variables, and one of them is "StartsWith". Here is the code I used to compare 4 different approaches to find out which strings in a string array starts with the word "end".

pro StrTest_StartsWith

 compile_opt idl2,logical_predicate

 

 f = file_which('amoeba.pro')

 str = strarr(file_lines(f))

 openr, lun, f, /get_lun

 readf, lun, str

 free_lun, lun

 

 first = str.StartsWith('end')

 n = 50000

 times = dblarr(4)

 methods = ['StartsWith','STRCMP','STREGEX','STRPOS']

 for method=0,3 do begin

   t0 = tic()

   case method of

   0: for i=0, n-1 do x = str.StartsWith('end')

   1: for i=0, n-1 do x = strcmp(str,'end',3)

   2: for i=0, n-1 do x = stregex(str,'^end',/boolean)

   3: for i=0, n-1 do x = strpos(str,'end') eq 0

   endcase

   times[method] = toc(t0)

   print, array_equal(x,first) ? 'Same answer' : 'Different answer'

 endfor

 print, string(methods[sort(times)] + ':', format='(a-15)') + $

   string(times[sort(times)], format='(g0)'), $

   format='(a)'

end

The first method is to use the new intrinsic "StartsWith" method, the next is to use STRCMP with a 3rd argument specifying how many characters to compare. The third method uses a regular expression with STREGEX, and the final method uses STRPOS and compare the result to 0, meaning the pattern was found starting at position 0. The result I get when I run this code in IDL 8.4 is:

Same answer

Same answer

Same answer

Same answer

STRCMP:        0.128

StartsWith:    0.147

STRPOS:        0.91

STREGEX:      1.497

All methods return a byte array of zeros and ones indicating where the matches are. STRCMP with 3 arguments ended up being the fastest, with the new "StartsWith" method being a close second. STREGEX should be avoided unless it is really needed for a more complex expression.

In this second example, the goal is to replace the first occurrence of an equal sign (=) with a color (:) on every line that contains at least one equal (=) sign. If there are additional equal signs, they should remain unchanged. This is mostly useful for converting the format of name/value pairs stored in a text file. I used 4 different methods to achieve the same result:

pro StrTest_Substring

 compile_opt idl2,logical_predicate

 

 f = file_which('amoeba.pro')

 str = strarr(file_lines(f))

 openr, lun, f, /get_lun

 readf, lun, str

 free_lun, lun

 

 n = 2000

 index = str.IndexOf('=')

 w = where(index ne -1)

 index = index[w]

 first = str

 first[w] = str[w].Substring(0,index-1)+':'+str[w].Substring(index+1)

 methods = ['Substring','STRPUT','Split/Join','BYTARR']

 times = dblarr(4)

 for method=0,3 do begin

   t0 = tic()

   case method of

     0: for i=0, n-1 do begin

       index = str.IndexOf('=')

       w = where(index ne -1)

       index = index[w]

       y = str[w]

       x = str

       x[w] = y.SubString(0,index-1)+':'+y.SubString(index+1)

     endfor

     1: for i=0, n-1 do begin

       x = str

       pos = strpos(str,'=')

       foreach xx, x, j do begin

          if pos[j] ne -1 then begin

            strput, xx, ':', pos[j]

            x[j] = xx

          endif

       endforeach

     endfor

     2: for i=0, n-1 do begin

       x = str

       foreach xx, x, j do begin

          parts = xx.Split('=')

          if parts.length gt 1 then x[j] = ([parts[0],parts[1:*].join('=')]).join(':')

       endforeach

     endfor

     3: for i=0, n-1 do begin

       b = byte(str)

       b[maxInd[where(max(b eq 61b, dimension=1, maxInd))]] = 58b

       x = string(b)

     endfor

   endcase

   times[method] = toc(t0)

   print, array_equal(x,first) ? 'Same answer' : 'Different answer'

 endfor

 print, string(methods[sort(times)] + ':', format='(a-15)') + $

   string(times[sort(times)], format='(g0)'), $

   format='(a)'

 

end

Same answer

Same answer

Same answer

Same answer

BYTARR:        0.148

STRPUT:        0.187

Substring:     0.188

Split/Join:   1.456

The cryptic byte array method ended up being the fastest, even though it does perform a lot of copying, and doesn't contain any obvious string processing functions. This is because IDL can run operations on arrays very efficiently to speed up the computations. For example, the internal array indexing gives good predictable memory access patterns. However, I would not really recommend using this approach here, since the code is very hard to understand, and to modify if needed. I would also avoid using the SPLIT/JOIN approach as that is very inefficient. Using "IndexOf" and "Substring" is nice here, especially notice that the "Substring" method is similar to STRMID, but can handle an array of different positions matching the size of the string array. This is a significant improvement over the old STRMID. For example, to extract the beginnings of every string up and including the first "e", you could use:

IDL> a=['!Hello!', 'test','this one!']

IDL> a.Substring(0,a.IndexOf('e'))

!He

te

this one

Or, to extract the characters after the first colon:

IDL> x = ((orderedhash(!cpu))._overloadPrint())

IDL> x

HW_VECTOR:            0

VECTOR_ENABLE:            0

HW_NCPU:            6

TPOOL_NTHREADS:            6

TPOOL_MIN_ELTS:                 100000

TPOOL_MAX_ELTS:                      0

IDL> x.Substring(x.IndexOf(':'))

:            0

:            0

:            6

:            6

:                 100000

:                     0

The final example is replacing every occurrence of = with =>. I used 2 different methods for this, using the new "Replace"method on string types, and using STRSPLIT/STRJOIN. The results show that the new Replace method is much more efficient.

pro StrTest_Replace

 compile_opt idl2,logical_predicate

 

  f = file_which('amoeba.pro')

 str = strarr(file_lines(f))

 openr, lun, f, /get_lun

 readf, lun, str

 free_lun, lun

 

 n = 5000

 first = str.Replace('=', '=>')

 methods = ['Replace','STRSPLIT']

 times = dblarr(2)

 for method=0,1 do begin

   t0 = tic()

   case method of

     0: for i=0, n-1 do begin

       x = str.Replace('=','=>')

     endfor

     1: for i=0, n-1 do begin

       x = str

       foreach xx, x, j do x[j] = strjoin(strsplit(xx,'=',/extract),'=>')

     endfor

   endcase

   times[method] = toc(t0)

   print, array_equal(x,first) ? 'Same answer' : 'Different answer'

 endfor

 print, string(methods[sort(times)] + ':', format='(a-15)') + $

   string(times[sort(times)], format='(g0)'), $

   format='(a)'

end

Same answer

Same answer

Replace:       0.545

STRSPLIT:     2.778

Comments (0) Number of views (216) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

12

Feb

2015

Data Structure Analysis

Author: Dain Cilke

One of the main questions anybody using a programming language has to ask themselves is "what data structures should I be using?" This can be a complicated and difficult question as there are many trade-offs to consider. If it is desired to have a dynamic data type, IDL provides multiple options. In this analysis we will consider: dynamic arrays, lists, hashes, and ordered hashes and their ability to insert and delete elements. To start, let's review our data structures. Dynamic arrays are based on the ability for IDL arrays to resize themselves. For example:

array = [1,2,3,4]               ; Declaration

array = [array, 5]              ; Insert

array = [array[0:1],array[3:*]] ; Remove

 

Lists use the IDL object LIST:

list = LIST(1,2,3,4)    ; Declaration

list.add,5              ; Insert

list.remove, 2          ; Remove

 

Hashes use the IDL object HASH:

hash = HASH([1,2,3,4],[1,2,3,4]) ; Declaration

hash[5] = 5                      ; Insert

hash.remove, 2                   ; Remove

 

Ordered hashes use the IDL object ORDEREDHASH:

ohash = ORDEREDHASH([1,2,3,4],[1,2,3,4]) ; Declaration

ohash[5] = 5                             ; Insert

ohash.remove, 2                          ; Remove

 

For each data structure we will time how long it takes to insert and remove n elements (Note: the inserts/removals are done inside of a FOR loop, one insert/removal per iteration. This is done to simulate an application which expects a high degree of volatility in the use of their data structures. However, since IDL is a vectorized language, it is always best to try to group multiple operations into a single call). Please see the attached plots for the results of the runs. The results are what we would expect from a simple big-O analysis. Dynamic arrays are comparable for small input sizes, however, as soon as the size of the input grows, it becomes much faster to use a hash (either type) or a list. In terms of pure speed for any arbitrary input size, list  is the fastest. However, if you know your input bounded to a few elements, all of the proposed data structures can offer a similar performance.

Note: For this analysis, all the data structures had similar performance up to 10,000 elements. This is in no way a comprehensive test and the results may differ on your system. However, the rule of thumb I follow is, if your input is less than 10,000 elements choose the data structure which is the easiest for you to work with.

Comments (0) Number of views (332) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

5

Feb

2015

New Features Coming Soon to IDL - With a Few Brief Examples

Author: Benjamin Foreback

IDL 8.4.1 is scheduled to release within the next few weeks. Although this is only a point release rather than a full new version, there are a few features that the IDL team has implemented that are very exciting to me.

HDF and NetCDF Updates

When working with scientific data, I often find myself using IDL to open and read HDF4 and HDF5 files. Although the design of HDF5 is quite a bit simpler than HDF4, working with the API for both formats can sometimes be rather painful, and I almost always need to reference the documentation examples just to get started. 

One of my favorite routines in IDL is H5_PARSE (which is a function in the IDL library and not part of the standard HDF5 interface). The reason I like this function so much is that I can use it to query datasets and attributes in the file without needing to write multiple lines of code to search for and access the desired information (and I can't forget to ensure the file gets closed even if an error occurs). Everything I want is simply dumped into a hierarchical structure.

My one complaint, however, about H5_PARSE is that it can be rather slow, especially with large files. The reason for this is that the structure is created dynamically as IDL descends through the file, using this logic:

sTree = CREATE_STRUCT(sTree, tagname, sMember)

Just like how appending an array is very inefficient, creating a structure in this fashion is slow and memory intensive, particularly when using the /READ_DATA keyword.

Luckily, IDL’s new OrderedHash data type provides a handy alternative; adding fields to a hash is much more efficient than appending a structure. Additionally, an ordered hash not only preserves the order, it also preserves names and cases (structures are not case-sensitive and special characters and spaces in object names must be converted to underscores to be valid structure tags).

Furthermore, what is cool about an ordered hash is that you can use Implied Print, and IDL will print out the file’s information in a hierarchical JSON format, mimicking the structure of the file. To use an ordered hash in place of a structure, use the /ORDEREDHASH keyword when calling H5_PARSE. Here is an example using an HDF5-EOS file:

file = 'C:\hdf5\hdfeos\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5'
result = H5_PARSE(file, /ORDEREDHASH)
result

IDL will print:

{
    "_NAME": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
    "_ICONTYPE": "hdf",
    "_TYPE": "GROUP",
    "_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
    "_PATH": "/",
    "_COMMENT": "",
    "HDFEOS": {
        "_NAME": "HDFEOS",
        "_ICONTYPE": "",
        "_TYPE": "GROUP",
        "_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
        "_PATH": "/",
        "_COMMENT": "",
        "ADDITIONAL": {
            "_NAME": "ADDITIONAL",
            "_ICONTYPE": "",
            "_TYPE": "GROUP",
            "_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
            "_PATH": "/HDFEOS",
            "_COMMENT": "",
            "FILE_ATTRIBUTES": {
                "_NAME": "FILE_ATTRIBUTES",
                "_ICONTYPE": "",
                "_TYPE": "GROUP",
                "_FILE": "C:\\hdf5\\hdfeos\\OMI-Aura_L2-OMNO2_2008m0720t2016-o21357_v003-2008m0721t101450.he5",
                "_PATH": "/HDFEOS/ADDITIONAL",
                "_COMMENT": "",
                "InstrumentName": {
                    "_NAME": "InstrumentName",
                    "_ICONTYPE": "text",
                    "_TYPE": "ATTRIBUTE",
                    "_NDIMENSIONS": 0,
                    "_DIMENSIONS": 0,
                    "_NELEMENTS": 1,
                    "_DATATYPE": "H5T_STRING",
                    "_STORAGESIZE": 3,
                    "_PRECISION": 24,
                    "_SIGN": "",
                    "_DATA": "OMI"


etc.

Another new function that IDL 8.4.1 will offer is HDF_PARSE. This is the equivalent to H5_PARSE but for HDF4 files. It will walk through a file and return information about groups, scientific datasets, images, palettes, annotations, and VData. This information will be returned in an ordered hash, following the same structure that the information is stored in the file. HDF_PARSE will always return an ordered hash; if you would like a structure instead, you can convert the hash to a structure using OrderedHash::ToStruct. Here is an example using a MODIS file:

file = 'C:\scratch\modisl2\MOD10_L2.A2002013.0930.003.2002017035237.hdf'
result = HDF_PARSE(file)
result

IDL prints:

{
    "_NAME": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
    "_TYPE": "GROUP",
    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
    "_PATH": "/",
    "MOD_Swath_Snow": {
        "_NAME": "MOD_Swath_Snow",
        "_TYPE": "GROUP",
        "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
        "_PATH": "/",
        "Geolocation Fields": {
            "_NAME": "Geolocation Fields",
            "_TYPE": "GROUP",
            "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
            "_PATH": "/MOD_Swath_Snow",
            "Latitude": {
                "_NAME": "Latitude",
                "_TYPE": "SD",
                "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                "_PATH": "/MOD_Swath_Snow/Geolocation Fields",
                "_NDIMENSIONS": 2,
                "_DIMENSIONS": [271, 406],
                "_IDL_DATATYPE": "FLOAT",
                "_DATA": "<unread>",
                "long_name": {
                    "_NAME": "long_name",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "STRING",
                    "_DATA": "Coarse 5 km resolution latitude"
                },
                "units": {
                    "_NAME": "units",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "STRING",
                    "_DATA": "degrees"
                },
                "valid_range": {
                    "_NAME": "valid_range",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "FLOAT",
                    "_DATA": [-90.000000, 90.000000]
                },
                "_FillValue": {
                    "_NAME": "_FillValue",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "FLOAT",
                    "_DATA": [-999.00000]
                },
                "source": {
                    "_NAME": "source",
                    "_TYPE": "ATTRIBUTE",
                    "_FILE": "C:\\scratch\\modisl2\\MOD10_L2.A2002013.0930.003.2002017035237.hdf",
                    "_PATH": "/MOD_Swath_Snow/Geolocation Fields/Latitude",
                    "_IDL_DATATYPE": "STRING",
                    "_DATA": "MOD03 geolocation product; data read from center pixel in 5 km box"
                }
            },
            "Longitude": {

etc.

IDL 8.4.1 will also include new NetCDF routines for querying, retrieving and creating NetCDF files. These routines include NCDF_GET, NCDF_LIST, and NCDF_PUT

Overloading the ++ and -- Operators

I have always liked IDL’s ability to overload operators of classes that inherit IDL_Object. This gives me the ability to create objects that behave like their own data types.

One set of operators that couldn't be overloaded until now is the increment/decrement operator, ++/--.  I like this operator because not only is it easier to type var++ than var = var + 1, it is more efficient. There are also cases where “increment” has a meaning other than “add one,” such as if the value is not numeric. Here is an example of an alphabet class that increments through letters (all lower-case). Once the value reaches the letter “z,” the next value in the sequence will be “aa.” When it reaches “az,” incrementing it again will change the value to “ba,” etc.

pro SequentialLetters::_OverloadPlusPlus
  compile_opt idl2, hidden
  
  if (StrCmp(self.value, '')) then begin
    self.value = 'a'
    return
  endif
  
  byteValue = Byte(self.value)
  if (byteValue[-1] eq 122) then begin
    if (StrLen(self.value) eq 1) then begin
      self.value = 'aa'
    endif else begin
      ; Create a new SequentialLetters object to recursively handle all but the 
      ; last letter.
      newObj = Obj_New('SequentialLetters', VALUE=StrMid(self.value, 0, StrLen(self.value)-1))
      newObj++
      self.value = newObj.VALUE + 'a'
      Obj_Destroy, newObj
    endelse
  endif else begin
    byteValue[-1]++
    self.value = String(byteValue)
  endelse
  
end

;-------------------------------------------------------------------------------

pro SequentialLetters::_OverloadMinusMinus
  compile_opt idl2, hidden

  ; If there is only one character and it is 'a', then the value becomes
  ; an empty string.
  if (StrCmp(self.value, 'a')) then begin
    self.value = ''
    return
  endif
  
  byteValue = Byte(self.value)
  if (byteValue[-1] eq 97) then begin
    ; Create a new SequentialLetters object to recursively handle all but the 
    ; last letter.
    newObj = Obj_New('SequentialLetters', VALUE=StrMid(self.value, 0, StrLen(self.value)-1))
    newObj--
    self.value = newObj.VALUE + 'z'
    Obj_Destroy, newObj
  endif else begin
    byteValue[-1]--
    self.value = String(byteValue)
  endelse

end

;-------------------------------------------------------------------------------

pro SequentialLetters::SetProperty, VALUE=value, _REF_EXTRA=extra
  compile_opt idl2
  
  if (N_Elements(value) gt 0) then begin
    self.value = value
  endif
  
  if (ISA(extra)) then begin
    self.IDL_Object::SetProperty, _EXTRA=extra
  endif

end

;-------------------------------------------------------------------------------

pro SequentialLetters::GetProperty, VALUE=value, _REF_EXTRA=extra
  compile_opt idl2

  if (Arg_Present(value)) then begin
    value = self.value
  endif

  if (ISA(extra)) then begin
    self.IDL_Object::GetProperty, _EXTRA=extra
  endif

end

;-------------------------------------------------------------------------------


function SequentialLetters::Init, _REF_EXTRA=extra
  compile_opt idl2
  
  if (~self.IDL_Object::Init()) then return, 0
  
  if (ISA(extra)) then begin
    self.SetProperty, _EXTRA=extra
  endif
  
  return, 1
  
end

;-------------------------------------------------------------------------------

pro SequentialLetters__Define
  compile_opt idl2
  
  !null = {SequentialLetters, $
           inherits IDL_Object, $
           value: ''}

end

Here is an example of using this object:

myLetters = Obj_New('SequentialLetters', VALUE='z')
myLetters++
print, myLetters.VALUE

IDL prints:
aa

myLetters--
print, myLetters.VALUE

IDL prints:
z

Other Updates

A full list of new features and product improvements in IDL 8.4.1 will be published soon. Stay tuned!

Comments (0) Number of views (467) Article rating: No rating

Categories: IDL Blog | IDL Data Point

Tags:

3

Feb

2015

Queuing ESE Tasks in a Loop

Author: Barrett Sather

When using desktop ENVI and IDL, it is useful to setup processing that you have to do many times in a batch script. The easiest way to do this is with an IDL for loop that does processing on one file at a time. However, for an instance running ENVI Services Engine (ESE), it is best to call each task on an individual file. This makes the processing more robust, as an error in processing on a single file will not halt processing for every file. This leaves the question though - how would one loop over every file that needs processing?

One solution is to use IDL to create a loop that goes over each file, then launch every task that needs to be preformed. This builds up a list of tasks queued for execution. To do this, create a list of input files and output files much like you would in a batch process, then call the HTTP address required to submit the processing request to ESE one file at a time.

This can be done using IDL's built in HTTP client, IDLNetURL. As an example, if the ESE process to be called is an asynchronous task named "apply_color_table", the full HTTP call to start the task will be:

http://(host):8181/ese/services/AsyncService/apply_color_table/submitJob?inFile=file&outputFile=file

where (host) is the name or IP address of the server, and the keywords "file" are the actual input and output file names. One way to set up this call so that it occurs on multiple files is as follows, where inFiles is a variable containing all of the files to be processed.

 oURL = Obj_New('IDLnetUrl')

 oUrl.SetProperty, URL_SCHEME='http'

 oUrl.SetProperty, URL_HOST = !SERVER.HOSTNAME

 oUrl.SetProperty, URL_PORT='8181'

 oUrl.SetProperty, $

   URL_PATH='ese/services/AsyncService/apply_color_table/submitJob'

 foreach inFile, inFiles do begin

   oUrl.SetProperty, URL_QUERY='inputFile=' + inFile + $

     '&outputFile=' + 'ct_' + inFile

   result = oURL.Get()

   json = JSON_Parse(result)

   print, 'status file: ' + json['jobStatusURL']

 endforeach

The task that is called will contain the processing, in this case the call that would be made using IDL would be:

apply_color_table, inputFile=inputFile, outputFile=outputFile

This procedure will take in the input file and output file names as arguments, which are passed in through the queuing script.

Once the queuing script completes, ESE will begin running through the tasks one at a time, distributing the workload across CPUs and across any workers that are set up.

Comments (0) Number of views (412) Article rating: 5.0

Categories: IDL Blog | IDL Data Point

Tags:

12345678910 Last

MOST POPULAR POSTS

AUTHORS

Authors

Authors

© 2015 Exelis Visual Information Solutions