Note: For updates to third-party libraries, see the IDL Release Notes inside the IDL installation.
New Features
BZIP2 Support
IDL can now compress and uncompress files in the BZIP2 compression format. See BZIP_COMPRESS and BZIP_UNCOMPRESS for details.
Embedded Python for IDL and ENVI
IDL now ships with a fully-featured Python distribution. This embedded Python includes a range of useful packages such as numpy, pandas, regex, urllib3, h5py, and the envipyengine. You can easily install new packages into a user-specific installation folder.
The new embedded Python removes any need to configure the IDL-Python bridge. When you execute your first Python command, IDL will automatically load the embedded Python. For example:
IDL> pd = Python.Import('pandas')
% Python is embedded version 3.13.
% Loaded DLM: PYTHON313.
% PYTHON_INIT: C:\Program Files\NV5\IDL92\bin\bin.x86_64\idl-python313.
IDL> dates = pd.date_range("20130101", periods=3)
IDL> df = pd.dataframe(randomu(seed, 4, 3), index = dates, columns = ["A", "B", "C", "D"])
IDL> print, df
A B C D
2013-01-01 0.647128 0.954983 0.171173 0.664392
2013-01-02 0.251179 0.635682 0.986357 0.125033
2013-01-03 0.003626 0.229533 0.639266 0.297249
The embedded Python also comes with a new PyUtils class which lets you easily add or remove packages using familiar pip commands. For example, you can install the Beautiful Soup package, which is useful for parsing HTML and XML files:
IDL> PyUtils.PipInstall, 'beautifulsoup4'
Looking in indexes: https://pypi.python.org/simple
Collecting beautifulsoup4
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.13.3
Package beautifulsoup4 successfully installed in C:\Users\myusername\.idl\idl\python_idlx_x.
Now that it is installed we can try out the package:
IDL> bs4 = Python.Import('bs4')
IDL> myhtml = '<html><title>IDL is great</title></html>'
IDL> soup = bs4.BeautifulSoup(myhtml, 'html.parser')
IDL> print, soup.title.string
IDL is great
IDL 9.2 still ships with support for other Python versions in case you prefer your own installation. The PyUtils class can be used to access other compatible installations. For example, if you wanted to use your own install of Python 3.13, you can call the new PyUtils.Load to pick the appropriate Python bridge library. For example:
IDL> PyUtils.Load, 'python313'
See Python Bridge and the PyUtils Class for details.
FILE_HASH
IDL has a new FILE_HASH function that lets you compute the cryptographic hash for a file or array of files. Cryptographic hashes are useful for ensuring that a file has all of the expected bytes and has not been tampered with or damaged. The new function can compute the MD5, SHA-1, SHA-2 (256 bit) and SHA-2 (512 bit) hashes. For example:
IDL> file = filepath('hdf5_test.h5', subdir=['examples', 'data'])
IDL> print, file_hash(file)
69121a6aa9bc4eed8bdbd668f98d5503
IDL> print, file_hash(file, /SHA256)
74e416059b19e63581f9d8d03266081ed6a9a9a18fa44c2e41709ff7a7c20dba
See FILE_HASH for details.
HttpRequest PATCH
The HttpRequest class (introduced in IDL 9.0) has a new ::Patch method that lets you make PATCH requests to a server. Unlike POST or PUT, a PATCH request allows you to update existing resources on the server.
For example, to upload a file helloworld.txt that contains the text "Hello World":
IDL> data = hash("name", "patched file", "file", "@helloworld.txt")
IDL> response = HttpRequest.Patch('https://httpbin.org/patch', multipart = data)
IDL> response.status
200
IDL> response.json()
{
"args": {
},
"data": "",
"files": {
"file": "Hello World\r\n"
},
"form": {
"name": "patched file"
},
...
}
See HttpRequest for details.
Serial Port Communication
Using the new embedded Python bridge, IDL now supports communication with serial ports via the pyserial package.
For example, you can install the Python package:
pyutils.pipinstall, 'pyserial'
Next, plug your serial device into your computer. Then, in IDL, you can import the pyserial library and get a list of the available ports:
serial = Python.import('serial')
ports = Python.import('serial.tools.list_ports')
com_ports = ports.comports()
foreach p, com_ports do print, p.device
On Windows this will print out ports such as COM2 or COM3. On Linux and Mac the port names will look like /dev/ttyUSB0.
Once you have identified the correct port, you can then connect to it and begin communicating.
For example, here we connect an Arduino to a Windows machine and send a string back and forth:
serial = Python.import('serial')
arduino = serial.Serial(port="COM3", baudrate=9600, timeout=0.1)
wait, 2
!null = arduino.write(byte("hello"))
wait, 0.05
data = arduino.readline()
print, data
!null = arduino.close()
For the complete example see Serial Port Communication via Python Bridge.
Windows Command Line
On Windows, the IDL command line has been completely rewritten. The new command line has the following additional features:
-
Better performance when outputting large streams of data.
-
Colored text for the IDL or ENVI prompt, as well as informational messages or error messages.
Tip: To use the new Windows command line, you can either run the idl.exe application directly, or you can add the IDL bin directory to your system path and run idl from a Windows or bash shell.
Tab Completion
Pressing the TAB key in the new Windows command line will perform tab completion, first using IDL's command history (if there are any matching commands), or if there are no matches, then IDL tries to match file names on your local system.
For an example of file path completion, imagine you have two folder hierarchies on your system:
dir1/
subdir/
myimage.jpeg
dir2/
At the IDL command line, you could type:
IDL> read_jpeg, "./dir
If you press the tab key, IDL will print:
dir1/ dir2/
If you continue by typing a 1/, followed by another TAB key, IDL will fill in the rest of the subfolder path:
IDL> read_jpeg, "./dir1/subdir/
Hitting TAB again will then complete the file path:
IDL> read_jpeg, ".\dir1\subdir\myimage.jpeg
If there had been multiple files in that subdir, then IDL would have printed out all of the files, and you could have continued to type and hit TAB to narrow down the choices.
Updates
ASDF File Format Enhancements
IDL's support for ASDF (Advanced Scientific Data Format) has been enhanced with the following features:
-
Handles files with mixed endian byte orders
-
Supports reading files with strides
-
Supports reading streamed files
-
Supports reading and writing structured arrays
-
Supports reading and writing the ASCII data type
-
Support for BZIP2 compression
For more details see ASDF_PARSE and ASDF_WRITE.
CREATE_STRUCT has a new PRESERVE_CASE keyword
By default, when you create an IDL structure using either the curly braces { } or the CREATE_STRUCT function, the tag names are automatically converted to uppercase. In CREATE_STRUCT, you can use the new PRESERVE_CASE keyword to keep lowercase characters unchanged. For example:
IDL> a = create_struct('tag1', 1.0, 'TaG2', 2, /preserve_case)
IDL> help, a
** Structure <b5161400>, 2 tags, length=8, data length=6, refs=1:
tag1 FLOAT 1.00000
TaG2 INT 2
IDL> print, tag_names(a)
tag1 TaG2
Notes:
-
In IDL, you can refer to tag names in structures using any combination of upper or lowercase letters; all matches for tag names are case insensitive. This applies consistently, even when working with arrays of structures that have mixed-case tag names. For example:
IDL> a = create_struct('tag1', 1.0, 'TaG2', 2, /preserve_case)
IDL> print, a.TAG1, a.tag1, a.TAG2, a.tag2
1.00000 1.00000 2 2
IDL> arr = replicate(a, 10)
IDL> arr[5] = {TAG1: 2.0, TAG2: 3}
IDL> print, tag_names(arr[5])
tag1 TaG2
-
Structures created using curly braces { } always have uppercase tags, regardless of the actual case used in your IDL code. This behavior is unchanged. For example:
IDL> b = {tag1: 3.14, Tag2: 5}
IDL> help, b
** Structure <83e80910>, 2 tags, length=8, data length=6, refs=1:
TAG1 FLOAT 3.14000
TAG2 INT 5
-
Since tag matching is case insensitive, you cannot create a structure with two tag names that are identical except for case. For example, trying to create a structure with tag names "tag1" ang "TAG1" will throw an error.
HttpRequest has a simpler API for Downloading and Uploading Files
The HttpRequest class (introduced in IDL 9.0) allows you to make GET, POST, PUT, and DELETE requests to an HTTP or HTTPS server. In IDL 9.2 it is now much simpler to download and upload files.
First, the Get, Post, and Put methods have a new FILENAME keyword. Setting this keyword will download the response to a file instead of returning the data as a byte array or string. This allows you to download huge files from a URL directly to a file on your local file system. For example:
response = HttpRequest.Get('https://httpbin.org/', FILENAME='c:/temp/result.txt')
print, response.status_code
print, file_lines('c:/temp/result.txt')
Next, for POST and PUT, when uploading a file as part of a multipart form, you can now specify the file using a shortcut "@filepath" instead of creating an IDL structure. For example:
multipart = hash("name", "David Stern", "resume", "@c:/myplainresume.txt")
a = HttpRequest.Post(url, multipart=multipart)
See HttpRequest for details on these new features.
IDL Package Manager (IPM) support for IDL/ENVI Repository Portals, new API, and Publish
The IDL Package Manager (IPM) has been rewritten to use static methods. This provides an API that is cleaner and is consistent with other IDL classes such as HttpRequest.
The IPM also has a new Publish feature, which lets you upload IDL packages to a remote server.
In addition, a new IDL/ENVI Repository reader has been added. This allows you to create, upload, and install IDL or ENVI packages from within IDL. For example, if your server URL is myrepository.com, with port 8000, and your package name is ExtractAttributes, then the following code would create the package, upload it to the server, and then download and install it:
url = 'http://myrepository.com:8000/'
packagename = 'ExtractAttributes'
dir = getenv('HOME') + '/' + packagename
IPM.Create, dir, $
name = packagename, $
display_name = "Extract Attributes", $
description = "Extracts the attributes table of a shapefile.", $
tags = ["Vector"], $
version = '1.0.0', $
author = "David Stern", $
dependencies = [{ $
name: "DumpVectorTable", $
version: ">=1.0.0"}, $
{ $
name: "ReadShapefile", $
version: "<2.0"} $
], $
url = url
We can now add all of our required files, and then upload the newly-created package to the server:
IPM.Publish, dir, destination = 'http://myrepository.com:8000/'
To test your package, you can now install it:
IPM.Install, 'http://myrepository.com:8000/', name='ExtractAttributes', version='1.0.0'
IDL prints:
Package: ExtractAttributes, Version: 1.0.0 installed
Package: DumpVectorTable, Version: 1.2.0 installed
Package: ReadShapefile, Version: 1.9.0 installed
For details see IPM.
ISA has a new STRICT_ARRAY keyword
The ISA function currently has a /ARRAY keyword. This is useful for checking if a variable is indexable, like a regular IDL array, a list, or a hash.
ISA now has a new STRICT_ARRAY keyword that returns true only if the variable is truly an IDL array or structure. For example, a hash or list will return false, but an object array of hashes or lists will return true.
See ISA for details.
Physical Constants
The !CONST system variable contains values for the most common physical constants.
In 2019, the SI base units for the kilogram, ampere, kelvin, and mole were redefined in terms of natural physical constants instead of human artifacts. This was done by setting exact numerical values for the Planck constant (h), the elementary electric charge (e), the Boltzmann constant (kB), and the Avogadro constant (NA). Because of these redefinitions, the following constants have new values:
|
Name |
Description |
Old Value |
New Value |
|
alpha |
Fine structure constant
|
7.2973525698 x 10-3
|
7.2973525643 x 10-3
|
|
e |
Elementary charge
|
1.602176565 x 10-19 C
|
1.602176634 x 10-19 C
|
|
eps0 |
Vacuum electric permittivity
|
8.854187817 x 10-12 F/m
|
8.8541878188 x 10-12 F/m
|
|
F |
Faraday constant
|
96485.3365 C/mol
|
96485.33212 C/mol
|
|
G |
Gravitation constant
|
6.67428 x 10-11 m3/kg/s2
|
6.67430 x 10-11 m3/kg/s2
|
|
h |
Planck constant |
6.62606957 x 10-34 J s
|
6.62607015 x 10-34 J s
|
|
hbar |
h/(2π) |
1.054571726 x 10-34 J s
|
1.054571817 x 10-34 J s
|
|
k |
Boltzmann constant
|
1.3806488 x 10-23 J/K
|
1.380649 x 10-23 J/K
|
|
me |
Electron mass |
9.10938291 x 10-31 kg
|
9.1093837139 x 10-31 kg
|
|
mn |
Neutron mass |
1.674927351 x 10-27 kg
|
1.67492750056 x 10-27 kg
|
|
mp |
Proton mass |
1.672621777 x 10-27 kg
|
1.67262192595 x 10-27 kg
|
|
mu0 |
Vacuum magnetic permeability
|
1.2566370614 x 10-6 N/A2
|
1.25663706127 x 10-6 N/A2
|
|
n0 |
Loschmidt constant
|
2.6867805 x 1025 m-3
|
2.686780111 x 1025 m-3
|
|
Na |
Avogadro constant
|
6.02214129 x 1023 mol-1
|
6.02214076 x 1023 mol-1
|
|
parsec |
Parsec distance |
3.0856775814671912 x 1016 m
|
3.0856775814913673 x 1016 m
|
|
R |
Molar gas constant
|
8.3144621 J/mol/K
|
8.3144626181532395 J/mol/K
|
|
re |
Classical electron radius
|
2.8179403267 x 10-15 m
|
2.8179403205 x 10-15 m
|
|
rydberg |
Rydberg constant
|
10973731.568539 m-1
|
10973731.568157 m-1
|
|
sigma |
Stefan-Boltzmann constant
|
5.670373 x 10-8 W/m2/K4
|
5.670374419 x 10-8 W/m2/K4
|
|
u |
Atomic mass constant
|
1.660538921 x 10-27 kg
|
1.66053906892 x 10-27 kg
|
|
Vm |
Molar volume, ideal gas at STP
|
22.413968 x 10-3 m3/mol
|
22.41396954 x 10-3 m3/mol
|
While most of these values can be verified with suitable lab equipment, we are anxiously awaiting the return of our plucky IDL interns from their voyage to measure the new parsec distance.
See !CONST system variable for details.
Python::GetAttr method
The IDL-to-Python bridge has a new static method, Python::GetAttr. The Python::GetAttr static method retrieves an attribute from a Python object. The GetAttr method is useful if you want to specify your attribute as a string, or if you want to use the WRAP keyword to return the attribute as a Python object instead of converting to an IDL type.
Example
Create a Pandas DataFrame and then retrieve the "shape" attribute, which gives the length of each dimension:
np = python.import('numpy')
pd = python.import('pandas')
arr = (np.random).randn(6,4)
dates = pd.date_range("20130101", periods=6)
df = pd.dataframe(arr, index = dates, columns = ["A", "B", "C", "D"])
print, `shape = ${df.shape}`
shape = Python.getattr(df, 'shape', /wrap)
help, shape
print, shape
IDL prints:
shape = [6,4]
SHAPE PYTHON <ID=837> <class 'tuple'>
(6, 4)
Note that in the call to GetAttr, we have used the WRAP keyword to return the Python tuple without conversion. See Python::GetAttr for details.
Shapefiles with huge integer attributes
Prior to IDL 9.2, if you had an ESRI shapefile with an integer attribute, the attribute values were always read into a 32-bit integer. This agrees with the ESRI shapefile specification, which limits integers to 32-bits. However, certain shapefiles can contain integers which are outside of the possible range (–2147483648 to 2147483647). In these cases, IDL will now automatically return these attribute fields as IDL type DOUBLE. Note that if you then write these attribute values out to a new shapefile, the type will automatically be Double, which conforms to the ESRI specification and should also allow these files to be successfully used with other tools.
SORT function has a new STABLE keyword
By default, when you call the SORT function on an array with identical elements, the returned indices are in an arbitrary order. The SORT function now has a new STABLE keyword which will preserve the order of identical elements:
a = [4.0, 3.5, 7.0, 1.0, 1.1, 7.0, 7.0]
indices = sort(a)
indices_stable = sort(a, /stable)
print, `${indices}`
print, `${indices_stable}`
IDL prints:
[3, 4, 1, 0, 5, 6, 2]
[3, 4, 1, 0, 2, 5, 6]
Similarly, if you use the ::Sort method on an IDL variable, you can now specify the STABLE keyword to return the values in the INDICES keyword in the correct order:
a = [4.0, 3.5, 7.0, 1.0, 1.1, 7.0, 7.0]
astable = a.sort(indices = indices_stable, /stable)
print, `${indices_stable}`
See the SORT function and the IDL_Variable::Sort method for details.
IDL Extension for VS Code
The IDL for VSCode extension is available for free within Visual Studio Code. The extension can be easily downloaded and installed from the VS Code extensions page. The extension has the following new features:
-
Full debugging support, including halting execution.
-
An IDL Tutorial, consisting of a series of interactive IDL Notebooks that walk you through the IDL language.
For details visit the IDL for VSCode page.
MAKE_RT Enhancements
MAKE_RT has been optimized to produce a much smaller distribution. There are three new keywords:
-
BROWSER - by default MAKE_RT will not include the files needed for the IDL browser widget. Set this keyword to include those files.
-
PYTHON - by default MAKE_RT will not include the embedded Python in the IDL bin folder. Set this keyword to include those files.
-
VERBOSE - set this keyword to print out the full filepath for each copied file.
MAKE_RT now makes use of CLI_PROGRESS to display a progress bar:
IDL> make_rt,'helloWorld','C:\Users\<User>\IDLWorkspace\make_rt_copy_trgt'
IDL prints:
8.0% [###-------------------------------------] Filtering files
61.3% [########################----------------] Copying files
etc...
See MAKE_RT for details on all of these enhancements.
Trailing Commas in Array and Structure Creation
Now, when creating arrays, a trailing comma at the end of a sequence of values is ignored. For example:
a = [ $
[10:19], $
[20:29], $
[30:39], $
]
You can easily add a fourth row:
a = [ $
[10:19], $
[20:29], $
[30:39], $
[40:49], $
]
Similarly, for structures:
a = { $
field1: 1, $
field2: "2", $
field3: 3.0, $
}
You can easily add a fourth row:
a = { $
field1: 1, $
field2: "2", $
field3: 3.0, $
field4: "four", $
}
This feature makes it simpler to modify existing arrays or structures in code by just adding in another row without worrying about a missing comma. Having a trailing comma (or not) is completely optional and makes no difference to the created array or structure or its contained data.
WIDGET_INFO function has a new SCREENSHOT keyword
The SCREENSHOT keyword makes WIDGET_INFO return an [M, N, 3] byte image of the specified widget. Virtually all widgets can be captured, including top-level bases, sub-bases, browser widgets, and others. The only widgets that cannot be specified are menu bars, menus, menu items, and tree widget nodes. Images can be used in automated testing, report generation, etc.
See the WIDGET_INFO function for details.
WIDGET_TABLE has better handling for scroll bars and selection events
The WIDGET_TABLE function now adds scroll bars only when necessary, otherwise scroll bars are suppressed. You can now set X_SCROLL_SIZE = 0 or Y_SCROLL_SIZE = 0 to suppress scroll bars in the horizontal or vertical direction.
WIDGET_TABLE has a new WHEEL_EVENTS keyword, which enables the generation of wheel events whenever the mouse wheel is used to scroll the table widget.
WIDGET_TABLE has a new EXTEND_SELECTION keyword, which changes the selection event values when a row or column header is clicked. This allows you to distinguish between events where the user clicks inside the table versus when the user click on a row or column header.
WIDGET_TABLE now allows you to set both the foreground and background colors for the column and row headers. Header cells are indexed by using –1.
See WIDGET_TABLE for details.
WIDGET_TABLE can now repeat Format strings across rows or columns
The WIDGET_TABLE function has a new REPEAT_FORMAT keyword. If this keyword is set, and the FORMAT keyword is set to a vector or array that is smaller than the table size, then that format array is repeated across the additional rows or columns.
By setting FORMAT to a vector of strings, along with the REPEAT_FORMAT keyword, you can easily set the format for the entire table, where each row of the table will have the same formats.
See WIDGET_TABLE for details.
FILE_POLL_INPUT now works with Windows pipes
The FILE_POLL_INPUT function now supports Windows pipes, in addition to Windows sockets and Unix file descriptors. One use case for this occurs when IDL has SPAWN'd a child process that reads and writes to stdin and stdout with something like:
spawn, command, unit = pipe, /noshell
Now, IDL can check whether the child process has data available for reading:
result = file_poll_input(pipe, timeout = double)
This enables IDL applications to avoid blocking reads and thereby remain active. If the result is 1 then the IDL application can safely read (at least one byte) without blocking for data that may never come or arrive at some unknown, distant time.
See FILE_POLL_INPUT for details.
FFT on arm64 Macs has an updated implementation
Arm64 Mac IDL now ships with the Arm Performance Libraries version 25.04. This provides performance improvements for one dimensional data.
See Also
See What's New (Previous IDL Releases) for an archive of What's New information.