ODE Home PDS Geosciences Node at Washington University

Help Downloading ODE Cart Requests

This document provides a brief explanation of the organization of files requested through the ODE cart and how to efficiently download the files.

In early 2018 a new ODE cart Request Download page was introduced. The new page provides access to a user's cart request via HTTP, FTP, or the Aspera Connect web browser plug-in. Details about the page and the Aspera tool are provided below.

Quick links to sections on this page

Explanation of the cart request file organization

Tips for downloading files through the FTP link provided in the cart request email

   Download with FTP client

   Additional download techniques


ODE Cart Request Download Page

   Aspera Description

   Aspera Connect Browser Plug-in System Requirements

   Aspera Connect Browser Plug-in Installation

   Downloading Files with Aspera Connect Browser plug-in

   Local Aspera Optimization Settings

      Limit Transfer Rates - Most helpful in slow or unstable networks

      Change Maximum Concurrent Transfers

      Change Default Download Location

   HTTP Download Option

      Wget and Python download examples

   FTP Download Option



Explanation of the file organization

The organization of the requested files is based on a combination of the selected packaging format and file organization from the checkout process.

During the checkout process, the user selects the desired packaging format. Zip, Tar, Tar.Gz, and no compression are the available options.
Select cart packaging format

The user's selected file organization directs the ODE cart process to group the requested files based on their PDS volumes or together in one directory structure.

Select cart file organization

When a user requests a cart order in Zip, Tar, or Tar.Gz format, the files are grouped accordingly. These formats are often limited to less than 1GB of pre-compressed size. There are individual PDS files larger than 1GB, which can cause their grouped files to be larger. The general packaged size limitation has been imposed because some users have problems downloading and unpackaging files larger than 1GB. Therefore, PDS volumes or the single directory selections may be split across multiple packaged files. If the requested files are split across multiple packaged files, please find the files of the volume or combined directory in files ending with _part_0002, _part_0003, etc.

Example 1. LRO LOLA order requested in Zip format, organized by PDS volume. Notice the volume is split across multiple zip files due to the request size.
Select packaging format

Example 2. Several PDS files requested in Tar format in one directory. The cart order is split across multiple files due to the request size.
tar files

The selection of "No Compression" during the checkout process will place the requested files in their corresponding PDS volume folders or one large directory based on the user's file organization selection. The no compression option will require the individual download of each file. It may be easier to download each file with an FTP client, but any FTP technique will work.

Example 3. MRO CRISM cart results for no compression and original PDS archive directory structure
cart directories

Tips for downloading the files

Download Files with FTP client
Using an FTP client software application is a recommended solution for users who request more than a handful of files. The free FileZilla (https://filezilla-project.org/) application is recommended, but any FTP client software will work. These programs allow a user to easily download all of the files using a "drag and drop" interface. Additionally, multiple download streams can be used to improve the potential performance of downloading a cart request. Simply enter the FTP address and login credentials from the order fulfillment email into the client application to connect to the FTP site.

Example 4. Downloading ODE cart request with FileZilla
FTP Client Example

Additional File Download Techniques
There are several additional ways to acquire the ODE cart request files via FTP. The following techniques are some common options. Several browser plug-ins and add-ons provide a convenient way to download multiple files at one time. All operating systems provide a command line FTP option for downloading files. An online search will provide commands appropriate to your operating system. Wget (https://www.gnu.org/software/wget/) is a free command line application that can be used to download files through FTP and HTTP protocols.

If you experience problems or have questions with cart requests, please contact us at ode@wunder.wustl.edu.

ODE Cart Request Download Page

The ODE Cart Request Download page provides access to a user's cart request via HTTP, FTP, or the Aspera Connect data transfer web browser plug-in.

The page is divided into two sections. The first section, download options, provides direct links to your entire cart request for the Aspera Connect browser plug-in (one click to download the entire cart request), HTTP, and FTP. The second section of the page allows you to navigate through the cart order to download specific directories and files using the Aspera Connect browser plug-in. Files can be downloaded through direct HTTP links, as well.

Example 5. ODE Cart Request Download Page
ODE Cart Request Download Page

Through Washington University in St. Louis, the PDS Geosciences Node has access to Aspera high speed data transfer software (http://asperasoft.com/). Aspera provides a web browser plug-in that can be used to facilitate high-speed downloads from our Aspera Connect Server. The plug-in is compatible with most operating systems and web browsers, see the follow list.

Aspera Connect Browser Plug-in System Requirements
  • Windows 7, 8, 10, 2008 R2, 2012, 2012 R2
  • OS X 10.6+, Intel Only
  • Ubuntu 8.04+, RHEL/Centos 6.0+, Debian 6.0+
  • Firefox 27+, Google Chrome 32+, Safari 6+

Notes: Microsoft Edge has a problem launching the Aspera Connect browser plug-in. We will post a workaround when one is posted on the Aspera support site.
The browser plug-in will not start on Ubuntu 16.04 LTS, unless the following work around is applied:
https://www.ibm.com/support/pages/connect-wont-start-due-missing-qt-platform-plugin-xcb

Aspera Connect Browser Plug-in Installation
[NOTE: Installing the Aspera Connect browser plug-in may require the browser to be closed and opened again. Please bookmark any necessary pages before installing the browser plug-in.]

The first time attempting an Aspera download from the cart request download page, the page will recognize if the plug-in is not installed or if the plug-in is out of date. If either of these situations is the case, you will be prompted with a screen containing steps to install or upgrade IBM Aspera Connect. Follow the prompts. If the add-on is required for your browser, click the highlighted "Install add-on" button.

See the following installation steps.

Example 6. Click the highlighted "Install add-on" button


Example 7. In Firefox, the browser Add-on options will open. Click the "Add to Firefox", so it will allow the plug-in. Other browsers will have a similar step.


Example 8. You will then need to click the "add" button when asked to give IBM Aspera Connect permissions on your browser.
Security warning

Example 9. We recommend that you also allow the extension to run in Private Windows.


Example 10. After the previous steps are complete, you can close the browser tab containing the Add-ons.


Example 11. The next step is to download the IBM Aspera Connect Application. Click the "Download the App" button and click "Save File" when prompted with the file.


Example 12. After the file has downloaded, click the browser download icon (1). Then click or double click on the exe file name to start the application installation.


Example 13. When prompted to run the file, confirm by clicking "Run".


Example 14. Confirm the installation by clicking the "Start" button. Then the application will proceed to install the application. You will see various files flash by as they are installed.
Installing progress

Example 15. Once the installation is complete, a confirmation will be displayed.
Installing complete

Example 16. After closing the previous installation window with the "Close" button, you will return to the Required Steps window. Click the "Refresh" link on that window. If all went well, the window should close.
Refresh



Downloading Files with Aspera Connect Browser Plug-in
After the latest Aspera Connect browser plug-in has been installed, you are ready to download files. Either click the link to download the entire cart request using Aspera client or click one of the Aspera links that is next to a listed directory or file at the bottom of the page.

You may need to click "Allow" button at the top of the browser to allow Aspera to run through the browser. This is only required the first time during a browser session.

Example 17. Prompt to allow Aspera Browser Plug-in to launch
Prompt to allow Aspera Browser Plug-in to launch

The first time executing a download with the browser session, you will also receive a popup asking you to confirm your connection the Aspera server. Click the Allow button.

Example 18. Prompt to Allow Aspera Plug-in to download files
Allow Aspera Plug-in to download files

On a Windows machine, the following Aspera Connect icon will be found in the tool bar tray. Aspera Tool Tray Icon The icon can be clicked to open a detail window, which displays the specific download progress. The files are downloaded to your machine's default download location. For example, on a Windows machine, your "downloads" directory. Below is an example of watching the download process through Aspera.

Example 19. View Aspera download progress
Aspera Connect Download Processing

If you click the link for the entire cart request, the cart request will be saved to a subdirectory named the same as the cart request directory (example 20171217T23152833236) in your computer's default downloads location.

Example 20. Apera one click to download the entire cart option
Aspera one click download option

As previously mentioned, Download Specific Directories and Files section of the page allows the user to navigate the cart request directory structure. Indvidual directories and files can be downloaded from this location.

Example 21. Download Specific Directories and Files section of the page
Download Specific Directories and Files section

This feature also includes an option to filter the results of the current directory based on a simple search function.

Example 22. Basic file display filter
Basic file display filter

If the user clicks the "Aspera" download link for a directory listed in the "Download Specific Directories and Files" list, the directory will be downloaded to the user's default download location. In the example below, the cartOrder directory would be downloaded to the following location on a Windows machine: c:\Users\UserName\Downloads\cartOrder. Progress will be displayed on the web page as the directory contents are downloaded.

Example 23. Downloading a specific cart directory with the Aspera Connect web browser plug-in
Downloading a specific cart directory with the Aspera web browser plug-in

If the user clicks an individual file's Aspera download link, that file will be saved directly to the default download location. For example, a file would be saved to the following location on a Windows machine: c:\Users\UserName\Downloads\ODECartHTMLList_manifest.txt

Example 24. Downloading a specific file with the Aspera web browser plug-in
Downloading a specific file with the Aspera web browser plug-in

A description of downloading files throught the HTTP links in the "Download Specific Directories and Files" section is listed below on this page.

Local Aspera Connect Optimization Settings
There are several Aspera Connect (client) preferences that users may want to adjust before or while transferring files. Below are the settings most frequently adjusted by our users. If you are experiencing download failures, we recommend that you set the transfer rate limit as shown below.

If you continue to experience problems or have questions, please contact us at ode@wunder.wustl.edu.

Additional Aspera Connect documentation: https://download.asperasoft.com/download/docs/connect/3.7.4/user_win/webhelp/index.html#dita/setting_up.html

Open Aspera Connect Preferences
Launch Connect in Windows (Start Menu->All Programs->Aspera->Aspera Connect) and open Preferences (System Tray->Right-click Aspera Connect->Preferences).

Example 25. Aspera Connect (client) access preferences
Aspera Connect preferences

Limit Transfer Rates
This setting adjustment can provide the largest performance improvement for slow or unstable networks.
The default transfer rate of the PDS Geosciences Node's Aspera data transfer server is set to 1 Gb/s. If local available bandwidth is significantly lower than 1 Gb/s, you may experience poor transfer speeds, unless you adjust the Aspera Connect client bandwidth settings to reflect local network conditions. For example, if you are downloading from overseas and the available bandwidth is 10 Mb/s or less, it is recommended to set your limits accordingly. The upload and download rate limits are adjusted under the Bandwidth tab.

Example 26. Aspera Connect (client) adjust transfer rate preferences
Adjust Aspera connect transfer rate preferences

Change Maximum Concurrent Transfers
The default maximum number of concurrent transfers is set to 3. This number is adjusted under the General tab. In most situations, 3 concurrent transfers is adequate. The available bandwidth will be divided across the number of transfers that are occurring. If a user is downloading many tiny files, a higher concurrent transfer number could be beneficial. We suggest that you do not use a number higher than 10 concurrent transfers.

Example 27. Aspera Connect (client) adjust transfer rate preferences
Change Aspera connect's maximum concurrent transfers

Change Default Download Location
The default download location can changed from the Transfers tab. Click Browse navigate to a new path.

Example 28. Aspera Connect (client) adjust transfer rate preferences
Change Aspera connect's default download directory



HTTP Download Option
As previously mentioned, the new ODE Cart Request Download page supports HTTP file access. The "Download Options" section of the page provides a direct HTTP link to the cart request directory.

Example 29. HTTP Access to the cart request location
HTTP Access to the cart request location

Clicking the direct HTTP link will open a new browser window directly to the cart request location. Directories can be traversed manually through the web browser. Files can be downloaded directly by right clicking on the files and selecting "save as".

Various browser plug-ins that can be used to automatically download the files in the directory. A user can also use a program, such as wget or a python script from a command line to download all the files of the HTTP directory to a local location. Wget (https://www.gnu.org/software/wget/) is a free command line application that can be used to download files through FTP and HTTP protocols.

Two examples from the PDS Geosciences Node Forum:
Python example for downloading PDS Geosciences Node archive directories or ODE cart requests

Wget example for downloading PDS Geosciences Node archive directories or ODE cart requests

Example 30. Example of a cart request home directory:
HTTP cart request directory

Example 31. Example of exploring a subdirectory of the previous cart request:
HTTP cart request sub directory

Example 32. Example of "save as" dialog box if a file is selected for download:
HTTP file download

HTTP Download from Download Specific Directories and Files Area
Individual files can be downloaded through HTTP from the "Download Specific Directories and Files" area by left clicking on the HTTP link or right clicking on the link and selecting "save as". When left clicking on the link, a dialog box pops up with options to save the file or open it with a selected application (see example 19). The right click and "save as" option will prompt for the location on the user machine to save the file.

Example 33. Location for downloading a specific file through HTTP:
Download specific file through HTTP

Example 34. Example dialog box from left clicking on the HTTP link:
HTTP open/save file dialog box

FTP Download Option
A direct FTP link to the ODE cart request is available on the page. Use the provided FTP link to access files through a client application or command line. You must use the displayed username and password to access your cart request with this method.

Example 35. FTP access to the cart request location:
FTP Access to the cart request location

Download Files with FTP client
Using an FTP client software application is a recommended solution for users who request more than a handful of files. The free FileZilla (https://filezilla-project.org/) application is recommended, but any FTP client software will work. These programs allow a user to easily download all of the files using a "drag and drop" interface. Additionally, multiple download streams can be used to improve the potential performance of downloading a cart request. Simply enter the FTP address and login credentials from the order fulfillment email or ODE Cart Request Download page into the client application to connect to the FTP site.

Example 36. Downloading ODE cart request with FileZilla
FTP Client Example

Additional File Download Techniques
There are several additional ways to acquire the ODE cart request files via FTP. The following techniques are some common options. Several browser plug-ins and add-ons provide a convenient way to download multiple files at one time. All operating systems provide a command line FTP option for downloading files. An online search will provide commands appropriate to your operating system. Wget (https://www.gnu.org/software/wget/) is a free command line application that can be used to download files through FTP and HTTP protocols.


If you experience problems or have questions with cart requests, please contact us at ode@wunder.wustl.edu.