Server-side Analysis Tasks

### Adding Server-side Analysis Tasks to JS9

A JS9 web site can define analysis tools that will be loaded into the user's JS9 Analysis menu. These analysis tasks are then executed by the back-end (CGI or node.js) server using the original FITS data, with results sent back to the browser for display.

To learn how to install the back-end helper, see: Installing a Server-side Helper.

Analysis tools are defined within files contained in the analysis-plugins directory. If a file in this directory ends with a ".json" extension, it is assumed to be an analysis definition file. The following sample analysis definition files will be found in the analysis-plugins directory by default:

- js9Analysis-fits.json: generic FITS tasks (e.g., display header)
- js9Analysis-funtools.json: funtools-based analysis tasks

The analysis definition files are loaded by the Node.js-based helper on startup and passed to each newly loaded image. In you change the definition of a tool in one of these files, or add a new definition file, you either need to restart the Node helper, or send it a SIGUSR2 signal:

      kill -USR2 `ps guwax | egrep js9Helper | egrep -v egrep | awk '{print $2}'`

Re-initialization is not required for CGI-based support, since the analysis tool definitions are sent to each image in raw form.

Analysis definition files are in JSON format (http://www.json.org/) and consist of an array of JavaScript objects, one for each analysis tool. The properties which define an analysis tool are:

- name: a short identifier string (typically one word)
- title: a longer string that will be displayed in the Analysis menu
- files: a rule that will be matched against to determine whether this task is available for the current image
- purl: a URL pointing to a web page containing a user parameter form
- action: the command to execute on the server side
- rtype: a return type, which can be text, plot, fits, regions, catalog, alert, or none
- hidden: if true, the analysis task is not shown in the Analysis menu

See the sample json files in the analysis-plugins directory for the JSON syntax of this file.

The files property is a rule that determines whether an analysis task is available for a particular file. The following rules are defined:

- fits: true if the displayed image has an associated FITS file
- table: true if the displayed image is a binary table
- image: true if the displayed image is an image
- fitsHeader(pname,pvalue): true if the FITS header of the displayed image has the named parameter with the specified value
- imVar(name,value): true if a property of the displayed image has the specified value
- js9Var(name,value): true if the JS9 variable has the specified value
- winVar(name,value): true if a variable in the window context has the specified value

Examples:

- fitsHeader(INSTRUME,HRC): true if the FITS instrument parameter is "HRC"
- imVar(params.wcssys,FK5): true if the wcs system for the current image is FK5
- js9Var(globalOpts.projectName,Chandra_EPO): true if the web page has set the JS9.globalOpts.projectName property to "Chandra_EPO"
- winVar(MyProject.name,foo): true if MyProject has a property "name" with the value "foo"

Note that all checks are case-insensitive.

The action string contains the command to execute. For security reasons, this action is never executed directly. Instead, the first argument of the command must be a wrapper script contained in the analysis-wrappers directory. The js9Xeq wrapper script contained therein is the default script for running JS9 analysis. The second argument in the action string is a command id, which is executed within the command-processing case statement of the js9Xeq. This wrapper script checks for valid commands and arguments, and then executes the specified command in a way that protects against malicious attack. You can modify this script to meet your needs by adding case statements for each of the new tasks you want to run.

Alternatively, you can run your analysis tasks through your own wrapper script, by placing it in the analysis-wrappers directory. Using your own wrapper script allows you to perform your own special setup (e.g., setting environment variables, cd to work directory) before executing a task. Obviously you can use js9Xeq as an example and modify it to suit your needs.

In either case, note that execution of a wrapper script is accomplished by pre-pending the path of the analysis wrapper directory to the wrapper script name and executing that full pathname directly. This prevents a would-be attacker from executing anything except the wrapper itself. Of course, you still have to be careful to avoid insecure shell coding in your wrapper script. See The World Wide Web Security FAQ for advice on writing CGI scripts. Note that JS9 scripts generally use bash or sh. Please make sure your bash and sh programs are properly patched to protect against the bash bug (CVE-2014-6271).

The action command line can contain macros (strings prefixed with the $ character) that will be expanded by the browser before the action is sent to the server for processing. Typically, these macros are used to specify filenames, regions, and (possibly) user parameters. The following intrinsic macros are defined:

- $id: id of the JS9 display
- $image: name of the image file being displayed
- $filename: pathname of the parent file, if available, or the original FITS file
- $filename(this): pathname of the original FITS file, even if a parent is available
- $fits: pathname of the original FITS file
- $parent: pathname of the parent file, if available
- $regions: all currently defined regions
- $sregions: all source regions
- $bregions: all background regions
- $imcenter:center of displayed image in image coords (x,y)
- $wcscenter:center of displayed image in WCS coords (ra,dec)

Note that $filename macro will return the pathname of the parent by default, so that analysis can be done on the parent file. Use $filename(this) or $fits to return the pathname of the displayed FITS file, regardless of the presence of a parent. This is useful if you want to run analysis on the representation file instead of the parent.

The three regions macros return region info in the current wcs system. To specify an alternate wcs system, append "physical", "image", or "wcs" in parentheses, e.g., $regions(physical).

For example, to run the funcnts program, specifying the original FITS file as well as source and background regions, use:

        funcnts $filename $sregions $bregions

Many analysis tasks do not require the user to select additional parameter options and can can be executed immediately. Other command require user-selected options. These latter options should be put into a web form, whose location is specified by the purl option. See the params/histplot.html file for an example.

Parameter options in a parameter form should have unique ids. These ids can then be added as macros to the action command line by prefixing them with a $. For example, the histplot command is:

        js9Xeq histplot '$filename[$regions]' '$norm $bwidth' $column $bins

Here, the norm, bwidth, column, and bins parameters are taken from the web form for histplot. The $filename and $regions macros are intrinsic to JS9.

The input element of the analysis form should run the JS9.SubmitAnalysis() command, which will serialize the form, pass it to the macro expander, and then send the resulting command string to the server for execution. See histplot.html for sample syntax.

The rtype property determine how JS9 handles return values:

- alert: call JavaScript alert() function on the returned string
- catalog: the returned URL to a catalog file is loaded on the current image
- fits: the returned URL to a FITS file is loaded (2nd argument can be a parent file)
- none: check for an error return, but otherwise do nothing
- plot: the returned plot data is plotted
- regions: the returned URL to a regions file is loaded on the current image
- text: the returned text is displayed in a lightweight window

The text and plot returns are the most commonly used types.

NB: when text is returned, be aware that file pathnames are often output by programs. For security reasons, you should remove or modify these pathnames before returning the text. For example, JS9 funtools analysis routines use a sed command like this:

      sed 's#'${JS9_DIR:-none}'#${JS9_DIR}#g;'

to change pathnames containing the JS9 working directory so that they display the string "${JS9_DIR}" instead.

Sample scripts such as funcnts2flot and funhist2plot generate a JSON string that contains a valid flot object. For example, the radial profile plot contains x, y, and error values:

    {
      "label" : "surf_bri(...) vs. avg_radius(...)",
      "points" : {"errorbars" : "y", "yerr" : {"show" : "true", "color" : "red"}},
      "data": [[x1, y1, e1], [x2, y2, e2], ... [xn, xn, en]]
    }

while the histogram plot contains only x and y values:

    {
      "label" : "counts vs. ...bin",
      "data": [[x1, y1], [x2, y2], ... [xn, xn]]
    }

If your analysis task generates one of these two plot formats, JS9 will display the plot data. By default, the plot is generated using the jQuery flot library (see http://www.flotcharts.org/), which is supplied with JS9.

You also can use the plotly library for plotting. To do this, set the JS9.globalOpts.plotLibrary property to "plotly" and load the library in the web page header. (Due to its relatively large size, plotly is not packaged in the JS9 support file.)

Plotting is still somewhat experimental and rudimentary. For example, flot (but not plotly) simply uses the x and y keys to toggle between linear and log scaling. We will be adding features and improvements with time. Please work with us to improve plotting support.

Finally, the hidden value determines whether or not a task is displayed in the Analysis menu. If hidden value is true, the analysis task is not shown. However, it still is available to the JS9.RunAnalysis() routine.

### Adding Analysis Tasks Directly to Your Web Page

By default, server-side JS9 analysis tasks are executed using the Analysis menu, but they also can be executed directly from the web page by means of HTML elements (buttons, forms, etc.) To do this, a web page author simply creates the desired interface and calls either JS9.RunAnalysis() (for buttons, menus, etc.) or JS9.SubmitAnalysis() (for forms). These are described below. See the js9analysis.html page for an example.

The JS9.RunAnalysis() routine is used to execute an analysis task and return the results for further processing within the web page. The JS9.SubmitAnalysis() routine works similarly, but it is used with a form: it automatically serializes the form values and passes them to the analysis macro expander so that these values can be put into the analysis command line. The calling sequences for these routines are similar:

        JS9.RunAnalysis(aname, [options], returnFunc);
        JS9.SubmitAnalysis(this, aname, returnFunc);

where:

- aname: the name value of the task (as defined in the js9Analysis.json file)
- options: an optional array of parameters to pass to the analysis macro expander. The array is in jQuery name/value serialized object format, which is described here:

          http://api.jquery.com/serializeArray/

- returnFunc: a function to process the returned data when the task is completed
- this: the 'this' value of the form element

The returnFunc() routine is a callback function to process the returned data from the analysis task. The calling sequence is:

        returnFunc(stdout, stderr, errcode, aobj)

where:

- stdout: a string containing the standard output from the task
- stderr: a string containing the standard error from the task
- errcode: error code value from the task or 0
- aobj: the analysis object from the js9Analysis.json file, containing the rtype (e.g., "text" or "plot") property

Typically, you would check the errcode and/or stderr string first and issue an error message if there is an error. Otherwise, the stdout string can be processed based on the return type (rtype) of output (e.g., "text" or "plot"). For plotting, you can use the flot functionality that is loaded into JS9, or you can use your own chosen plotting package. See js9analysis.html for a simple example.

### Temporary Work Directories for Users

By default, server-side analysis is initiated in the directory in which Node.js or the CGI script is run. For tasks not creating temp files, this usually is sufficient. For other cases, it can be desirable to isolate users in their own temporary work directories. To do this automatically (i.e. without the task itself having to change directories), set the following property in the globalOpts object of the js9Prefs.json file:

- workDir: set to the relative path of a directory which will hold temporary working directories for each loaded web page (default: ./tmp)

The workDir directory should be relative to the directory in which the JS9 helper is run. This will ensure that you have access permission to temporary retrieve files via the web server (e.g., loading a remote file via the LoadProxy() command into a temporary directory, so that JS9 can subsequently load it via the web server). This restriction can be relaxed when using the file:// URI, although some browsers on some operating systems (e.g., Firefox on Linux) prevent you from loading files outside the web page domain even with the file:// URI.

You also can set the following globalOpts configuration parameters in js9Prefs.json:

- workDirQuota: the disk quota on working directory (default: 100Mb)
- rmWorkDir: remove workdir when web page disconnects? (default: true)

When workDir is configured, user sub-directories will be created in the workDir directory to contain temporary files. Analysis tasks will be started in the work sub-directory.

For Node.js-base helpers, the work directory is removed when the web page is unloaded. This can be changed by setting rmWorkDir to false in the preference file. For CGI-based helpers, you must remove the work directory manually (e.g., using a cron job that checks for time-last-accessed of work directories).

Note that the workDirQuota is not a hard quota: JS9 tasks such as load proxy enforce their own checks using this value. For example, load proxy only checks whether the quota has been exceeded before loading the next file. This means that users can load one file that will bring the total size above the quota. Thus, if the quota is 10Mb and the work directory is empty, the load proxy will load a 50Mb file. But it will not more load files until already-loaded files have been closed in JS9 (and thereby deleted in the temp directory) so as to bring the total under the quota.

### The Proxy Load Service

NB: use this service with care. It allows users to consume disk resources!

For security reasons, JavaScript contained in one web page can access data in another web page only if both web pages have the same origin (i.e., basically coming from the same host). This policy is called the Same Origin Policy. This means that JS9 cannot load a FITS file from an arbitrary URL without using special techniques.

One such technique is to use a proxy server: the URL is not loaded directly into JS9, but instead is copied back to the server from which JS9 itself was loaded. The file is then retrieved by JS9 so that the "same origin" rules are not violated.

A CGI or Node.js back-end server can be set up as a proxy server by setting the following globalOpts properties in the js9Prefs.json file (before starting the server):

- loadProxy: set to true to enable proxy load (default: true)
- workDir: see above
- workDirQuota: see above
- rmWorkDir: see above

When the back-end server supports the proxy service, you will see a menu option in the File menu called open link via proxy ...

Enter a FITS URL into the proxy dialog box and press the Load button. The data file will be retrieved by the server and stored in a directory specifically tied to the web page. (The directory and its contents will be deleted when the page is unloaded.) JS9 then will load the file from this directory. Note that since the file resides on the back-end server, all back-end analysis defined on that server is available.

This technique can also be used FITS links from other sources. For example, if you have a FITS file on Google Drive, Dropbox or even a static web page, you can generate/copy the link and paste it into the proxy dialog box for display. (Dropbox files also can utilize the open link via CORS ... option, which downloads the data directly to JS9, bypassing the proxy server.)

Note that individual proxy FITS files are deleted from the working directory when they are closed in JS9 using the File:close image menu option.

##### Last updated: June 1, 2021