# Development

This module contains the classes

- refactor, a module that assists in refactoring Modelica classes,
- Tester that runs the unit tests of the Buildings library,
- Validator that validates the html code of the info section of the .mo files, and
- Annex60 that synchronizes Modelica libraries with the Annex60 library.
- ErrorDictionary that contains information about possible error strings.

## Refactoring code

This module provides functions to

- create Modelica packages and autopopulate for example the package.mo and package.order files
- move Modelica classes include any associated .mos scripts, reference results and images, and
- rewrite the package.order file.

`buildingspy.development.refactor.``create_modelica_package`(directory)

  Create in directory a Modelica package.

  If directory exists, this method returns and does nothing. Otherwise, it creates the directory and populates it with a package.mo and package.order file.

  Parameters: | directory – The directory in which the package is created.

`buildingspy.development.refactor.``move_class`(source, target)

  Move the class from the source to the target name.

  Both arguments need to be Modelica class names such as Buildings.Fluid.Sensors.TemperatureTwoPort.

  Parameters: | source – Class name of the source. target – Class name of the target.

  Usage: Type

      >>> import buildingspy.development.refactor as r
      >>> r.move_class("Buildings.Fluid.Movers.FlowControlled_dp",         >>>              "Buildings.Fluid.Movers.Flow_dp")

`buildingspy.development.refactor.``write_package_order`(directory='.', recursive=False)

  Write the package.order file in the directory directory.

  Any existing package.order file will be overwritten.

  Parameters: | directory – The name of the directory in which the package.order file will be written. recursive – Set to True to recursively include all sub directories.

  Usage: To rewrite package.order in the current directory, type

      >>> import buildingspy.development.refactor as r
      >>> r.write_package_order(".")

## Regression tests

class `buildingspy.development.regressiontest.``Tester`(check_html=True, executable='dymola', cleanup=True)

  Class that runs all regression tests using Dymola.

  Initiate with the following optional arguments:

  Parameters: | check_html – bool (default=True). Specify whether to load tidylib and perform validation of html documentation executable – {‘dymola’, ‘omc’}. Default is ‘dymola’, specifies the executable to use for running the regression test with `run()`. cleanup – bool (default=True). Specify whether to delete temporary directories.

  This class can be used to run all regression tests. It searches the directory `CURRENT_DIRECTORY\Resources\Scripts\Dymola` for all `*.mos` files that contain the string `simulate`, where `CURRENT_DIRECTORY` is the name of the directory in which the Python script is started, as returned by the function `getLibraryName()`. All these files will be executed as part of the regression tests. Any variables or parameters that are plotted by these `*.mos` files will be compared to previous results that are stored in `CURRENT_DIRECTORY\Resources\ReferenceResults\Dymola`. If no reference results exist, then they will be created. Otherwise, the accuracy of the new results is compared to the reference results. If they differ by more than a prescibed tolerance, a plot such as the one below is shown.

  Plot that compares the new results (solid line) of the regression test with the old results (dotted line). The blue line indicates the time where the largest error occurs.

  In this plot, the vertical line indicates the time where the biggest error occurs. The user is then asked to accept or reject the new results.

  For Dymola, the regression tests also store and compare the following statistics for the initialization problem and the time domain simulation:

  > 1. The number and the size of the linear system of equations,
  > 2. the number and the size of the nonlinear system of equations, and
  > 3. the number of the numerical Jacobians.

  To run the regression tests, type

      >>> import os
      >>> import buildingspy.development.regressiontest as r
      >>> rt = r.Tester()
      >>> myMoLib = os.path.join("buildingspy", "tests", "MyModelicaLibrary")
      >>> rt.setLibraryRoot(myMoLib)
      >>> rt.run()
      Using  ...  of ... processors to run unit tests.
      Number of models   :  2
                blocks   :  0
                functions:  0
      Generated  5  regression tests.

      Script that runs unit tests had 0 warnings and 0 errors.

      See 'simulator.log' for details.
      Unit tests completed successfully.

      Execution time = ...

  To run regression tests only for a single package, call `setSinglePackage()` prior to `run()`.

  `areResultsEqual`(tOld, yOld, tNew, yNew, varNam, filNam)

    Return True if the data series are equal within a tolerance.

    Parameters: | tOld – List of old time values. yOld – Old simulation results. tNew – Time stamps of new results. yNew – New simulation results. varNam – Variable name, used for reporting. filNam – File name, used for reporting.
    Returns: | A list with `False` if the results are not equal, and the time of the maximum error, and a warning message or None. In case of errors, the time of the maximum error may by None.

  `are_statistics_equal`(s1, s2)

    Compare the simulation statistics s1 and s2 and return True if they are equal, or False otherwise.

  `batchMode`(batchMode)

    Set the batch mode flag.

    Parameters: | batchMode – Set to `True` to run without interactive prompts and without plot windows.

    By default, the regression tests require the user to respond if results differ from previous simulations. This method can be used to run the script in batch mode, suppressing all prompts that require the user to enter a response. If run in batch mode, no new results will be stored. To run the regression tests in batch mode, enter

        >>> import os
        >>> import buildingspy.development.regressiontest as r
        >>> r = r.Tester()
        >>> r.batchMode(True)
        >>> r.run()

  `checkPythonModuleAvailability`()

    Check whether all required python modules are installed.

    If some modules are missing, then an ImportError is raised.

  `deleteTemporaryDirectories`(delete)

    Flag, if set to `False`, then the temporary directories will not be deleted after the regression tests are run.

    Parameters: | delete – Flag, set to `False` to avoid the temporary directories to be deleted.

    Unless this method is called prior to running the regression tests with `delete=False`, all temporary directories will be deleted after the regression tests.

  `getLibraryName`()

    Return the name of the library that will be run by this regression test.

    Returns: | The name of the library that will be run by this regression test.

  `getModelicaCommand`()

    Return the name of the modelica executable.

    Returns: | The name of the modelica executable.

  static `get_plot_variables`(line)

    For a string of the form *y={aa,bb,cc}*, optionally with whitespace characters, return the list [aa, bb, cc]. If the string does not contain y = ..., return None.

    A usage may be as follows. Note that the second call returns None as it has a different format.

        >>> import buildingspy.development.regressiontest as r
        >>> r.Tester.get_plot_variables('y = {"a", "b", "c"}')
        ['a', 'b', 'c']
        >>> r.Tester.get_plot_variables('... x}, y = {"a", "b", "c"}, z = {...')
        ['a', 'b', 'c']
        >>> r.Tester.get_plot_variables("y=abc") is None
        True

  `include_fmu_tests`(fmu_export)

    Sets a flag that, if `False`, does not test the export of FMUs.

    Parameters: | fmu_export – Set to `True` to test the export of FMUs (default), or `False` to not test the FMU export.

    To run the unit tests but do not test the export of FMUs, type

        >>> import os
        >>> import buildingspy.development.regressiontest as r
        >>> r = r.Tester()
        >>> r.include_fmu_tests(False)
        >>> r.run()

  `isExecutable`(program)

    Return `True` if the `program` is an executable

  static `isValidLibrary`(library_home)
    Returns true if the regression tester points to a valid library
      that implements the scripts for the regression tests.

    Parameters: | library_home – top-level directory of the library, such as `Buildings`.
    Returns: | `True` if the library implements regression tests, `False` otherwise.

  `pedanticModelica`(pedanticModelica)

    Set the pedantic Modelica mode flag.

    Parameters: | pedanticModelica – Set to `True` to run the unit tests in the pedantic Modelica mode.

    By default, regression tests are run in non-pedantic Modelica mode. This however will be changed in the near future.

        >>> import os
        >>> import buildingspy.development.regressiontest as r
        >>> r = r.Tester()
        >>> r.pedanticModelica(True)
        >>> r.run()

  `printNumberOfClasses`()

    Print the number of models, blocks and functions to the standard output stream

  `run`()

    Run all regression tests and checks the results.

    Returns: | 0 if no errros occurred during the regression tests, otherwise a non-zero value.

    This method

    - creates temporary directories for each processors,
    - copies the directory `CURRENT_DIRECTORY` into these temporary directories,
    - creates run scripts that run all regression tests,
    - runs these regression tests,
    - collects the dymola log files from each process,
    - writes the combined log file `dymola.log` to the current directory,
    - compares the results of the new simulations with reference results that are stored in `Resources/ReferenceResults`,
    - writes the message Regression tests completed successfully. if no error occured,
    - returns 0 if no errors occurred, or non-zero otherwise.
  `setDataDictionary`()

    Build the data structures that are needed to parse the output files.

  `setLibraryRoot`(rootDir)

    Set the root directory of the library.

    Parameters: | rootDir – The top-most directory of the library.

    The root directory is the directory that contains the `Resources` folder and the top-level `package.mo` file.

    Usage: Type

          >>> import os
          >>> import buildingspy.development.regressiontest as r
          >>> rt = r.Tester()
          >>> myMoLib = os.path.join("buildingspy", "tests", "MyModelicaLibrary")
          >>> rt.setLibraryRoot(myMoLib)

  `setNumberOfThreads`(number)

    Set the number of parallel threads that are used to run the regression tests.

    Parameters: | number – The number of parallel threads that are used to run the regression tests.

    By default, the number of parallel threads are set to be equal to the number of processors of the computer.

  `setSinglePackage`(packageName)

    Set the name of a single Modelica package to be tested.

    Parameters: | packageName – The name of the package to be tested.

    Calling this method will cause the regression tests to run only for the examples in the package `packageName`, and in all its sub-packages. For example, if `packageName = Annex60.Fluid.Sensors`, then a test of the `Annex60` library will run all examples in `Annex60.Fluid.Sensors.*`.

  `showGUI`(show=True)

    Call this function to show the GUI of the simulator.

    By default, the simulator runs without GUI

  `test_JModelica`(cmpl=True, load=False, simulate=False, packages=['Examples'], number=-1)

    Test the library compliance with JModelica.org.

    This is the high-level method to test a complete library, even if there are no specific `.mos` files in the library for regression testing.

    This method sets self._nPro to 1 as it only works on a single core. It also executes self.setTemporaryDirectories()

    Parameters: | cpml – Set to `True` for the model to be compiled. load – Set to `True` to load the model from the FMU. simulate – Set to `True` to cause the model to be simulated. packages – Set to an array whose elements are the packages that contain the test models of the library. number – Number of models to test. Set to `-1` to test all models.

    Usage:

    > 1. Open a JModelica environment (ipython or pylab with JModelica environment variables set).

    > 2. cd to the root folder of the library

    > 3. type the following commands:

    >        >>> t = Tester()
    >        >>> t.testJmodelica(...)

  `test_OpenModelica`(cmpl=True, simulate=False, packages=['Examples'], number=-1)

    Test the library compliance with OpenModelica.

    This is the high-level method to test a complete library, even if there are no specific `.mos` files in the library for regression testing.

    This method sets self._nPro to 1 as it only works on a single core. It also executes self.setTemporaryDirectories()

    Parameters: | cpml – Set to `True` for the model to be compiled. simulate – Set to `True` to cause the model to be simulated (from 0 to 1s). packages – Set to a list whose elements are the packages that contain the test models of the library number – Number of models to test. Set to `-1` to test all models.

    Usage:

    > 1. In a python console or script, cd to the root folder of the library

    >        >>> t = Tester()
    >        >>> t.test_OpenModelica()
    >        OpenModelica script ...OMTests.mos created
    >        Logfile created: ...OMTests.log
    >        Starting analysis of logfile

    >        ######################################################################
    >        Tested 4 models:
    >          * 0 compiled successfully (=0.0%)

    >        Successfully checked models:
    >        Failed model checks:
    >          * BuildingsPy.buildingspy.tests.MyModelicaLibrary.Examples.BooleanParameters
    >          * BuildingsPy.buildingspy.tests.MyModelicaLibrary.Examples.Constants
    >          * BuildingsPy.buildingspy.tests.MyModelicaLibrary.Examples.MyStep
    >          * BuildingsPy.buildingspy.tests.MyModelicaLibrary.Examples.ParameterEvaluation

    >        More detailed information is stored in self._omstats
    >        ######################################################################

  `useExistingResults`(dirs)

    This function allows to use existing results, as opposed to running a simulation.

    Parameters: | dirs – A non-empty list of directories that contain existing results.

    This method can be used for testing and debugging. If called, then no simulation is run. If the directories `['/tmp/tmp-Buildings-0-zABC44', '/tmp/tmp-Buildings-0-zQNS41']` contain previous results, then this method can be used as

        >>> import buildingspy.development.regressiontest as r
        >>> l=['/tmp/tmp-Buildings-0-zABC44', '/tmp/tmp-Buildings-0-zQNS41']
        >>> rt = r.Tester()
        >>> rt.useExistingResults(l)
        >>> rt.run()

  `writeOpenModelicaResultDictionary`()

    Write in `Resources/Scripts/OpenModelica/compareVars` files whose name are the name of the example model, and whose content is:

        compareVars :=
          {
            "controler.y",
            "sensor.T",
            "heater.Q_flow"
          };

    These files are then used in the regression testing that is done by the OpenModelica development team.

## Validator of syntax

class `buildingspy.development.validator.``Validator`

  Class that validates `.mo` files for the correct html syntax.

  `validateHTMLInPackage`(rootDir)

    This function recursively validates all `.mo` files in a package.

    If there is malformed html code in the `info` or the `revision` section, then this function write the error message of tidy to the standard output.

    Note that the line number correspond to an intermediate format (e.g., the output format of tidy), which may be different from the `.mo` file.

    Parameters: | rootDir – The root directory of the package.
    Returns: | str[] Warning/error messages from tidylib.

    Usage: Type

          >>> import os
          >>> import buildingspy.development.validator as v
          >>> val = v.Validator()
          >>> myMoLib = os.path.join(                    "buildingspy", "tests", "MyModelicaLibrary")
          >>> # Get a list whose elements are the error strings
          >>> errStr = val.validateHTMLInPackage(myMoLib)

## Merging Annex60 library

class `buildingspy.development.merger.``Annex60`(annex60_dir, dest_dir)

  Class that merges a Modelica library with the Annex60 library.

  Both libraries need to have the same package structure.

  By default, the top-level packages Experimental and Obsolete are not included in the merge. This can be overwritten with the function `set_excluded_packages()`.

  `merge`()

    Merge all files except the license file and the top-level `package.mo`

    Warning

    This method is experimental. Do not use it without having a backup of your code.

    This function merges the Annex60 library into other Modelica libraries.

    In the top-level directory of the destination library, this function creates the file .copiedFiles.txt that lists all files that have been copied. In subsequent calls, this function deletes all files listed in .copiedFiles.txt, then merges the libraries, and creates a new version of .copiedFiles.txt. Therefore, if a model is moved in the Annex60 library, it will also be moved in the target library by deleting the old file and copying the new file.

    This function will merge all Modelica files, Modelica scripts, regression results and images. An exception is the file Annex60/package.mo, as libraries typically have their own top-level package file that contains their release notes and version information.

    When copying the files, all references and file names that contain the string Annex60 will be renamed with the name of the top-level directory of the destination library. Afterwards, the package.order files will be regenerated, which allows libraries to have Modelica classes in the same directory as are used by the Annex60 library, as long as their name differs.

    A typical usage is

          >>> import buildingspy.development.merger as m
          >>> import os
          >>> home = os.path.expanduser("~")
          >>> root = os.path.join(home, "test")
          >>> annex60_dir = os.path.join(root, "modelica-annex60", "Annex60")
          >>> dest_dir = os.path.join(root, "modelica-buildings", "Buildings")
          >>> mer = m.Annex60(annex60_dir, dest_dir)
          >>> mer.merge()

  `set_excluded_packages`(packages)

    Set the packages that are excluded from the merge.

    Parameters: | packages – A list of packages to be excluded.

## Error dictionary

class `buildingspy.development.error_dictionary.``ErrorDictionary`

  Class that contains data fields needed for the error checking of the regression tests.

  If additional error messages need to be checked, then they should be added to the constructor of this class.

  `get_dictionary`()

    Return the dictionary with all error data

  `increment_counter`(key)

    Increment the error counter by one for the error type defined by key.

    Parameters: | key – The json key of the error type.

  `keys`()

    Return a copy of the dictionary’s list of keys.

  `tool_messages`()

    Return a copy of the tool messages as a list.