Special facilities
==================

.. _module_mechanism:

Transcrypt's module mechanism
-----------------------------

Transcrypt's module mechanism looks a lot like Python's but there are a few differences.
Firstly, in Transcrypt it is good practice to use url-based unique module identifiers, e.g.

- *com.github.<my name>.<my module name>*
- *org.python.pypi.<my module name>*
- *com.<my company name>.<my module name>*

To achieve optimal CPython compatibility, an exception is made for modules that are part of the CPython distribution, e.g. *math*.
	
Note that Transcrypt is meant be to used with JavaScript rather than Python libraries, since its focus isn't on the desktop but on the browser. Nevertheless a growing set of standard Python modules is part of the distribution, currently cmat, datetime, itertools, logging, math, random (partially), re (almost complete), time, turtle and warnings. Other modules are in the making.

Finding the code for a module proceeds as follows:

Suppose you import a module *all.the.kings.men*. Then the following paths will be searched respectively:

	+ *<directory of your main module>/all/the/kings/men.py*
	+ *<directory of your main module>/all/the/kings/men.js*
	+ *<directory of your main module>/all/the/kings/men/__init__.py*
	+ *<directory of your main module>/all/the/kings/men/__init__.js*
	
	- *transcrypt/Transcrypt/modules/all/the/kings/men.py*
	- *transcrypt/Transcrypt/modules/all/the/kings/men.js*
	- *transcrypt/Transcrypt/modules/all/the/kings/men/__init__py*
	- *transcrypt/Transcrypt/modules/all/the/kings/men/__init__js*
	
	+ *<CPython packages directory 1>/all/the/kings/men.py*
	+ *<CPython packages directory 1>/all/the/kings/men.js*
	+ *<CPython packages directory 1>/all/the/kings/men/__init__.py*
	+ *<CPython packages directory 1>/all/the/kings/men/__init__.js*
	
	- *<CPython packages directory 2>/all/the/kings/men.py*
	- *<CPython packages directory 2>/all/the/kings/men.js*
	- *<CPython packages directory 2>/all/the/kings/men/__init__.py*
	- *<CPython packages directory 2>/all/the/kings/men/__init__.js*

	+ *More CPython packages directories*

As can be seen from the above list, modules local to your project take precedence over modules available in Transcrypt, which again take precedence over modules available globally in CPython. Note that even if modules are made available globally in CPython, importing them in Transcrypt gives special possibilities and restrictions. They are allowed to be written in native JavaScript, in which case they reside in the __javascript__ subdirectory of the module. They should not depend on C, C++ or features that are outside Python subset supported by Transcrypt.

Although under these guidelines it's quite possible to write modules that are importable both by CPyton and Transcrypt, most Transcrypt modules will be come from the JavaScript, rather than from the Python ecosystem. If both a Python and a JavaScript incarnation of a module are present, the Python module is only recompiled if it's younger than the corresponding JavaScript module, unless the -b switch is used.

Furthermore, note that the *__init__.py* or *__init__.js* file of a module is executed if and only if that module is imported, not if it's just somewhere in the hierarchy of directories containing that module. Furthermore the global code of a module is executed only once, no matter how often that module is imported, as is equally the case with CPython.

Another deviation from CPython is that the inclusion of a module has to be decided upon compiletime. This means that imports cannot be runtime conditional, so e.g. cannot be under an *if*. For compiletime conditional imports you can use
:ref:`\_\_pragma\_\_ ('ifdef') <pragma_ifdef>`. Also, since imports are resolved in one pass, cyclic imports are not supported.

As a consequence of the above, modules may be distributed as Python *.py* files, but also as JavaScript-only *.js* files. The JavaScript files may be hand written or generated by any tool including Transcrypt.

Using browser stubs to test non-GUI code that uses console.log and window.alert
-------------------------------------------------------------------------------
To test the non-GUI part of your code in a desktop rather than a browser environment, use *from org.transcrypt.stubs.browser import \**. This will allow you to call the *window.alert* and *console.log* functions in your code when you run it from the command prompt, using the -r command line switch: *transcrypt -r <my main module name>*. This will invoke CPython, searching the appropriate module paths as compilation would have done.

Creating JavaScript objects with __new__ (<constructor call>)
-------------------------------------------------------------

While creating objects in Transcrypt mostly happens in the plain Python way, e.g. *canvas = Canvas (<parameters>)*, objects from 3rd party JavaScript libraries often have to be created using *new*. In Transcrypt such objects are created by calling the  *__new__* function, e.g. *canvas = __new__ (Canvas (<parameters>)*, as can be seen in the :ref:`Pong example<code_pong>`. This mechanism is simple, follows Python's syntax rules and doesn't require updating of an encapsulation layer if a later version of the underlying JavaScript library features additional constructor functions. Therefore in most cases it is the preferred way to create objects who's initialization requires calling 3rd party JavaScript constructor functions.

As an alternative, instantiation and construction can be encapsulated in one function call. Since this in fact creates an alternative API facade for the 3rd party JavaScript library, such an encapsulation should be kept separate from the original library, e.g. by putting it in a separate importable module. The JavaScript code for this encapsulation would e.g. be *<facade module name>.Canvas = function (<parameters>) {return new Canvas (<parameters>);};*. After importing the facade module, canvas creation is straightforward: *canvas = Canvas (<parameters>)*.

As a third alternative, encapsulation can be done in Python rather than JavaScript: *def Canvas (<parameters>): return __new__ (<3rd party module name>.Canvas (<parameters>)*. Also in this case the creation syntax is simple: *canvas = Canvas (<parameters>)*.

The __pragma__ mechanism
------------------------

Pragma's are directives in the source code, that locally alter the behaviour of the compiler. Pragma's come in four varieties.

The function-like variety
~~~~~~~~~~~~~~~~~~~~~~~~~

*__pragma__ (<parameters>)*

is acceptable only to Transcrypt, CPython requires a stub with parameter *\*args*. Such a stub can either be defined locally or imported:

*from org.transcrypt.stubs.browser import __pragma__*

The comment-like variety
~~~~~~~~~~~~~~~~~~~~~~~~

# __pragma__ (<parameters>)

is acceptable both to Transcrypt and CPython. In CPython it does nothing.

N.B. Both varieties above have to be properly indented, matching indentation of their context.

The single line activation variety
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

<line of code> # __:<single parameter>

It will switch a facility on or off just for the line of code it's part of.
Single line pragma's can only be used for pragma's which have a single parameter, it's name.

For example the following line:

*vector2 = vector0 + vector1 # __:opov*

will be compiled identically to:

*__pragma__ ('opov'); vector2 = vector0 + vector1; __pragma__ ('noopov')*

The single line deactivation variety
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*scalar2 = scalar0 + scalar1 # __:noopov*

will be compiled identically to:

*__pragma__ ('noopov'); scalar2 = scalar0 + scalar1; __pragma__ ('opov')*

.. _pragma_alias:

Identifier aliasing: __pragma__ ('alias', ...) and __pragma__ ('noalias', ...)
--------------------------------------------------------------------------------------------------------------------------------
Calling *__pragma__ ('alias', <Python id part>, <JavaScript id part>)* at the start of a module will replace identifiers or parts thereof like in the following examples:

Example 1:

+---------------------------------------------------------------------+
| Used at the start of the module: *__pragma__ ('alias', 'S', '$')*   |
+------------------------------------------+--------------------------+
| Original in Python:                      | Alias in JavaScript:     |
+------------------------------------------+--------------------------+
| *S ('body')*                             | *$ ('body')*             |
+------------------------------------------+--------------------------+
| *S__body*                                | *$body*                  |
+------------------------------------------+--------------------------+
| *S__She__S__Sells__S__Sea__S__Shells__S* | *$She$Sells$Sea$Shells$* |
+------------------------------------------+--------------------------+

Using the above alias, a piece of jQuery code will look like this in Python:

.. literalinclude:: ../../demos/jquery_demo/jquery_demo.py
   :tab-width: 4
   :caption: Use of __pragma__ ('alias', 'S', '$') in jquery_demo.py

Example 2:

+-------------------------------------------------------------------+
| Used at the start of the module *__pragma__ ('alias', 'jq', '$')* |
+-----------------------------------------+-------------------------+
| *jq__body = jq (body)*                  | *$body = $ ('body')*    |
+-----------------------------------------+-------------------------+

Note that in the generated JavaScript only the modified identifiers will be present, not the original ones. So the JavaScript identifiers are only aliases for the Python ones, not for any identifier in the JavaScript code itself.

A number of aliases are predefined in the source code of Transcrypt as follows:

.. literalinclude:: ../../modules/org/transcrypt/compiler.py
	:start-after: # START predef_aliases
	:end-before: # END predef_aliases
	:tab-width: 4
	:caption: Transcrypt's predefined aliases

The predefined aliases are used to resolve name conflicts between Python and JavaScript. Calling e.g *<object>.sort (<params>)* from Python will invoke a *sort* method with Python semantics, which in the generated JavaScript is called *py_sort*. Calling *<object>.js_sort <params>* from Python will invoke a *sort* method with JavaScript semantics, which in the generated JavaScript is simply called *sort*, as 3rd party JavaScript libraries will expect.

Calling *__pragma__ ('noalias', <Python id part>)* will remove the alias. Calling *__pragma__ ('noalias')* without second parameter will remove all aliases. WARNING: This will also remove the predefined aliases.

The alias mechanism is vital to both the compactness and speed of code generated by Transcrypt and to its seamless integration with JavaScript. It allows Transcrypt to extend native JavaScript objects without causing name conflicts even when JavaScript gets expanded over the years, but also without any conversions or other overhead. That's why in Transcrypt this approach is favored over wrapping native objects.

Generating __doc__ attributes from docstrings: __pragma__ ('docat') and __pragma__ ('nodocat')
----------------------------------------------------------------------------------------------
Even though docstrings may be present in your code, by default *__doc__* attributes are not generated, to keep the generated JavaScript code lean. You can switch on and off generation of *__doc__* attributes for modules, classes, methods and functions with *__pragma__ ('docat')* and *__pragma__ ('nodocat')* respectively. There's also a *-d* / *--docat* command line switch, which can be overruled per module by *__pragma__ ('nodocat')*. Advice: Though these pragma's are flexible, its advisable to use them only at the spot indicated in the :ref:`docstrings testlet <autotest_docstrings>`, to retain CPython compatibility. Note that *__pragma__ ('docat')* follows rather than precedes the module docstring, since CPython dictates the module docstring to be the first statement in a module.

Surpassing the speed of native JavaScript: __pragma__ ('fcall') and __pragma ('nofcall')
----------------------------------------------------------------------------------------
Fast calls or fcalls are method calls where the method isn't an attribute of an object's prototype, but of the object itself, even if this method is inherited over multiple levels. This means that only for the first call the prototype chain is searched and the method is bound to the object. All subsequent calls invoke the bound method directly on the object. You can use the -f command line switch to generate fcall's pervasively, making your objects slightly larger since they will contain references to their methods. If you don't want that, just place the definition of the method(s) or class(es) you wish to optimize between   *__pragma__ ('fcall')* and *__pragma__ ('nofcall')*. You can also do the opposite, using the -f switch and exempting some method(s) or class(es) by embedding them between *__pragma__ ('nofcall')* and *__pragma__ ('fcall')*.

Note that these pragmas have to be applied on the function definition location rather than the call location. Placing *__pragma__ ('fcall')* or *__pragma__ ('nofcall')* at the beginning of a module will influence all methods defined in that module. The fcall mechanism is a form of memoization and one example of a transpiler being able to generate optimized code that surpasses common hand coding practice. The fcall mechanism influences neither the pure Python syntax nor the semantics of your program.

Enabling Pythons *send* syntax: __pragma__ ('gsend') and __pragma ('nogsend')
-----------------------------------------------------------------------------

These pragmas enable respectively disable use of Pythons *send* syntax for generators, which is disabled by default. An example of its use is in the following code:

.. literalinclude:: ../../development/automated_tests/transcrypt/iterators_and_generators/__init__.py
	:start-after: BEGIN 1st example with 'send'
	:end-before: END 1st example with 'send'
	:tab-width: 4
	:caption: Use of __pragma ('gsend') to enable Pythons *send* syntax fo generators 

Automatic conversion to iterable: __pragma__ ('iconv') and __pragma__ ('noiconv')
---------------------------------------------------------------------------------
In CPython sometimes automatic conversion from a non-iterable to an iterable type takes place. This comes at the expense of a runtime typecheck and is by default avoided in Transcrypt for that reason. Iteration through the keys of a *dict* without explicitly calling its *keys ()* member is a frequent use case of automatic conversion. To switch on automatic conversion for dicts locally, *__pragma__ ('iconv')* and *__pragma__ ('noiconv')* can be used. The alternative is to switch on automatic conversion globally using the -i command line switch. Use of this switch is disadvised, especially for numerical processing code containing nested loops, since it adds the mentioned typecheck to each *for .. in ..* loop. When designing numerical processing libraries, it's advisable to use *__pragma__ ('noiconv')* explicitly at the start of each performance-sensitive module. The result will be that even when an application developer chooses to use the -i switch, the performance of the computations won't suffer.

.. _pragma_ifdef:

Conditional compilation: __pragma__ ('ifdef', <symbol>), __pragma__ ('ifndef', <symbol>), __pragma__ ('else') and __pragma__ ('endif')
--------------------------------------------------------------------------------------------------------------------------------------
A piece of code in between *__pragma__ ('ifdef', <symbol>)* and *__pragma__ ('endif')* will only be compiled if <symbol> occurs in the global list of defined symbols.

This pragma works mainly in combination with the *-s* / *--symbols <names joined by $>* command line option. On top of that, some command line options automatically add symbols, without using the *-s* switch.

TODO: Add example

Code after *__pragma__ ('ifndef', <symbol>)* is compiled if <symbol> is NOT defined. To choose between two alternative source code blocks, precede the second block with *__pragma__ ('else')* and terminate it with *__pragma__ ('endif')*.

Important: The conditional compilation pragma's also work when placed at the start of a line in precompiled or hand-written JavaScript code.

.. _pragma_js:

Inserting literal JavaScript: __pragma__ ('js', ...) and __include__ (...)
--------------------------------------------------------------------------
During compilation the *__pragma__ ('js', code, <format parameters>)* is replaced by the JavaScript code given in the *code* parameter. This code is formatted using the Python *str.format* method, using *<format parameters>*.

An example of its use is to encapsulate a JavaScript library as a Python module, as is :ref:`shown  for the fabric.js library <code_encaps_fabric>`. In that case there's usually one format parameter, namely a call to *__include__ (<relative module path>)*. The module path is either relative to the directory holding the main module of your project, or to the root of the modules directory, and searched in that order. So modules local to your project prevail over generally available modules.

Note that since {} is used as placeholder in Python formatting, any normal { and } in your JavaScript in the code parameter have to be doubled. If you just want to include a literal piece of JavaScript without any replacements, you can avoid this doubling by using __pragma__ ('js', '{}', <my_piece_of_JS_code>). 

Using an external transpiler: __pragma__ ('xtrans', <translator>, ..., cwd = <workingdirectory>)
------------------------------------------------------------------------------------------------
This pragma works the same as *__pragma__ ('js', ...)*, only now an external translator application can be specified, e.g. to translate JSX code.
If the *cwd* parameter is present, it the external translator will use it as working directory.
The code offered to this pragma is routed through this external translator by means of pipes.
This pragma can use *__include__ (...)* just like *__pragma__ ('js', ...)*.

Example:

.. literalinclude:: ../../development/manual_tests/xtrans/test.py
	:tab-width: 4
	:caption: Test program using __pragma__ ('xtrans', ...)
    
.. literalinclude:: ../../development/manual_tests/xtrans/change_case.cpp
	:tab-width: 4
	:caption: Simple external translator change_case.cpp, changing the case to upper or lower, depending on -l switch
    
.. literalinclude:: ../../development/manual_tests/xtrans/test_for_docs.js
	:tab-width: 4
	:caption: The resulting translated file test.js

Create bare JavaScript objects and iterate over their attributes from Python: __pragma__ ('jsiter') and __pragma__ ('nojsiter')
-------------------------------------------------------------------------------------------------------------------------------
Normally a Python *{...}* literal is compiled to *dict ({...})* to include the special attributes and methods of a Python dict, including e.g. an iterator. When *__pragma__ ('jsiter')* is active, a *Python {...}* literal is compiled to a bare *{...}*, without special attributes or methods. To still be able to iterate over the attributes of such a bare JavaScript object from Python, when *__pragma__ ('jsiter')* is active, a Python *for ... in ...* is literally translated to a JavaScript *for (var ... in ...)*. The main use case for this pragma is conveniently looping through class attributes in the *__new__* method of a metaclass. As a more flexible, but less convenient alternative, *__pragma__ ('js', '{}', '''...''')* can be used.

An example of the use of this pragma is the following:

.. literalinclude:: ../../development/automated_tests/transcrypt/metaclasses/__init__.py
	:start-after: from org.transcrypt.stubs.browser import __pragma__
	:end-before: class Uppercaser (metaclass = UppercaserMeta):
	:tab-width: 4
	:caption: Use of __pragma__ ('jsiter') and __pragma ('nojsiter') to manipulate class attributes in a metaclass

__pragma__ ('jskeys') and __pragma__ ('nojskeys')
-------------------------------------------------
Normally in Python, dictionary keys without quotes are interpreted as identifiers and dictionary keys with quotes as string literals. This is more flexible than the JavaScript approach, where dictionary keys with or without quotes are always interpreted as string literals. However to better match the documentation and habits with some JavaScript libraries, such as plotly.js, dictionary keys without quotes can be optionally interpreted as string literals. While the -jk command line switch achieves this globally, the preferred way is to switch on and off this facility locally.

__pragma__ ('keycheck') and __pragma__ ('nokeycheck')
-----------------------------------------------------
When *__pragma__ ('keycheck')* is active, an attempt to retrieve an element from a dictionary via a non-existing key results in a KeyError being raised, unless it happens at the lefthand side of an augmented assignment. The run time support for this is expensive, as it requires passing the result of every such retrieval to a function that checks the definedness of the result. While the -kc command line switch switches key checking on globally, the preferred way is to switch on and off this facility locally to prevent code bloat.

Keeping your code lean: __pragma__ ('jsmod') and __pragma__ ('nojsmod')
-----------------------------------------------------------------------
When *__pragma__ ('jsmod')* is active, *%* has JavaScript rather than Python semantics. While the -jm command line switch achieves this globally, the preferred way is to switch on and off this facility locally.

Keeping your code lean: __pragma__ ('kwargs') and __pragma__ ('nokwargs')
-------------------------------------------------------------------------
While it's possible to compile with the -k command line switch, allowing keyword arguments in all flavors supported by Python 3.5 in all places, this disadvised, as it leads to bloated code. It is better to use the 'kwargs' and 'nokwargs' pragmas, to enable this feature only at definition (as opposed to calling) of functions that require it. You'll find an example of how to use these pragma's in the :ref:`arguments autotest <autotest_arguments>`. You can use them on whole modules or any part thereof. Note that at due to the dynamic nature of Python, use of keyword arguments at call time cannot be predicted at definition time. When running with CPython from the command prompt using the browser stubs, these pragma's are ignored.

Preventing target annotation: __pragma__ ('noanno')
---------------------------------------------------
The -a command line switch will comment target code compiled from Python with source file names and source line numbers. Since this interferes with literal inclusion of JavaScript code with multi-line comments, it can be switched off by including *__pragma__ ('noanno')* at the beginning of a module. An example of this can be seen in the :ref:`encapsulation code for the fabric.js library <code_encaps_fabric>`.

Operator overloading: __pragma__ ('opov') and __pragma__ ('noopov')
-------------------------------------------------------------------
Transcrypt currently supports overloading of the following operators: \*, /, +, -, @, [], (), ==, !=, <, <=, >, and >=.

These operators have been chosen since they can enhance the readability of computations involving matrices and vectors, enable the use of callable objects (functors) or are used to compare tuples, lists and sets. Using the -o command line switch will activate operator overloading globally. *This is strongly disadvised*, since even 1 + 2 will result in two function calls in that case. It's better to use *__pragma__ ('opov')* to switch it on locally and *__pragma__ ('noopov')* to switch it off again, activating operator overloading only for lines of code involving extensive matrix / vector computations or functor calls (as opposed to definitions), in which case the overhead is negligeable in most cases.

Formula v4 = M \* (v1 + v2) + v3 is probably preferred over v4 = add (mul (M, add (v1, v2), v3)), which by the way closely resembles the JavaScript that will be generated for the expression that uses overloaded + and \* operators. In order to support operator overloading your matrix and vector classes have to support the appropriate selection of functions out of *__mul__*, *__rmul__*, *__div__*, *__rdiv__*, *__add__*, *__radd__*, *__sub__*, *__rsub__*, *__matmul__*, *__rmatmul__*, *__getitem__*, *__setitem__* and *__call__* as described in the CPython documentation. The comparison operators are supported by the functions *__eq__*, *__ne__*, *__lt__*, *__le__*, *__gt__* and *__ge__*.

.. _skipping_fragments:

Skipping fragments while generating code : __pragma__ ('skip') and __pragma__ ('noskip')
----------------------------------------------------------------------------------------
On some occasions it's handy to be able to skip code generation for fragments of Python source. The most important use case is when you perform a static check using the :ref:`-c or --check <command_line_switches>` command line switch. If you use global JavaScript names without importing them from some encapsulation module, you'll have to tell the static checker that these names are not to be flagged as undefined. Insert a line containing dummy definitions of the unknown names for the static checker, surrounded by *__pragma__ ('skip') and __pragma__ ('noskip')*, to make code generator skip these definitions. The beginning of the source code of the Pong demo, shown below, demonstates how this is done.

.. literalinclude:: ../../demos/pong/pong.py
	:end-before: class Attribute:
	:tab-width: 4
	:caption: Preventing complaints of the static checker about unknown global JavaScript names in pong.py

Automatic conversion to truth value: __pragma__ ('tconv') and __pragma__ ('notconv')
------------------------------------------------------------------------------------

When automatic conversion to truth value is activated, an empty array, dict or set has truth value False just as in CPyton, rather than True, as would be the case natively in JavaScript. This comes at the expense of a runtime typecheck and is by default avoided in Transcrypt for that reason. To switch on automatic conversion to truth values locally, *__pragma__ ('tconv')* and *__pragma__ ('notconv')* can be used. The alternative is to switch on automatic conversion globally using the -t command line switch. Use of this switch is disadvised if a lot of boolean expressions are computed in inner loops, since it would result in many extra typechecks. When designing performance sensitive libraries, it's advisable to use *__pragma__ ('notconv')* explicitly at the start of each performance-sensitive module. The result will be that even when an application developer chooses to use the -t switch, performance of won't suffer.

Adding directories to the module search path: __pragma__ ('xpath', <directory list>)
------------------------------------------------------------------------------------

In addition to the :ref:`regular module mechanism <module_mechanism>` you can use *# __pragma__ ('xpath', <directory list>)* to add directories to the module search path. Note that resolving search paths happens compile time, so dynamic manipulation of the module search path doesn't apply. If you want to alter the search path in code that is used both by Transcrypt and CPython, you can additionally manipulate *sys.path* in a *try* clause or under an *if* with condition *__envir__.executor_name == __envir__.interpreter_name*.

The *-xp / --xpath* :ref:`command line switch <command_line_switches>` has the same effect.