.c: Standard superkinds

.import:: boolean+, Any+, dict+, Exception+, iterable+, string+, type+
..from: kindnames

.c: Standard kinds
.import:: Any, boolean, int, frame, iterator, list, None
..from: kindnames

.import:: ObjectClassifier, NodeSet, RelationStructure, HeapView, NodeGraph
..from: kindnames

.import:: NodeGraph+, NodeSet+, ObjectClassifier+,
    type_with_hiding_tag+
..from: kindnames

.superkind:: HeapDefs+
..eg: ()

.superkind:: command+
..eg: "print 'hello'"

.kind:: thread_id
..subkind of: int

.superkind:: thread_id+
..superkind of: thread_id

.superkind:: TargetSpec+

.kind:: module heapyc
..d: The heapyc module.
..attr:: HeapView
...mapping
....returns: HeapView
.....d: a new HeapView object.
....arg: root:Any+
.....d: The initial value of the root member.
....arg: heapdefs:HeapDefs+
.....d: Definitions of specially treated extension types.

..attr:: interpreter
...mapping
....d: Create a new interpreter structure with a new thread.
....arg: command:command+
.....d: A command that will be exec'd in the new environment.
....optionals
.....arg: locals: dict+
......d: Local variables passed to the command when exec'd.
....returns: thread_id
.....d: the thread identity number.
....d
The new interpreter and thread is started in a new environment.  This
environment consists of a new '__main__' module, with the optional
locals dict as local variables.
....d
The site-specific initializations are not automatically made. To do
that, 'import site' could be used as the first statement in the
command string.
....d
The interpreter() function will return after the new thread structure
has been created. The command will execute sooner or later.  The
thread will terminate, and the interpreter structure be deallocated,
when the command has been executed, and dependent threads have
terminated.

..attr:: NodeGraph
...mapping
....d: Construct a new NodeGraph object.
....optionals
.....arg: edges:iterable+
......d: The edges that will be used to
         initialize the new nodegraph. It should yield a
         sequence of pairs being edges of the form (source, target).
......default: ()
.....arg: is_mapping:boolean+
......d: If True, will cause the nodegraph
         to be treated like a 'mapping'. It will then, for the
         purpose of indexing, be expected to contain a single
         target for each source node.
......default: False
....returns: NodeGraph

..attr:: RootState
...d: The single instance of RootStateType.

..attr:: set_async_exc
...mapping
....d: Set an exception to be raised asynchronously in a thread.
....arg: thread_id:thread_id+
....arg: exception:Exception+

..attr:: xmemstats
...mapping
....d: Print system-dependent memory statistics.
....d: What is printed depends on the system configuration.

.and: HeapView
..self: HV

..d
...p
A HeapView object provides methods to get memory related information
about the system heap and about individual objects. 

...p
It implements much of the low-level functionality for the Heapy
system. It is intended to provide what can not be done at all or would
be much slower if programmed directly in Python. It is not intended to
be used directly by a user, but to be wrapped in higher level objects.

...p
Some terms that are referred to in the method descriptions:

...label: Visible objects
...h3: Visible objects.
...p
The HeapView object attempts to restrict its view of the heap to only
the 'visible objects'. This is to make it possible to analyse the heap
via a Python library that inevitably itself is continually allocating
and deallocating objects. These should be hidden from the heap view
presented. This is primarily done via a special tag attribute, see
....ref: .mykind._hiding_tag_
....t: and
....ref: .mykind.register__hiding_tag__type
....t:. Frames can be hidden
with another mechanism, see
....ref: .mykind.limitframe
....t:. For hiding all objects of a special type,
....ref: .mykind.register_hidden_exact_type
....t: may be used. It is also
possible to use a separate interpreter and hide its root objects, see
....ref: .mykind.is_hiding_calling_interpreter
....t:.

...h3: Classifiers.
...text
The methods named cli_* are factory methods that create objects of
type ObjectClassifier. The principal difference between classifiers is
how a single object is classified. The single-object classification
function is available in classifier objects; it is the classify
method. There are also methods that operate on collections of objects,
namely partition and select. These eliminate the per-object
Python-level function call overhead that would occur if the classify
method were to be called from Python for each object in a collection.
See also the ObjectClassifier type.

...label: Individual size
...h3: Individual size.
...p
The individual size of an object is its individually allocated memory size. 

...p
It includes:

...ul
....li: The basic object size, as can be found out in a standard way.
....li: The extra memory for variable size objects.
....li: For GC collected objects, the size of the GC information.
....li: An alignment to the next highest multiple of a pointer size.
....li: The size of any other memory allocated that belongs to the object.
...p

Some types of objects have extra memory allocated that can not be
accounted for in the standard way. This memory should nevertheless be
included in the individual size. To determine the size of these
objects, special functions are needed. These are defined for standard
builtin types, such as lists and dicts. Other types should be defined
via the heapdefs argument to the HeapView constructor.

...p: The individual size does not include:
...ul
....li: Subobjects that are accounted for separately.
....li: Overhead for the memory allocation system. This varies depending
  on the kind of memory allocator, the requested size, etc.

..constructor: module heapyc.HeapView

..attr:: _hiding_tag_
...self: HV
...d: The hiding tag defining what objects are hidden from the view defined
by HV. Objects that contain a _hiding_tag_ object which is identical
to HV._hiding_tag_, will be hidden from view, in the following cases:
....ul
.....li: The object is of a type that has been registered for hiding via
......ref: .mykind.register__hiding_tag__type
......t:, or is of a subtype of such a type.
.....li: The object is of instance type. Such an object will be checked
  for a _hiding_tag_ item in its __dict__.

..attr:: cli_class
...mapping
....returns: ObjectClassifier
.....d: a classifier that classifies by "class".
.....d:
If an object is of instance type, its classification will be its
class, as given in its C-level member 'in_class', otherwise it will
be its type, as given in 'ob_type'.
....dwh: Note
This is like classifying by the Python-level '__class__' attribute
in newer versions of Python, except that even though the '__class__'
attribute may be redefined via the class definition, the classification
as made by cli_class isn't affected.

..attr:: cli_clodo
...mapping
....returns: ObjectClassifier
.....d: a classifier, that classifies by "Class Or Dict Owner".
.....d
The classification of an object is its class (as by cli_class),
unless the object is a dict object that is 'owned' by some owner.
If the object is such an owned dict, the classification will be a
1-tuple containing the class (as by cli_class) of its owner.

....arg: owners:NodeGraph+
.....d:
Used to map each dict object to its owner, or to None if it has no
owner. The graph will be automatically updated, from heap information
defined by HV, whenever an attempt is made to classify a dict that
maps to nothing.
....arg: memo:dict+
.....d: Used to memoize the 1-tuples  generated when classifying owned dicts.

..attr:: cli_indisize
...mapping
....returns: ObjectClassifier
.....d: a classifier that classifies by "individual size".
.....d: The classification of each object is an int, containing the
object's individual memory size.
....arg: memo:dict+
.....d: Used to memoize the classification objects.

..attr:: cli_none
...mapping
....returns: ObjectClassifier
.....d: a classifier that classifies all objects the same.
.....d: The classification of each object is None.

..attr:: cli_rcs
...mapping
....returns: ObjectClassifier
.....d: a classifier that classifies by Referrer Classification Set.
.....d: The classification of an object is the classifications of its
referrers, collected in an immutable NodeSet object. 

....arg: referrers: NodeGraph+
.....d: Used to map each object to its referrers.
....arg: classifier:ObjectClassifier+
.....d: Used to classify each referrer.
....arg: memo:dict+
.....d: Used to memoize the classification sets.

..attr:: cli_type
...mapping
....returns: ObjectClassifier
.....d: a classifier that classifies by type.
.....d:
The classification of each object is the type, as given by its
C-level member 'ob_type'. (This is the same as the type returned
by the Python-level builtin 'type'.)

..attr:: delete_extra_type
...description: For Internal Use

..attr:: heap
...mapping
....args
....returns: NodeSet
.....d: a set containing all 'visible objects' in the heap view
defined by HV.
....dwh: See also
.....ref: .mykind.Visible objects
......t: Visible objects

..attr:: indisize_sum
...mapping
....arg: S:iterable+
....returns: int
.....d: the sum of the 'individual size' of the objects in S.
....dwh: See also
.....ref: .mykind.Individual size
......t: Individual size

..attr:: numedges
...mapping
....arg: src:Any+
....arg: tgt:Any+
....returns: int
.....d: the number of edges from src to tgt.

..attr:: reachable
...mapping
....arg: X:NodeSet+
....arg: Y:NodeSet+
....returns: NodeSet
.....d: the set of objects reached via a path in the visible heap as
defined by HV, from some object in X, avoiding any object in Y.

..attr:: reachable_x
...mapping
....arg: X:NodeSet+
....arg: Y:NodeSet+
....returns: NodeSet
.....d: the set of objects reached via a path in the visible heap as
defined by HV, from some object in X, avoiding any object in Y except
at the end of the path.

..attr:: register__hiding_tag__type
...mapping
....args: type_with_hiding_tag+
....d:
Register a type of objects that may be hidden from the heap view
defined by HV. The type must have a slot named _hiding_tag_. An object
that is an instance of the type, or of a subtype, is hidden when its
_hiding_tag_ is HV._hiding_tag_.

..attr:: register_hidden_exact_type
...mapping
....args: type+
....d:
Register a type of objects that should be hidden from the heap view
defined by HV. Objects of the exact type registered -- not including
subtypes -- will be hidden.

..attr:: relate
...mapping
....arg: src:Any+
....arg: tgt:Any+
....returns: RelationStructure
.....d: a description of the relation between src and tgt.
.....d: This is used for descriptions of edges in paths.
.....d: [The result is in a special format that I choose to not define here
since it is for special low-level use and subject to change.]

..attr:: relimg
...mapping
....returns: NodeSet
.....d: the 'relational image of HV wrt S'.
.....d: That is, the set of nodes
that are directly referred to from the nodes in S via the visible heap
reachability relation as defined by HV.
....arg: S:iterable+

..attr:: shpathstep
...mapping
....d: This method implements one step of a shortest path algorithm.

....arg: G:NodeGraph+
.....d:         Updated by the method, with the edges from nodes in the
                source set to the new nodes visited.

....arg: U:NodeSet+
.....d: The source set for this step.

....arg: S:NodeSet+
.....d: The set of already visited nodes.

....optionals
.....arg: AvoidEdges:NodeGraph+
......d: Edges to avoid.

.....arg: find_one:boolean+
......d: If True, at most one edge will be found from each node
         in the source set. Normally, all edges will be found.

....returns: NodeSet
.....d: the new nodes visited.
.....d: This may be used for the U argument the next time the method is called.

....dwh: See also
.....t: shpgraph_algorithm in Path.py.

..attr:: update_dictowners
...mapping
....d: Update owners with ownership edges.
....d: The dict owners graph will be updated with an edge from each dict
object in the heap, to either its owner or to None.

....arg: owners:NodeGraph+

..attr:: update_referrers
...mapping
....d: Update referrer graph X for Y.
....d: The visible heap defined by HV will be traversed from the root of HV
so that the edges of every path from the root to nodes in Y will be
represented, inverted, in X.
....arg: X:NodeGraph+
....arg: Y:NodeSet+

..attr:: update_referrers_completely
...mapping
....arg: X:NodeGraph+
....d: Update referrer graph X 'completely'.
....d: [Experimental algorithm that updates X with the referrers to all
objects in the heap (of visible nodes as defined in HV). It is not
normally used.]

..attr:: is_hiding_calling_interpreter
...kind of: boolean
...d:
If True, the data of the interpreter using the HV will be hidden from
the heap view as seen from RootState.
...d
This is used when multiple Python interpreters are used. One
interpreter will be monitoring the operation of the other
interpreter(s). It would set is_hiding_calling_interpreter to True in
the HV it is using. Its own data will then be hidden from view, making
memory leak detection more practical.

..attr:: is_using_traversing_owner_update
...kind of: boolean
...d:
True if update_dictowners is using a recursive traversal algorithm to
find the dicts in the heap. When False, the normal case, it will use the
dicts found in the gc collection structure, by gc.get_objects(). This was
found to be much faster in usual cases, but the old version is available
by setting this flag. -- It may be removed in a later release!


..attr:: limitframe
...either: None, frame
...d: The traversal limiting frame.
...d: If limitframe is set to a frame object, the frames that are more
recently entered than limitframe will be hidden when traversing the
heap from the root RootState. It will start traversing from limitframe
rather than from the most recent frame as it would otherwise do.

.and: NodeGraph
..d
A NodeGraph object contains pairs of nodes (edges) and can be indexed
on the first node of such a pair (the source of an edge) to find all
second nodes of such pairs (the targets of those edges).
...p
NodeGraph objects are used internally in the Heapy system, for example
to record dict ownership and shortest-path graphs.
...p
They may be used generally for mapping and dict-like purposes, but
differ from dicts in the following:
...ul
....li: The mapping is based on object identity - no equality or hashing is
  assumed, so any object can be used as a key. Only the address is used.
  To distinguish this usage from that of ordinary dicts and sets, such
  objects are called 'nodes'.
....li: There may be any number of targets associated with each source.
....li: Performance characteristics differ from dicts, in somewhat subtle ways.

..self: NG

..constructor: module heapyc.NodeGraph

..attr:: _hiding_tag_
...d: The hiding tag: if it is the the same object as
....ref: .myfile.HeapView._hiding_tag_
....t: of a HeapView object, the nodegraph will be hidden from the corresponding heap view .

..attr:: is_mapping
...kind of: boolean

...d: Read only. True if NG is a 'mapping'. Then, only one edge is
allowed for each source; indexing returns the actual target object
instead of a tuple of targets.

..attr:: is_sorted
...kind of: boolean
...d: Read only. True if NG is sorted.

...d: It will become unsorted after any update. It will need to be
sorted to make it possible to find edges (implementation uses binary
search). Any indexing operation will automatically sort it if it was
not already sorted.  The flag is currently used from Python to see if
the nodegraph has been used at least once after update, so that it
will not be cleared too early.

..fop: len
...returns: int
....d: the number of edges in NG.

..getitem
...returns: Any
....d: the target(s) of all edges with a particular source. The value
depends on if NG was initialized to be a 'mapping' or not:
.....dl
......dt: NG.is_mapping == False
......dd: Return a tuple of all targets of edges from the source.
          The tuple will be empty if there are no such edges.
......dt: NG.is_mapping == True
......dd: Return the target of the edge from the source.
          If there is no such edge, KeyError will be raised.
          If there is more than one edge, ValueError will be raised.
...arg: source: Any+

..setitem
...d: Assign new target(s) to the edge(s) from a given source. How the
targets are specified, depends on whether NG is a 'mapping' or not:
....dl:
.....dt: NG.is_mapping == False
.....dd: The target_spec argument is a tuple of targets.
    There must already be the same number of edges in NG from the given
    source as the number of objects in target_spec.
.....dt: NG.is_mapping == True
.....dd: The target_spec argument is the target itself.
    There must already be exactly one edge in NG from the given source.
...d:
If there were not already the same number of edges as the targets specified,
ValueError will be raised. The reason for the restriction is to make sure
assignment will be efficient when interleaved with indexing.
...dwh: See also
....ref: .mykind.add_edge
....t:,
....ref: .mykind.add_edges_n1
...arg: source: Any+
...arg: target_spec: TargetSpec+

..method:: add_edge
...d: Add to NG, an edge from source to target.
...arg: source: Any+
...arg: target: Any+

..method:: add_edges_n1
...d: Add to NG, for each src in srcs, an edge from src to tgt.
...arg: srcs: iterable+
...arg: tgt: Any+

..method:: as_flat_list
...returns: list
....d: the edges of NG in the form [src0, tgt0, src1, tgt1 ...].

..method:: clear
...d: Remove all items from NG.

..method:: copy
...returns: NodeGraph
....d: a copy of NG.

..method:: domain_covers
...returns: boolean
....d: True if each node in X is the source of some edge in NG, False otherwise.
...arg: X: iterable+

..method:: domain_restricted
...returns: NodeGraph
....d: a new NodeGraph, containing those edges in NG that have the source in X.
...arg: X: iterable+

..method:: get_domain
...returns: NodeSet
....d: the set of nodes that are the source of some edge in NG.

..method:: get_range
...returns: NodeSet
....d: the set of nodes that are the target of some edge in NG.

..method:: invert
...d: Invert the edges of NG.

..method:: inverted
...returns: NodeGraph
....d: a copy of NG with the edges inverted.

..fop: iter
...returns: iterator
....d: an iterator yielding a pair (source, target) for each edge in NG.

..method:: relimg
...returns: NodeSet
....d: the relational image of NG wrt X.
....d: That is, the set of nodes that are the target of some edge that have
       its source in X.
...arg: X:iterable+

..method:: update
...d: Update NG with the edges from X,
specified as pairs of the form (source, target).
...arg: X:iterable+

..method:: updated
...returns: NodeGraph
....d: a copy of NG updated with the edges from X,
specified as pairs of the form (source, target).
...arg: X:iterable+



