"""This module defines classes and functions to
read files with individual results
from phoneme identification experiments.

Phoneme-identification data may be collected for
several test subjects in one or more group_factors.
Each subject may be tested in one or more test-conditions.

*** Class Overview:

StimGroup: defines ONE (sub-)group of stimuli and allowed responses

ConfMatrixFrame: defines structure of a phoneme-identification experiment,
    with StimGroup instance(s) specifying all stimuli and responses,
    and specifies participant group_factors and test conditions to be included in the analysis.

StimRespCountSet: all phoneme-identification data for
    selected group(s), subjects, and test conditions,
    to be used as input for statistical analysis.
    Each subject should be tested in most (or all) predefined test conditions.

Data may be stored in various table-style file formats allowed by package Pandas,
e.g., xlsx, csv, odt, etc.
Each data file may include stimulus-response data recorded
in ONE or SEVERAL test conditions,
from ONE or SEVERAL participants,
recruited from ONE or SEVERAL populations,

Stimulus and corresponding responses are preferably stored in a "long" format,
with one table row for each (stim, resp) pair, and
one column for the presented stimulus categories,
one column for the corresponding response category,
and (optionally) another column stating the number of occurrences of each (stim, resp) pair.

For backward compatibility, response data may also be stored in "wide" table format,
as a conventional confusion matrix, where each cell shows the count for one (stim, resp) pair.
The stimulus and response categories must then be identified by one header row and one header column.

Each table row may correspond to one stimulus category, and one column for each response category,
or vice versa, as a transposed matrix.

The participant id may be stored in a designated column of the table,
or the file name may be used as participant id,
or (in Excel-type files) the participant may be identified by the sheet name.

Group category(-ies) and test conditions may be specified in table column(s),
OR defined by sub-string(s) in the absolute path for each file,
OR (in Excel-type files) by the sheet name.

Groups may be specified by ONE or SEVERAL group factors ("dimensions"), e.g.,
"Age", with categories 'old' or 'young', and "Gender" with categories 'm' or 'f'.

Test conditions can also be specified in ONE or SEVERAL factors ("dimensions"), e.g.,
"Hearing Aid" with categories 'program A', 'program B', etc.

All group and test-factor categories, and all stimulus and response categories,
MUST agree exactly with string labels predefined by the user in a ConfMatrixFrame instance.

All primary data collected in a StimRespCountSet instance
can be saved in various file formats that Pandas supports.

*** Input Data Files:

All input files from an experiment must be stored in ONE directory tree.

If a group dimension is specified by the file path, the path must include a sub-string
combining the group dimension and ONE allowed category,
for example, 'Age_old', if the group dimension 'Age' has allowed categories 'young' and 'old'.
The same style is required for any test-factor category specified in the path string.

Each participant in the same group MUST have a unique participant ID.
Participants in different group_factors may have the same participant ID labels,
because the group_factors are separated anyway,
but normally the participant IDs should be unique across all group_factors.


*** Example Directory Tree:

Assume we have data files in the following directory structure:
~/cm_study / Age_old / Gender_male, containing files Test_64.xlsx and Response_Data_LAB_64.xlsx
    containing data to be analyzed for group id = ('old', 'male') in dimensions ('Age', 'Gender').
~/cm_study / Age_old / Gender_female, containing files Test_64.xlsx and Data_CM_65.xlsx
~/cm_study / Age_young / Gender_male,  containing files Test_64.xlsx and CM_65.xlsx
~/cm_study / Age_young / Gender_female, containing files Test_64.xlsx and CM_65.xlsx

Four separate populations / group_factors are then defined by categories in dimensions 'Age' and 'Gender'.
The analysis can also be restricted to only use data in files with names including 'Test_64',
if a separate group- or test-factor "Test" is predefined with only one possible category = '64'.

Files for different test conditions may be similarly stored in separate subdirectories.
The directory name can then be used to indicate a test-factor category,
if the category is not defined in the data file itself.

Each data file may contain phoneme-identification results
for one or more test subject(s) in one or more test-conditions.

One or more separate files may include data for the same participant,
e.g., results obtained in different test sessions.

Results from different participants may be stored in the same file,
if the participant is identified by a separate table column (or by the sheet name).

File names are arbitrary, although they may be somehow associated with
the encoded name of the participant, to facilitate data organisation.

All input data are collected by a single function call, for example, as
ds = StimRespCountSet.load(cmf, top_dir, ...)
Keyword arguments ... specify how results are stored in the input tables.
Special keyword arguments can also control how Pandas reader functions parse the data.
See examples in script run_cm.py

If, for example, fmt='csv', the load method reads only files with the given suffix.
If the fmt argument is left unspecified, or fmt=None,
StimRespCountSet.load will attempt to read ALL files in the designated directory tree,
and use all files that seem to contain phoneme-identification results,
provided that the tables are organised in the same way, regardless of the file type.


*** Version History
* Version 0.8:
2026-02-21, ConfMatrixFram.population_weight for merged populations,
            to display results in subset of group_factor dimension
2025-12-08, StimRespCountSet.load tested with both wide and long table formats
2025-12-01, accept path-string category like 'old', if desired pattern 'Age_old' not found.
2025-11-28, Using pandas for file input and internal storage,
            new StimRespCountSet.load(...) method
2025-11-23, test-condition id is always a tuple with one or more category labels,
            one category from each test-factor "dimension", in the pre-defined order.
2025-10-29, group id is always a tuple with zero, one, or more category labels,
            one category from each group_factor "dimension", in pre-defined order.
2025-10-18, changed internal structure of ConfMatrixFrame, new setup method

* Version 0.7:
2018-09-15, first functional version
2020-03-22, StimRespCountSet.get_response_count returns counts in array form,
    regardless of internal storage format.
"""
import numpy as np
from pathlib import Path

from collections import Counter  # , namedtuple
from itertools import product  #, chain
import pandas as pd
from difflib import get_close_matches

import logging

from .cm_file import stim_resp_gen, Table, FileReadError, FileWriteError


logger = logging.getLogger(__name__)

# Default values:
COUNT_TYPE = np.uint32  # data type for all (stim, resp) counts  *** -> cm_base?
COUNT_COL = 'Count'  # default count column header

STIM_GROUP_HEAD = 'StimGroup'
STIM_HEAD = 'Stim'
RESP_HEAD = 'Resp'


# -------------------------- Exceptions:
class TableFormatError(FileReadError):
    """Any format error causing non-usable data from a file
    """

class UserInputError(RuntimeError):
    """Any kind of user-input mismatch.
    """
# -------------------------------------

MISSING_LIMIT = 0.3
# = proportion of missing subject test conditions to trigger a warning


# ----------------------------------------------------------------
# Help classes: Categorical Data Types used in ConfMatrix analysis.

# Category labels should preferably be (short) string literals,
# identical to the codes stored in data records.
# Integer labels can also be used, but this can cause some ambiguity:
# Pandas.reader functions might re-interpret labels,
# e.g., string '3' might be read as the number 3.


# ---------------------------------------------- Help classes:
class NominalDtype(pd.CategoricalDtype):
    """Data type used for group and test-factor categories.
    """
    def __init__(self, cats):
        """
        :param cats: sequence of string (or integer) labels
        """
        super().__init__(categories=cats, ordered=False)


class StimGroup:
    """Defines (stim, resp) for ONE confusion-matrix (sub-)block, where
    stim and resp are both sequences of string labels, such that
    ANY element of resp is allowed as response for EVERY element of stim.
    Thus, stim and resp may be row and column indices of a confusion count matrix.
    """
    def __init__(self, stim, no_response=None):
        """
        :param stim: sequence of stimulus labels
        :param no_response: (optional) label for missing response, if allowed
            or sequence of such labels with same meaning
        """
        self.stim = list(stim)
        if no_response is None:
            no_response = []
        if type(no_response) is not list:
            no_response = [no_response]
        self.no_response = no_response

    @property
    def resp(self):
        return self.stim + self.no_response

    @property
    def shape(self):
        return len(self.stim), len(self.stim) + len(self.no_response)


# ---------------------------------------------- Main classes:
class ConfMatrixFrame:
    """Defines layout of a phoneme-identification experiment,
    and specifies stimulus-response data to be analysed.
    """
    def __init__(self,
                 stim_groups,
                 stim_group_head=STIM_GROUP_HEAD,
                 stim_head=STIM_HEAD,
                 resp_head=RESP_HEAD,
                 stim_count=None,
                 test_factors=None,  # **** allow reduced index defined by available data ? ***
                 group_factors=None,  # *** some combinations might have no data, allowed!
                 population_weights=None # needed only to merge population models
                 ):
        """
        :param stim_groups: dict with elements (stim_group_key, StimGroup instance)
        :param stim_group_head: (optional) name of index level for stimulus group_factors
        :param stim_head: (optional) name of index level for allowed stimulus categories
        :param resp_head: (optional) name of index level for allowed response categories
        :param stim_count: (optional) scalar, OR nested dict: {stim_group:{stim: count}}
        :param test_factors: (optional) dict with elements (test_factor, categories), where
            categories is a sequence defining NOMINAL categories within this test_factor.
        :param group_factors: (optional) dict with elements (dimension, categories), where
            dimension is a string label identifying one grouping "dimension", e.g., 'Age'
            categories is a sequence defining NOMINAL categories within this dimension.
            May be empty if there is only one unnamed group of participants.
        :param population_weights: (optional) dict or iterable with elements (group_key, weight value)
            for relative weights of cm_model.PopulationModel instances in case they are merged for display.
            Weights are = 1. for populations that are not explicitly specified.
        """
        self.stim_groups = stim_groups
        self.stim_group_head = stim_group_head
        self.stim_head = stim_head
        self.resp_head = resp_head
        self.stim_count = stim_count
        if test_factors is None:
            test_factors = dict()
        self.test_factors = test_factors
        if group_factors is None:
            group_factors = dict()  # NO group comparisons desired
        self.group_factors = group_factors
        if population_weights is None:
            population_weights = dict()
        # ensure population_weights.keys() match with group_factors:
        pop_keys = [self.match_group_key(pop_k)
                    for pop_k in population_weights.keys()]
        for (pk_new, pk_old) in zip(pop_keys, population_weights.keys()):
            if pk_new != pk_old:
                population_weights[pk_new] = population_weights.pop(pk_old)
                logger.warning(f'Population key {pk_old} changed to {(pk_new,)}.')
        self.population_weights = population_weights

    def __repr__(self):
        return (self.__class__.__name__ +  '(\n\t' +
                ',\n\t'.join(f'{key}={repr(v)}'
                             for (key, v) in vars(self).items()) +
                '\n\t)')

    @classmethod
    def setup(cls,
              stim_groups,
              no_response=None,
              stim_group_head=STIM_GROUP_HEAD,
              stim_head=STIM_HEAD,
              resp_head=RESP_HEAD,
              stim_count=None,
              test_factors=None,
              group_factors=None,
              population_weights=None
              ):   # *** keep for ease of use? equiv to __init__(stim_groups = {sg: StimGroup(stim_list), ...}
        """
        :param stim_groups: dict with elements (key, stim_list), where
            key = a label for one subgroup of stimuli
            stim_list = list of strings, one for each test item in the group, e.g., a phonetic symbol
                MUST agree with stimulus labels found in input files.
        :param no_response: (optional) label for missing response, if allowed
        :param stim_group_head: (optional) name of index level for stimulus group_factors
        :param stim_head: (optional) name of index level for allowed stimulus categories
        :param resp_head: (optional) name of index level for allowed response categories
        :param stim_count: (optional) scalar, OR nested dict: {stim_group: {stim: count}}
        :param test_factors: dict with elements (test_factor, category_list), where
            test_factor is a string,
            category_list is a list of labels for allowed categories within test_factor.
        :param group_factors: dict with elements (dimension, cat_list), where
            dimension is a string label identifying one grouping "dimension", e.g., 'Age'
            cat_list is a list defining categories within this dimension.
        :param population_weights: (optional) dict or iterable with elements (group_key, weight value)
            for relative weights of cm_model.PopulationModel instances in case they are merged for display.
            Weights are = 1. for populations that are not explicitly specified.
        :return: a cls instance
        """
        conf_groups = {sg: StimGroup(sg_list, no_response)
                       for (sg, sg_list) in stim_groups.items()}
        return cls(stim_groups=conf_groups,
                   stim_group_head=stim_group_head,
                   stim_head=stim_head,
                   resp_head=resp_head,
                   stim_count=stim_count,
                   test_factors=test_factors,
                   group_factors=group_factors,
                   population_weights=population_weights)

    @property
    def n_test_factor_conditions(self):   # needed ? ***** -> test_factor_shape
        """1D list with number of test-condition alternatives in each test factor"""
        # return [len(v.categories) for v in self.test_factors.values()]
        return [len(v) for v in self.test_factors.values()]

    @property
    def n_test_conditions(self):
        """Total number of actually used test conditions,
        *** future: possibly subset of product of test_factors.categories
        """
        return np.prod(self.n_test_factor_conditions, dtype=int)

    def test_conditions(self):
        """Generator of all test-condition tuples
        :return: iterator of test-conditions.
            Each element is always a tuple, even if only one test_factor
        """
        return product(*(tc for tc in self.test_factors.values()))

    def test_cond_index(self):  # **********************
        """pandas.Index or MultiIndex with one level for each test factor (dimension),
        :return: sri = a pd.MultiIndex instance
        """
        # *** allow to set reduced index values after data load ? ***
        if len(self.test_factors) > 1:
            return pd.MultiIndex.from_product([tc
                                               for tc in self.test_factors.values()],
                                              names=[*self.test_factors.keys()])
        elif len(self.test_factors) == 1:
            (tf, tc) = [*self.test_factors.items()][0]
            return pd.CategoricalIndex(tc, name=tf)
        else:   # NO test factors, need one unnamed test condition anyway
            return pd.CategoricalIndex(['-'], name='-')   # plain Index ? empty labels ? ***

    def rvs_index(self, n_samples, sample_head='_sample'):
        """Make row MultiIndex to agree with sampled representation of resp-prob distribution
        :param n_samples: integer number of samples
        :return: a pd.MultiIndex suitable for cm_model.population.rvs(n_samples)
        """
        tc_index = self.test_cond_index()
        # stc_tuples_old = [(n, *tc) for (n, tc) in product(range(n_samples), self.test_conditions())]
        if len(self.test_factors) > 1:
            stc_tuples = [(n, *tc) for (n, tc) in product(range(n_samples), tc_index)]
        else:
            stc_tuples = [(n, tc) for (n, tc) in product(range(n_samples), tc_index)]
        return pd.MultiIndex.from_tuples(stc_tuples,
                                         names=[sample_head, *tc_index.names])

    @property
    def stim_index(self):
        """
        :return: pd.MultiIndex with levels (stim_group, stim)
        """
        stim_tuples = [(g, s)
                       for (g, sr_group) in self.stim_groups.items()
                       for s in sr_group.stim]
        return pd.MultiIndex.from_tuples(stim_tuples,
                                         names=[self.stim_group_head, self.stim_head])

    @property
    def stim_resp_index(self):
        """
        :return: a pd.MultiIndex with levels (stim_group, stim, resp)
            to be used for DataFrame storage of counts and results
        """
        stim_resp_tuples = [(g, s, r)
                            for (g, g_sr) in self.stim_groups.items()
                            for (s, r) in product(g_sr.stim, g_sr.resp)
                            ]
        sr_level_names = [self.stim_group_head, self.stim_head, self.resp_head]
        return pd.MultiIndex.from_tuples(stim_resp_tuples,
                                         names=sr_level_names)

    def _count_from_dict(self, sc):
        """Make Series representation of input stim count dict
        :param sc: scalar or nested dict: {stim_group: {stim: count}}
        :return: Series with index = self.stim_index
        """
        sc = {(sg, s): c
              for (sg, c_sg) in sc.items()
              for (s, c) in c_sg.items()}
        return pd.Series(list(sc.values()),
                            index=pd.MultiIndex.from_tuples(sc.keys()),
                            name=COUNT_COL)  # reindex-ed later
        # return s_count.reindex(self.stim_index, fill_value=0)

    def stim_prob(self):
        """Probability mass for all stim categories
        :return: ps = pd.Series with normalized probability mass,
            with index = self.stim_index,
            sum(ps) == 1
        """
        sc = self.stim_count
        if sc is None:
            sc = pd.Series(np.ones(len(self.stim_index)),
                           index=self.stim_index,
                           name=COUNT_COL)
        elif np.isscalar(sc):
            sc = pd.Series(sc * np.ones(len(self.stim_index)),
                             index=self.stim_index,
                             name=COUNT_COL)
        elif type(sc) is dict:
            sc = self._count_from_dict(sc)
        # else sc is already a pd.Series:
        sc = sc.reindex(self.stim_index, fill_value=0)
        return sc / np.sum(sc)
        # return self.stim_count / np.sum(self.stim_count)

    def gen_sr_slices(self):
        """Generator of index slices for 1D storage of
        response counts and probabilities.
        *** NEW in v. 0.8 ***
        :return: iterator yielding slices s_ind, for all stim categories:
            s_ind is an index slice for s-th (group, stim) category, such that
            x[..., s_ind] = response count/probability array,
                for all allowed responses to this stim
        """
        start = 0
        for (sg, sr_group) in self.stim_groups.items():
            for (i_s, s) in enumerate(sr_group.stim):
                n_resp = len(sr_group.resp)
                s_ind = slice(start, start + n_resp)
                yield s_ind
                start += n_resp

    def set_stim_count(self, sc):
        """Set stim count from observed data, IFF not already defined by user.
        :param sc: pd.Series of observed counts with
            index= MultiIndex with levels (stim_group, stim)
        :return: None
        Result: modified property self.stim_count, stored as pd.Series
        """
        # ***** keep this only in StimRespCountSet ?
        # NO, because cm_display needs to access it from here
        if self.stim_count is None:  # NOT already defined by user
            self.stim_count = sc.reindex(self.stim_index, fill_value=0)

    def group_head(self):
        return tuple(self.group_factors.keys())

    def gen_population_keys_in_factor(self, gf):
        """Generator all possible group categories within requested group "dimensions"
        :param gf: tuple with zero, one or more group "dimension" keys
        :return: iterator yielding tuples with group category labels, one for each given "dimension"
        """
        if len(self.group_factors) == 0:
            return [()]
        else:
            return product(*(self.group_factors[gf_k] for gf_k in gf))

    def gen_matching_group_keys(self, g_pattern):
        """Generate all possible group categories that match g_pattern
        :param g_pattern: dict defining ONE selected category in each of
            zero, one or more group "dimensions".
            if empty: generate ALL possible group categories
        :return: iterator yielding complete tuples of matching group categories
            in the order defined by self.group_head
        """
        if len(self.group_factors) == 0:
            return [()]
        g_keys = product(*([g_pattern[g_dim]] if g_dim in g_pattern.keys()
                           else self.group_factors[g_dim]
                           for g_dim in self.group_factors.keys()))
        return g_keys

    def population_model_weights(self, g_keys):
        """User-defined population weights for subset of population categories
        to be used for merging corresponding ema_group.PopulationModel instances
        :param g_keys: list of tuples, each defining ONE group category,
            i.e., with one category defined for each value in self.groups_dtypes.keys()
            in the correct order.
        :return: list with positive weight values, same length as g_keys
        """
        return np.array([self.population_weights[g] if g in self.population_weights else 1.
                      for g in g_keys])

    # ------------------------------------ correcting user typing errors:
    # NOTE: No warnings here, caller must warn if needed.
    def match_group_factor(self, g_factor):
        """Ensure that a tentative user-defined grouping factor (dimension)
        agrees with previous definition.
        :param g_factor: string label
        :return: g_factor_new <- g_factor corrected if needed and possible
        """
        return _match_string_label(g_factor, self.group_factors.keys(), 'group_factor')

    def match_group_key(self, g_key):
        """Ensure that a tentative user-defined group key
        is a tuple with group categories exactly matching definition in self.
        :param g_key: tuple (gc_1, ..., gc_K),
            with one selected category gc_i from each group factor in self.group_factors
        :return: g_new_key <- g_key corrected if needed and possible
        """
        if not isinstance(g_key, tuple):
            logger.warning(f'Group key {g_key} changed to {(g_key,)}. Must be a tuple.')
            g_key = (g_key,)
        if len(g_key) != len(self.group_factors):
            raise UserInputError(f'group-key tuple must have {len(self.group_factors)} labels'
                                 + ' to match ConfMatrixFrame.')
        g_new_key = tuple(_match_string_label(gk_i, g_cats_i,
                                              'group category')
                          for (gk_i, g_cats_i) in zip(g_key,
                                                      self.group_factors.values()))
        return g_new_key

    def match_test_factor(self, tf_key):
        """Ensure that a tentative user-defined test-factor key (= dimension)
        agrees with previous definition.
        :param tf_key: string label
        :return: sit_key_new <- sit_key corrected if needed and possible
        """
        return _match_string_label(tf_key, self.test_factors.keys(),
                                   'test factor')


# ----------------------------------------------------------
class StimRespCountSet:
    """Container for all observed confusion-count data from one experiment, i.e.,
    for ALL group_factors, ALL listeners, in ALL Test Conditions.
    """
    def __init__(self, cmf, cmd):
        """
        :param cmf: one ConfMatrixFrame instance defining common properties for the analysis.
        :param cmd: nested dict with individual stim-response results, stored as
            cm = cmd[group_id][subject_id] = one pd.Series instance, with
                one integer count for each unique index value, with
                index = MultiIndex with levels (*test-condition, stim_group, stim, resp), where
            group_id = tuple (g_cat_0, g_cat_1,...), identifying a population
            by one category selected for each group "dimension", defined in cmf.group_factors
            subject_id = any hashable object identifying a participant within the group,
            i.e., unique within each group, but may be re-used in different group_factors.
        """
        # *** Change to storing as
        # cmd[group_id][subject_id] = DataFrame
        #       with index = MultiIndex with levels (*test_factors, stim_group, stim)
        #       with columns like input data: Resp, or [Resp, Count] and non-unique index ? ***
        #       Summarize counts later, only when needed for calc?
        # *** OR more dict level(s):
        # *** cmd[group_id][subject_id][(stim_group, stim)] = pd.DataFrame with Resp, Count ?
        self.cmf = cmf
        self.cmd = cmd

    def __repr__(self):
        return self.__class__.__name__ + '(cmf=cmf, cmd=cmd)'

    def __str__(self):  # *** display average n stim presentations, too! ***
        def sum_stim(s_dict):
            """Sum total stim count across one group of subjects
            :param s_dict: dict with (s_id, pd.Series) elements
            :return: integer = total n stim across all subjects
            """
            return sum((s_ser.sum() for s_ser in s_dict.values()))
        # ------------------------------------------------------
        n_g = len(self.cmd)
        return (self.__class__.__name__ + ' with ' + f'{n_g} '
                + ('groups' if n_g > 1 else 'group')
                + ', including ' + ('\n\t' if n_g > 1 else '')
                + '\n\t'.join([f'{sum_stim(g_subjects)} stim-responses ' +
                               f'from {len(g_subjects)} subjects in group {repr(g)}'
                               for (g, g_subjects) in self.cmd.items()
                               ])
                + '\n')

    # *** separate classmethod initialize, method add; load = initialize + add ?
    # *** to allow collecting data from several sources, with different file formats...

    @classmethod
    def load(cls, cmf, path,
             fmt=None,
             participant='file',
             count_col=None,
             correct_col=None,
             path_groups=(),
             path_test_factors=(),
             path_join_str='_',
             rename_cols=None,
             read_fcn=None,
             **kwargs):
        """Create one class instance with selected data from input file(s).
        :param cmf: ConfMatrixFrame instance
        :param path: string or Path defining top of directory tree containing all data files.
            Files in all subdirectories are searched, hierarchically.  *** Can be a single file ? ****
        :param fmt: (optional) string with file suffix for data files.
            If None, all files are tried, so files with mixed formats can be used as input.
        :param participant: string defining where to find participant ID in a file,
            = column head in file data, 'file', or 'sheet' if excel-type file that has 'sheet's.
        :param count_col: (optional) string header of counts column in input files.
            None -> each (stim, resp) occurrence represents ONE test item.
        :param path_groups: (optional) sequence with group-factor "dimensions"
            for group categories that MAY be specified by substrings in file paths,
            used only if NOT specified in a column in the data file.
        :param path_test_factors: (optional) sequence with test-factor "dimensions"
            for test conditions that MAY be specified by substrings in file paths.
        :param path_join_str: (optional) string between factor label and category in file path,
            e.g., '_' in '... / Age_old / ...'
        :param rename_cols: (optional) dict with (old_name, new_name) for columns to be renamed
        :param read_fcn: (optional) user-defined file_reader function,
            in case standard pandas (read_csv, etc.) cannot handle input file format.
        :param kwargs: (optional) any additional arguments for the reader function
            e.g., converters={...}, use_cols=[...], header=..., etc..
        :return: a single cls object

        NOTE: All stim, resp, group, and test-condition categories found in data files
            MUST AGREE with labels previously defined in cmf.
            If needed, use keyword argument
            converters={column_head: convert_fcn, ...} to pandas reader function
            with converter function(s) to make saved data elements agree with pre-defined categories.
            Use keyword argument
            rename_cols={old_header: new_header, ...} to change file column headers to desired names.
        """
        def gen_sr_chunks(fmt, path_factors):  # *** for part -> load ?
            """Generator of DataFrame chunks from files
            :param fmt: suffix for files to be tried
            :param path_factors: dict with elements (factor, NominalDtype instance)
                to be found in file path, IF NOT in dataframe from file
            :return: iterator yielding pd.Series objects with integer stim-response counts, with
                index = (possibly non-unique) MultiIndex with levels
                        (*group_factors, participant, *test_factors, stim, resp)
            """
            req_col_types = {gf_key: NominalDtype(gf_val)
                             for (gf_key, gf_val) in cmf.group_factors.items()}
            req_col_types |= {participant: str}
            req_col_types |= {tf_key: NominalDtype(tf_val)
                             for (tf_key, tf_val) in cmf.test_factors.items()}
            # = colum data to be extracted first, as index levels
            for f_path in _gen_file_paths(path, None, fmt):
                logger.info(f'Reading file\n{f_path}')
                path_vars = _find_path_vars(f_path, path_factors, path_join_str)
                # = path data used if not already specified in some column
                try:
                    for df in stim_resp_gen(f_path,
                                            read_fcn,
                                            **kwargs):  # separate chunks from same file
                        if df.empty:  # in case file has some empty sheets
                            break  # and try next chunk
                        df.rename(columns=rename_cols, inplace=True, errors='ignore')
                        if correct_col is not None:
                            df = _set_correct_resp(df,
                                                   stim_col=cmf.stim_head,
                                                   resp_col=cmf.resp_head,
                                                   correct_col=correct_col)
                        df = _ffill_columns(df, list(req_col_types.keys()))  # **** sg, s, r ? *******
                        # with last specified value, in case NaN in input file
                        if participant == 'file':
                            df[participant] = f_path.stem
                        for (p_key, p_value) in path_vars.items():
                            if p_key not in df.columns and p_value is not None:
                                df[p_key] = p_value
                        try:
                            df = _check_cols(df, req_col_types)
                            df.set_index(list(req_col_types.keys()),
                                         inplace=True, append=False)
                            # df.columns should now include stim_groups, stim, and/or resp,
                            # and possibly count and other irrelevant data
                            if cmf.stim_head in df.columns and cmf.resp_head in df.columns:  # long format
                                yield _from_long_table(df, cmf, count_col)
                                # = pd.Series with index = MultiIndex with all required factors
                            else:  # try wide format
                                c_col = COUNT_COL if count_col is None else count_col
                                yield _from_wide_table(df, cmf, c_col)
                        except TableFormatError as e:  # something found missing
                            logger.warning(f'One unused data chunk: {e}')
                            # and just try next chunk
                except FileReadError as e:
                    logger.warning(f'Cannot read file: {e}')
                    # and just try next file

        def sum_up(s_sr):  # *** do this later, in gen_resp_counts(..) *****
            """Sum counts for each unique test condition and (stim, resp) pair
            for ONE participant in ONE group.
            :param s_sr: a pd.Series for ONE participant, with
                index = MultiIndex (*group_factors, participant, *test_factors, stim_group, stim, resp)
                but (group_id, participant) is identical in all s_sr rows, not needed.
                MAY contain more than one item for each (test-condition, stim, resp),
                in case the participant data were accumulated from more than one file.
                Therefore, need groupby(...).sum()
            :return: pd.Series object with stim-resp counts in each test condition, with
                UNIQUE MultiIndex, with levels (*test_factors, stim, resp)
                possibly still NOT ordered as defined by cmf.test_condition_index().
                Will be re-indexed later...
            """
            s_sr.name = count_col  # in case not already done
            index_cols = list(cmf.test_factors.keys()) + [cmf.stim_group_head, cmf.stim_head, cmf.resp_head]
            return s_sr.groupby(index_cols, observed=True, sort=False).sum()
        # ------------------------------------------------------------------------------

        # --------- check args:
        if fmt is not None and fmt[0] == '.':
            fmt = fmt[1:]  # used without the dot
        path = Path(path)
        if not path.exists():
            raise FileReadError(f'Path {path} does not exist')
        path_groups = [cmf.match_group_factor(g_key) for g_key in path_groups]  # allow minor spelling error
        path_groups = {gf: cmf.group_factors[gf]
                       for gf in path_groups}
        path_test_factors = [cmf.match_test_factor(tf) for tf in path_test_factors]
        path_test_factors = {tf: cmf.test_factors[tf]
                             for tf in path_test_factors}
        path_factors = path_groups | path_test_factors
        # = all data elements that MIGHT be specified as substring of absolute path
        if rename_cols is None:
            rename_cols = dict()
        # ----------------- args OK, ready to run
        # **** following: -> add method, to allow collecting data from different table formats ?
        sr_data = list(gen_sr_chunks(fmt, path_factors))
        if len(sr_data) == 0:
            raise FileReadError(f'*** No valid stim-response data found in {path}')
        sr_data = pd.concat(sr_data)
        # = ONE Series with index= MultiIndex
        #       with levels (*group_factors, participant, *test_factors, stim_group, stim, resp)
        # Store counts as nested dict instead: cmd[group_id][subject_id] = Series with
        # index = MultiIndex with levels (*test_factors, stim_group, 0stim, resp):
        if len(cmf.group_factors) == 0:
            cmd = {(): {s: sum_up(sr_s)
                        for (s, sr_s) in sr_data.groupby(participant,
                                                          observed=True, sort=False)}}
        else:
            cmd = {g: {s: sum_up(sr_gs)
                       for (s, sr_gs) in sr_g.groupby(participant,
                                                       observed=True, sort=False)}
                   for (g, sr_g) in sr_data.groupby(list(cmf.group_factors.keys()),
                                                     observed=True, sort=False)}
        # save total stim-count in cmf:  ***** do this in method check_complete ? ***********
        cmf.set_stim_count(sum((sum(s.groupby([cmf.stim_group_head, cmf.stim_head],
                                              observed=True, sort=False).sum()  # observed=False?
                                    for s in s_dict.values())
                                for s_dict in cmd.values())))
        return cls(cmf, cmd)

    # def add method, to include data from new files with different layout ???

    def save(self, dir, fmt='csv',
             # count_col=COUNT_COL,  # ****** check that participant Series always has Count name ******
             **file_kwargs):
        """Save all data to one directory, one file for each subject,
        in 'long' table format, i.e.,
        one row for each (test-cond, stim, resp) case.
        :param dir: path to directory for saving
        :param fmt: file format for saving
        :param file_kwargs: (optional) dict with keyword args for cm_file.Table.save() method,
            e.g.,
            write_fcn=...,
            allow_overwrite=False,
            sep=';',  for pd.DataFrame.to_csv()
        :return: None
        """
        if fmt[0] != '.':
            fmt = '.' + fmt
        if fmt in ['.xlsx', '.xls', '.odf', '.ods', '.odt']:
            if 'merge_cells' not in file_kwargs.keys():
                file_kwargs['merge_cells'] = False  # fix pd inconsistency v 3.0 != v 2.3 ? *******
        for (g, g_subjects) in self.cmd.items():
            if len(self.cmd) > 1 and len(g) > 0:
                group_dir = dir / group_dir_str(self.cmf.group_head(), g)
            else:
                group_dir = dir
            group_dir.mkdir(parents=True, exist_ok=True)
            for (s, s_data) in g_subjects.items():
                s_table = Table(s_data)  # ****** test if df.to_excel saves OK ? ********
                s_table.save((group_dir / s).with_suffix(fmt), **file_kwargs)

    def check_complete(self):
        """Check that most subjects have complete data
        for most test conditions.
        :return: None
            Only logger.warning about missing data
        """
        # *** simplify?
        # *** remove test-cond with too few or no data ?
        n_tfc = self.cmf.n_test_conditions
        n_groups = len(self.cmd)
        for (g, g_subjects) in self.cmd.items():
            n_subjects = len(g_subjects)
            incomplete_subjects = Counter()
            for (s, s_data) in g_subjects.items():
                if len(self.cmf.test_factors) > 0:
                    n_tc_pres = s_data.groupby([*self.cmf.test_factors.keys()], observed=False).sum()
                else:
                    n_tc_pres = {'No TestCond': s_data.sum()}
                for (tc, n_pres) in n_tc_pres.items():
                    if n_pres == 0:
                        logger.warning(f'No results for {tc} with subject {repr(s)} in group {repr(g)}')
                        incomplete_subjects.update({s: 1})
            for (s, c) in incomplete_subjects.items():
                logger.warning(f'Missing {c} of {n_tfc} test conditions for {repr(s)} ' +
                               (f'in group {repr(g)}' if n_groups > 1 else ''))
            if sum(c for c in incomplete_subjects.values()) > MISSING_LIMIT * n_tfc * n_subjects:
                logger.warning(f'*** Too much missing data in group {repr(g)}. Results may be unreliable. ***')

    def sum_stim_response_counts(self):
        """Generator yielding sum of all recorded stim-response counts
        for all subjects in each group,
        in predefined pd.DataFrame format, by self.cmf
        Used to check that data have been imported correctly.
        :return: iterator over groups,
            each yielding another iterator over stim_groups,
                yielding (sg, count_df) for each stim group, where
                count_df is a pd.DataFrame with
                index = MultiIndex with levels (*cmf.test_factors, Stim)
                one column for each allowed response
        """
        def sum_sr_count(sc):
            """Generator yielding sum of stim-response counts over subjects
            separated by stim groups
            :param sc: iterator yielding elements (s, c_s) where
                c_s is a pd.DataFrame with recorded stim response counts for ONE subject
                with index = test_factors, columns with levels (stim_group, stim, resp)
            :return: iterator yielding tuples (sg, c_sg), where
                sg is label of ONE stim group,
                c_sg is pd.DataFrame of counts, with index stim, columns response
            """
            sc = sum(c_s for (s, c_s) in sc)
            return ((sg, sc[sg].stack(level=0, future_stack=True))
                    for sg in self.cmf.stim_groups.keys()
                    )
        # --------------------------------------------------------
        return ((g_key, sum_sr_count(count_g))
                for (g_key, count_g) in self.gen_response_counts())

    # def get_response_count(self, group, participant):  # *** needed ? ***
    #     """Format response counts as a pd.DataFrame
    #     for ONE participant in ONE group.
    #     :param group: group id
    #     :param participant: subject id within given group
    #     :return: rc = pd.DataFrame with
    #         index = self.cmf.test_cond_index(), in predefined order
    #         columns = self.cmf.stim_resp_index_OLD(), in predefined order
    #     """
    #     return self._stim_resp_df(self.cmd[group][participant])

    def gen_response_counts(self):
        """Generator yielding all recorded stim-response counts
        in fixed pd.DataFrame format, defined by self.cmf
        :return: iterator over groups,
            each yielding another iterator over participants within the group
                yielding tuples (s, df_s), where
                s is subject id label,
                df_s is pd.DataFrame with
                index = self.cmf.test_factor_index
                colums = self.cmf.stim_resp_index
        """
        return ((g_key, ((s_key, self._stim_resp_df(sr))
                         for (s_key, sr) in g_data.items()))
                for (g_key, g_data) in self.cmd.items())

    # --------------------------- private:
    def _stim_resp_df(self, sr):
        """Show stim-resp counts for ONE participant
        in FIXED pd.DataFrame format as defined by self.cmf,
        with same indexing for all participants.
        :param sr: a pd.Series instance with stim-resp data for ONE participant
        :return: rc = pd.DataFrame with
            index = self.cmf.test_cond_index(), in predefined order
            columns = self.cmf.stim_resp_index, in predefined order
        """
        # 2025-12-11, detected unexpected result with unstack(..., sort=False ) pandas bug?
        if len(self.cmf.test_factors) == 0:
            sr_df = sr.to_frame(name='-').T  # to agree with cmf.test_cond_index()
        else:
            sr_df = sr.unstack(level=self.cmf.stim_resp_index.names, fill_value=0, sort=True)  # OK
        # *** reindex deletes any row or column that do not agree with given indices !
        sr_df = sr_df.reindex(index=self.cmf.test_cond_index(),
                              columns=self.cmf.stim_resp_index,
                              fill_value=0)
        return sr_df


# ----------------------------------------------- general help functions

# ----------------------------------------------- module-local help functions
# def _make_sr_index(stim_groups, no_response, level_names):
#     """Create index for (stim_group, stim, resp) tuples in fixed order
#     :param stim_groups: dict with elements (stim_group, stim_list)
#     :param no_response: single string label for missing response, if allowed
#         = None if not allowed in experimental setup
#     :param level_names: list with 3 names for resulting MultiIndex
#     :return: pd.MultiIndex with 3 levels (stim_group, stim, resp)
#     """
#     if no_response is None:
#         no_response = []
#     else:
#         if type(no_response) is not list:
#             no_response = [no_response]
#     sr = []  # list of (sg, s, r) tuples
#     for (sg, s_list) in stim_groups.items():
#         sr.extend(product([sg], s_list, s_list + no_response))
#     return pd.MultiIndex.from_tuples(sr, names=level_names)


def _set_correct_resp(df, stim_col, resp_col, correct_col):
    """Assign df[resp] = df[stim], where correct_col == True defines resp
    :param df: a pd.DataFrame instance
    :param stim_col: stim column name
    :param resp_col: resp column name
    :param correct_col: correct column name
    :return: copy of df with modified resp_col values
    """
    if not df[resp_col].hasnans:
        return df
    corr = df[correct_col].astype(bool)  # and df[resp_col].isnan()
    df.loc[corr, resp_col] = df.loc[corr, stim_col]
    return df


def _match_string_label(k, k_list, k_msg):
    """Ensure one of allowed alternatives
    :param k: key string to be checked
    :param k_list: iterable of different allowed alternatives
    :param k_msg: string label for error msg
    :return: matched_k = best-matching element in k_list
        otherwise EMAinputError raised
    """
    k_matches = get_close_matches(k, k_list)
    if len(k_matches) == 0:
        raise UserInputError('Unknown ' + k_msg + ': ' + repr(k))
    if k_matches[0] != k:
        if len(k_matches) == 1:  # Unique but approximate match
            logger.warning('Misspelled ' + k_msg + ': ' + repr(k) + '? '
                           + f'Using {repr(k_matches[0])}')
        else:  # non-unique approximate match
            logger.warning('Ambiguous ' + k_msg + ': ' + repr(k) + '. '
                           + f'Using the first of {k_matches}')
    return k_matches[0]  # use best match anyway


def group_dir_str(g_head, g, sep='_'):
    """Convert group id to a directory path string
    :param g_head: tuple of strings with group "dimensions"
    :param g: tuple of labels with corresponding group categories
    :param sep: string joining group dimension and category
    :return: string to be used as directory path
    """
    s = '/'.join((str(h_i) + sep + str(g_i)
                     for (h_i, g_i) in zip(g_head, g)))
    if s == sep:  # only a single group with empty head and category
        return ''
    else:
        return s


def _ffill_columns(df, col_names):
    """Fill DataFrame columns inplace, by copy downwards, if NaN
    :param df: a pd.DataFrame instance
    :param col_names: list with names of columns to check and fill
    :return: df with modified columns
    """
    for c in col_names:
        try:
            df[c] = df[c].ffill()
        except KeyError:
            pass
    return df


# def _include_path_data(df, f_path, path_factors, path_join_str):
#     """Scan path string for needed categories not already present in df
#     :param df: a pd.DataFrame obtained from a file
#     :param f_path: path to file from which df was read
#     :param path_factors: dict with (factor_key, categories)
#     :param path_join_str: string connecting factor and category,
#     :return: df with some new columns, if any found in f_path
#     """
#     for (p_key, p_cats) in path_factors.items():
#         if p_key not in df.columns:  # MUST be defined somewhere in the file path
#             try:
#                 df[p_key] = _find_category_in_path(f_path,
#                                                    p_key, p_cats,
#                                                    path_join_str)
#             except ValueError:  # required category not found
#                 raise TableFormatError(f'Factor {p_key} not found in path.')
#     return df


def _find_path_vars(f_path, path_factors, join_str):
    """Find requested factor-categories in path
    :param f_path: complete file path to search
    :param path_factors: dict with items (factor, cat_list)
    :param join_str: string connecting factor and category in path string
    :return: dict with elements (factor, category) for all factors
        category = None if not found
    """
    return {p_key: _find_category_in_path(f_path,
                                          p_key, p_cats,
                                          join_str)
            for (p_key, p_cats) in path_factors.items()}


def _find_category_in_path(f_path, p_key, p_cats, join_str):
    """Search for allowed (p_key, category) in path string
    :param f_path: path to scan
    :param p_key: string factor label
    :param p_cats: NominalDtype instance defining allowed categories for p_key
    :param join_str: string connecting factor and category,
        e.g., '_' in 'Age_old' for p_key='Age' and 'old' in p_dtype.categories.
    :return: cat = first found category, if any
    """
    f_path = str(f_path)
    # for c in p_cats.categories:
    for c in p_cats:
        sub_str = (p_key + join_str + str(c))
        if sub_str in f_path:
            return c
    # -------- try to find only a p_cats category instead:
    # for c in p_cats.categories:
    for c in p_cats:
        if str(c) in f_path:
            logger.warning(f'No pattern {repr(p_key + join_str + "xxx")} found in path. '
                           +f'Using {repr(p_key)}={repr(c)}')
            return c
    return None


def _gen_file_paths(p, sub_dir=None, suffix=None):
    """generator of all file Paths in directory tree p,
    recursively, with desired name pattern
    :param p: Path instance defining top directory to be searched
    :param sub_dir: (optional) subdirectory name
    :param suffix: (optional) file suffix of desired files, without dot
    :return: iterator of Path objects, each defining one existing data file
    """
    p = Path(p)  # just in case
    if p.is_file():
        return [p]
    if sub_dir is not None and len(sub_dir) > 0:
        p = p / sub_dir  # search only in sub_dir
    if suffix is None:
        glob_pattern = '*.*'  # read any file types
    elif suffix == 'ost':  # special private file format
        glob_pattern = '*.txt'
    else:
        glob_pattern = '*.' + suffix  # require suffix
    return (f for f in p.rglob(glob_pattern)
            if f.is_file() and f.stem[0] != '.')


def _check_cols(df, col_dtypes):
    """Ensure that required DataFrame columns are present and have the desired data type
    :param df: a pd.DataFrame with tentative data read from a file
    :param col_dtypes: dict with elements (col_name, dtype)
    :return: copy of df with data modified if needed
        TableFormatError, if missing column, or non-acceptable data type
    """
    missing_cols = []
    for (c, c_type) in col_dtypes.items():
        if c not in df.columns:
            missing_cols.append(c)
    if len(missing_cols) > 0:
        raise TableFormatError('Missing factor(s): ' + repr(missing_cols))
    try:
        df = df.astype(col_dtypes, copy=True, errors='raise')
        # sets NaN where type is incorrect, without error,
        # raises error only if typecast is impossible.
    except Exception as e:
        raise TableFormatError(f'Unacceptable data type: {e}')
    # *** check if any col is all-NaN ********
    na_col = df[list(col_dtypes)].isna().all()
    na_col = [c_key for (c_key, c_na) in na_col.items() if c_na]
    if len(na_col) > 0:
        raise TableFormatError(f'No acceptable values of {na_col}')
    df = df.dropna(axis=0, how='any', subset=list(col_dtypes.keys()))
    return df


def _from_wide_table(cm, cmf, count_head):
    """Convert file-input table from "wide" to default "long" format.
    :param cm: a pd.DataFrame instance with
        index = MultiIndex with all required factors, EXCEPT (stim, resp), and EITHER
        one column stim_head, and other columns for elements in cmf.resp.categories,
        OR
        one column resp_col, and other columns for elements in cmf.stim.categories
    :param cmf: a ConfMatrixFrame instance defining (stim, resp) names and categories
    :param count_head: head of created count column in result
    :return: a pd.Series, with
        index = MultiIndex with all required factors, INCL (stim, resp)
    """
    # *** store result as DataFrame with columns cmf.resp_head, count_head ? *****
    if cmf.stim_head in cm.columns:  # wide-format matrix indexed (stim, resp)
        if cmf.stim_group_head not in cm.columns:
            if len(cmf.stim_groups) == 1:
                cm[cmf.stim_group_head] = cmf.stim_groups[0]
            else:
                raise TableFormatError('Wide-format stim-response data can have only ONE stim group')
        cm = cm.set_index([cmf.stim_group_head, cmf.stim_head], inplace=False, append=True)
        resp_cat = cmf.stim_resp_index.levels[2]
        r_cols = [c for c in cm.columns if c in resp_cat]
        if len(r_cols) == 0:
            raise TableFormatError('No response columns in input data')
        cm = cm[r_cols]
        cm.columns.name = cmf.resp_head
        return cm.stack().astype(COUNT_TYPE)
    elif cmf.resp_head in cm.columns:  # transposed indexing (resp, stim)
        # *** allow (stim_group, stim) as column MultiIndex ?
        if len(cmf.stim_groups) > 1:
            raise TableFormatError('Wide-format stim-response data can have only ONE stim group')
        cm[cmf.stim_group_head] = list(cmf.stim_groups.keys())[0]  # *** only one anyway
        cm = cm.set_index([cmf.stim_group_head, cmf.resp_head], inplace=False, append=True)
        stim_cat = cmf.stim_resp_index.levels[1]
        s_cols = [c for c in cm.columns if c in stim_cat]
        if len(s_cols) == 0:
            raise TableFormatError('No stimulus columns in input chunk')
        cm = cm[s_cols]
        cm.columns.name = cmf.stim_head
        cm = cm.stack().astype(COUNT_TYPE)
        return cm.swaplevel()   # (stim_group, resp, stim) -> (stim_group, stim, resp)
    else:
        raise TableFormatError('No stim-response columns in input data')


def _from_long_table(df, cmf, count_col):
    """Check and re-format to standard index / column style
    :param df: pd.DataFrame instance in long format, with
        index = MultiIndex with all required factors, EXCEPT (stim_group, stim, resp),
        columns = stim_group, stim, resp, and optionally count
    :param cmf: a ConfMatrixFrame instance defining stim_resp_index with names and categories
    :param count_col: (optional) name of count column, if expected in data
        None -> each (stim, resp) row represents ONE count stored in default column COUNT_COL
    :return: a pd.Series, with
        index = MultiIndex with all required factors, INCL (stim_group, stim, resp)
    """
    if len(cmf.stim_groups) == 1:  # backward compatibility fix
        if cmf.stim_group_head not in df.columns:
            df[cmf.stim_group_head] = list(cmf.stim_groups.keys())[0]  # label of this ONLY stim group
    if count_col is None:
        count_col = COUNT_COL
    if count_col not in df.columns:   # ***** WARNING or Error ? *****
        df[count_col] = 1  # each (stim, resp) row means ONE response
    # ******* dtype checks not needed here, any incorrect items deleted by reindex in _stim_resp_df(self, sr)
    df.set_index(cmf.stim_resp_index.names,
                 inplace=True, append=True)  # ****** inplace False ?
    return df[count_col]  # i.e., skip any other irrelevant data columns   **** keep until later check?
